Top 13 Python pretraining Projects

pretraining

• Add a project

1. LMOps

   1 6 4,403 6.9 Python

   General technology for enabling AI capabilities w/ LLMs and MLLMs

   

2. SaaSHub

   www.saashub.com featured

   SaaSHub - Software Alternatives and Reviews. SaaSHub helps you find the best software and product alternatives

   

3. OFA

   2 3 2,557 2.8 Python

   Official repository of OFA (ICML 2022). Paper: OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Framework

   

4. mPLUG-Owl

   3 2 2,539 6.3 Python

   mPLUG-Owl: The Powerful Multi-modal Large Language Model Family

   

5. SparK

   4 3 1,370 7.3 Python

   [ICLR'23 Spotlight🔥] The first successful BERT/MAE-style pretraining on any convolutional network; Pytorch impl. of "Designing BERT for Convolutional Networks: Sparse and Hierarchical Masked Modeling" (by keyu-tian)

   

6. ImageNet21K

   5 1 780 10.0 Python

   Official Pytorch Implementation of: "ImageNet-21K Pretraining for the Masses"(NeurIPS, 2021) paper

   

7. PITI

   6 1 502 0.6 Python

   PITI: Pretraining is All You Need for Image-to-Image Translation

   

8. LinkBERT

   7 2 449 1.8 Python

   [ACL 2022] LinkBERT: A Knowledgeable Language Model 😎 Pretrained with Document Links

   

9. llm-baselines

   8 1 118 6.9 Python

   nanoGPT-like codebase for LLM training

   

10. Revisiting-Contrastive-SSL

    9 1 89 0.0 Python

    Revisiting Contrastive Methods for Unsupervised Learning of Visual Representations. [NeurIPS 2021]

    

11. awesome-clip-papers

    10 1 78 5.4 Python

    The most impactful papers related to contrastive pretraining for multimodal models!

    

12. tabular-dl-pretrain-objectives

    11 1 68 10.0 Python

    Revisiting Pretrarining Objectives for Tabular Deep Learning

    

13. autolab

    12 1 6 6.4 Python

    Companion tools for Karpathy's autoresearch - smarter evaluation, guided steering, and multi-agent competitions for GPT pretraining

    

    Project mention: How I Built Eval Tools for Karpathy's Autoresearch | dev.to | 2026-03-18

    Repo: github.com/dean0x/autolab

14. easy-torch-tpu

    13 2 5 8.4 Python

    A flexible pipeline for training custom research-scale models on Google Cloud TPUs using PyTorch/XLA

    

    Project mention: TorchTPU: Running PyTorch Natively on TPUs at Google Scale | news.ycombinator.com | 2026-04-23

    This is great to see.

    I did trained some research models using the existing PyTorch/XLA on TPUs, and it was a mess of undocumented behavior and bugs (silently hanging after 8 hours of training!).

    If anyone is trying to use PyTorch on TPU before TorchTPU is released, you can check out the training pipeline that I ended up building to support my research: https://github.com/aklein4/easy-torch-tpu

Python pretraining related posts

• TorchTPU: Running PyTorch Natively on TPUs at Google Scale

  1 project | news.ycombinator.com | 23 Apr 2026

• [D] Fine tuning LLM - VM requirements.

  1 project | /r/MachineLearning | 2 Jan 2023

Index

Sponsored

SaaSHub - Software Alternatives and Reviews

SaaSHub helps you find the best software and product alternatives

www.saashub.com

Do not miss the trending Python projects with our weekly report!