Top 23 Transformer Open-Source Projects

  • transformers

    🤗 Transformers: State-of-the-art Machine Learning for Pytorch, TensorFlow, and JAX.

    Project mention: GPU Comparisons: RTX 6000 ADA vs A100 80GB vs 2x 4090s | | 2022-12-02

    Looked into this last night and yeah, NVLink works the way you described because of misleading marketing- no contiguous memory pool, just a faster interconnect so maybe model parallelisation scales a bit better but you still have to implement it. Also saw an example where some PyTorch GPT2 models scaled horrifically in training with multiple PCIe V100s and 3090s that didn’t have NVLink so that’s a caveat with dual 4090s not having NVLink.

  • nn

    🧑‍🏫 59 Implementations/tutorials of deep learning papers with side-by-side notes 📝; including transformers (original, xl, switch, feedback, vit, ...), optimizers (adam, adabelief, ...), gans(cyclegan, stylegan2, ...), 🎮 reinforcement learning (ppo, dqn), capsnet, distillation, ... 🧠

    Project mention: [D] Recent ML papers to implement from scratch | | 2022-10-10
  • InfluxDB

    Build time-series-based applications quickly and at scale.. InfluxDB is the Time Series Data Platform where developers build real-time applications for analytics, IoT and cloud-native services in less time with less code.

  • CVPR2022-Papers-with-Code

    CVPR 2022 论文和开源项目合集

  • trax

    Trax — Deep Learning with Clear Code and Speed

  • PaddleNLP

    👑 Easy-to-use and powerful NLP library with 🤗 Awesome model zoo, supporting wide-range of NLP tasks from research to industrial applications, including 🗂Text Classification, 🔍 Neural Search, ❓ Question Answering, ℹ️ Information Extraction, 📄 Document Intelligence, 💌 Sentiment Analysis and 🖼 Diffusion AIGC system etc.

    Project mention: The 10 Trending Python Repositories on GitHub (May 2022) | | 2022-06-23


  • PaddleSeg

    Easy-to-use image segmentation library with awesome pre-trained model zoo, supporting wide-range of practical tasks in Semantic Segmentation, Interactive Segmentation, Panoptic Segmentation, Image Matting, 3D Segmentation, etc.

    Project mention: [P] New Features of Image Segmentation Project PaddleSeg (6k stars) | | 2022-12-01


  • jukebox

    Code for the paper "Jukebox: A Generative Model for Music"

    Project mention: Would my idea of using SD for Audio generation make sense or am I nuts? | | 2022-12-06

    I would suggest checking out OpenAI Jukebox if you haven't already. People slept on this and called it useless, but it's the most compelling generative AI music project I've heard personally. There's a Colab notebook for it.

  • Zigi

    Workflow assistant built for devs & their teams. Automate the mundane part of your day, with live actionable messages for your GitHub & Jira tasks.

  • PaddleSpeech

    Easy-to-use Speech Toolkit including Self-Supervised Learning model, SOTA/Streaming ASR with punctuation, Streaming TTS with text frontend, Speaker Verification System, End-to-End Speech Translation and Keyword Spotting. Won NAACL2022 Best Demo Award.

    Project mention: project | | 2022-12-07
  • GPT2-Chinese

    Chinese version of GPT2 training code, using BERT tokenizer.

  • BERT-pytorch

    Google AI 2018 BERT pytorch implementation

  • mmsegmentation

    OpenMMLab Semantic Segmentation Toolbox and Benchmark.

    Project mention: MMDeploy: Deploy All the Algorithms of OpenMMLab | | 2022-11-21

    MMSegmentation: OpenMMLab semantic segmentation toolbox and benchmark.

  • bertviz

    BertViz: Visualize Attention in NLP Models (BERT, GPT2, BART, etc.)

    Project mention: using bert for relation extraction | | 2022-08-10

    2) BERT learns a lot in its embeddings: the BERTOLOGY paper ( provides a great in-depth look at some of the broader linguistic traits that BERT learns. Different layers often learn different patterns, so the embeddings aren't really interpretable, but you can use something like bertviz ( to explore attention weights across layers for predetermined examples

  • pytorch-seq2seq

    Tutorials on implementing a few sequence-to-sequence (seq2seq) models with PyTorch and TorchText.

    Project mention: A Good Github Repo to Look at (CS388 Natural Language Processing) | | 2022-10-29

    I don't know how many people are taking CS388 NLP in Fall 2022, but the assignment is really putting lots of stress on me. I was searching some good materials to prepare for NLP class, and a really good resource to look at is this github repo:

  • Efficient-AI-Backbones

    Efficient AI Backbones including GhostNet, TNT and MLP, developed by Huawei Noah's Ark Lab.

    Project mention: Researchers From China Introduce Vision GNN (ViG): A Graph Neural Network For Computer Vision Systems | | 2022-06-08

    Continue reading | Check out the paper, github

  • posthtml

    PostHTML is a tool to transform HTML/XML with JS plugins

    Project mention: How to set base url for root-relative url? | | 2022-08-09

    A bundler running an html transformer like could do this. Then the html would just be dependent on an environment variable to set your prefix conditions.

  • tsai

    Time series Timeseries Deep Learning Machine Learning Pytorch fastai | State-of-the-art Deep Learning library for Time Series and Sequences in Pytorch / fastai

    Project mention: What is the current state-of-art in sequence classification? | | 2022-10-13

    You might be interested in tsai. I am not affiliated with them and have not used tsai, but I have been planning to try it for too long … well :p

  • HRNet-Semantic-Segmentation

    The OCR approach is rephrased as Segmentation Transformer: This is an official implementation of semantic segmentation for HRNet.

    Project mention: Some general and HRNetV2+OCR related questions (Semantic Segmentation) | | 2022-11-30

    Hi everyone. I'm trying utilize popular HRNetV2+OCR model posted in this repo

  • wenet

    Production First and Production Ready End-to-End Speech Recognition Toolkit

    Project mention: Project Alice – an open source virtual assistant that can run offline | | 2022-05-01
  • LaTeX-OCR

    pix2tex: Using a ViT to convert images of equations into LaTeX code.

    Project mention: How do I make this in Latex and what's it called? | | 2022-11-05

    Maybe this open source project is more for you then. It’s basically the same idea: I’ve not used this though, just mathpix. I ran into this while googling to see if there were alternatives to mathpix

  • lightseq

    LightSeq: A High Performance Library for Sequence Processing and Generation

  • SwinIR

    SwinIR: Image Restoration Using Swin Transformer (official repository)

    Project mention: Robot dance animation with Robo-Diffusion (1024x576) | | 2022-11-15

    Use SwinIR medium model to upscale by 2 times. This will result in a video of 2048x1152.

  • OpenPrompt

    An Open-Source Framework for Prompt-Learning.

  • whisper.cpp

    Port of OpenAI's Whisper model in C/C++

    Project mention: 뉴스 스크랩 2022-12-08 | | 2022-12-07
  • Scout APM

    Truly a developer’s best friend. Scout APM is great for developers who want to find and fix performance issues in their applications. With Scout, we'll take care of the bugs so you can focus on building great things 🚀.

NOTE: The open source projects on this list are ordered by number of github stars. The number of mentions indicates repo mentiontions in the last 12 Months or since we started tracking (Dec 2020). The latest post mention was on 2022-12-07.

Transformer related posts


What are some of the best open-source Transformer projects? This list will help you:

Project Stars
1 transformers 75,627
2 nn 15,087
3 CVPR2022-Papers-with-Code 9,817
4 trax 7,201
5 PaddleNLP 6,740
6 PaddleSeg 6,107
7 jukebox 5,772
8 PaddleSpeech 5,474
9 GPT2-Chinese 5,463
10 BERT-pytorch 5,244
11 mmsegmentation 4,858
12 bertviz 4,656
13 pytorch-seq2seq 4,186
14 Efficient-AI-Backbones 2,920
15 posthtml 2,856
16 tsai 2,774
17 HRNet-Semantic-Segmentation 2,734
18 wenet 2,665
19 LaTeX-OCR 2,513
20 lightseq 2,500
21 SwinIR 2,387
22 OpenPrompt 2,144
23 whisper.cpp 2,097
Write Clean Python Code. Always.
Sonar helps you commit clean code every time. With over 225 unique rules to find Python bugs, code smells & vulnerabilities, Sonar finds the issues while you focus on the work.