Python Transformers

Open-source Python projects categorized as Transformers

Top 23 Python Transformer Projects

  • vit-pytorch

    Implementation of Vision Transformer, a simple way to achieve SOTA in vision classification with only a single transformer encoder, in Pytorch

    Project mention: Is it easier to go from Pytorch to TF and Keras than the other way around? | /r/pytorch | 2023-05-13

    I also need to learn Pyspark so right now I am going to download the Fashion Mnist dataset, use Pyspark to downsize each image and put the into separate folders according to their labels (just to show employers I can do some basic ETL with Pyspark, not sure how I am going to load for training in Pytorch yet though). Then I am going to write the simplest Le Net to try to categorize the fashion MNIST dataset (results will most likely be bad but it's okay). Next, try to learn transfer learning in Pytorch for both CNN or maybe skip ahead to ViT. Ideally at this point I want to study the Attention mechanism a bit more and try to implement Simple Vit which I saw here:

  • PaddleNLP

    👑 Easy-to-use and powerful NLP library with 🤗 Awesome model zoo, supporting wide-range of NLP tasks from research to industrial applications, including 🗂Text Classification, 🔍 Neural Search, ❓ Question Answering, ℹ️ Information Extraction, 📄 Document Intelligence, 💌 Sentiment Analysis and 🖼 Diffusion AIGC system etc.

    Project mention: Chatgpt 到底是不是开源的? | /r/China_irl | 2023-03-25
  • Sonar

    Write Clean Python Code. Always.. Sonar helps you commit clean code every time. With over 225 unique rules to find Python bugs, code smells & vulnerabilities, Sonar finds the issues while you focus on the work.

  • haystack

    :mag: Haystack is an open source NLP framework to interact with your data using Transformer models and LLMs (GPT-4, ChatGPT and alike). Haystack offers production-ready tools to quickly build complex question answering, semantic search, text generation applications, and more.

    Project mention: Is anyone doing Machine Learning in Rust? | /r/rust | 2023-05-11
  • nebuly

    The next-generation platform to monitor and optimize your AI costs in one place 🚀

    Project mention: What are you building with LLMs? I'm writing an article about what people are building with LLMs | /r/programming | 2023-03-27

    Hi everyone. I’m the creator of ChatLLaMA, an opensource framework to train LLMs with limited resources and create There’s been amazing usage of LLMs in these days, from chatbots to retrieve about company’s product information, to cooking assistants for traditional dishes, and much more. And you? What you building or would love to build with LLMs? Let me know and I’ll share the article about your stories soon. Cheers


    RWKV is an RNN with transformer-level LLM performance. It can be directly trained like a GPT (parallelizable). So it's combining the best of RNN and transformer - great performance, fast inference, saves VRAM, fast training, "infinite" ctx_len, and free sentence embedding.

    Project mention: New model: RWKV-4-Raven-7B-v12-Eng49%-Chn49%-Jpn1%-Other1%-20230530-ctx8192.pth | /r/LocalLLaMA | 2023-05-31

    See for details on the RWKV Language Model (100% RNN).

  • PaLM-rlhf-pytorch

    Implementation of RLHF (Reinforcement Learning with Human Feedback) on top of the PaLM architecture. Basically ChatGPT but with PaLM

    Project mention: How should I get an in-depth mathematical understanding of generative AI? | /r/datascience | 2023-05-18

    ChatGPT isn't open sourced so we don't know what the actual implementation is. I think you can read Open Assistant's source code for application design. If that is too much, try Open Chat Toolkit's source code for developer tools . If you need very bare implementation, you should go for lucidrains/PaLM-rlhf-pytorch.

  • peft

    🤗 PEFT: State-of-the-art Parameter-Efficient Fine-Tuning.

    Project mention: [D] [LoRA + weight merge every N step] for pre-training? | /r/MachineLearning | 2023-05-29

    you could use a callback, like show here, and call code to merge them here.

  • InfluxDB

    Access the most powerful time series database as a service. Ingest, store, & analyze all types of time series data in a fully-managed, purpose-built database. Keep data forever with low-cost storage and superior data compression.

  • speechbrain

    A PyTorch-based Speech Toolkit

    Project mention: Whisper.cpp | /r/LocalLLaMA | 2023-06-02

  • gpt-neox

    An implementation of model parallel autoregressive transformers on GPUs, based on the DeepSpeed library.

    Project mention: Best open source LLM model for commercial use | /r/LocalLLaMA | 2023-05-10

    Gpt neox 20B can be used commerically.

  • bertviz

    BertViz: Visualize Attention in NLP Models (BERT, GPT2, BART, etc.)

    Project mention: Ask HN: Can someone ELI5 Transformers and the “Attention is all we need” paper | | 2023-05-17

    The Illustrated Transfomer ( ) and Visualizing attention ( ), are both really good resources. For a more ELI5 approach this non-technical explainer ( ) covers it at a high level.

  • DALLE-pytorch

    Implementation / replication of DALL-E, OpenAI's Text to Image Transformer, in Pytorch

    Project mention: Thoughts on AI image generators from text | /r/conspiracy | 2022-08-09

    Here you go:

  • deep-daze

    Simple command line tool for text to image generation using OpenAI's CLIP and Siren (Implicit neural representation network). Technique was originally created by

    Project mention: Besides Gaming - for what can be a 4080 useful? | /r/nvidia | 2023-04-15
  • BERTopic

    Leveraging BERT and c-TF-IDF to create easily interpretable topics.

    Project mention: SBERT Embeddings from Conversations | /r/LanguageTechnology | 2023-03-03

    Try out this notebook which comes with the BERTopic repository.

  • simpletransformers

    Transformers for Classification, NER, QA, Language Modelling, Language Generation, T5, Multi-Modal, and Conversational AI

  • yolov7_d2

    🔥🔥🔥🔥 (Earlier YOLOv7 not official one) YOLO with Transformers and Instance Segmentation, with TensorRT acceleration! 🔥🔥🔥

    Project mention: YOLOv7: Trainable Bag-of-Freebies | | 2022-07-16

    Especially hilarious considering some other people ALSO jumped on the "we made an object detector so let's call it YOLOvX" wagon and released...

    Something called YOLOv7.

  • x-transformers

    A simple but complete full-attention transformer with a set of promising experimental features from various papers

    Project mention: GPT-4 architecture: what we can deduce from research literature | | 2023-03-14
  • courses

    This repository is a curated collection of links to various courses and resources about Artificial Intelligence (AI) (by SkalskiP)

    Project mention: GitHub - SkalskiP/courses: This repository is a curated collection of links to various courses and resources about Artificial Intelligence (AI) | /r/integratedai | 2023-05-26
  • musiclm-pytorch

    Implementation of MusicLM, Google's new SOTA model for music generation using attention networks, in Pytorch

    Project mention: Training musiclm | /r/singularity | 2023-06-01

    Has anyone tried to train this model : lucidrains/musiclm-pytorch: Implementation of MusicLM, Google's new SOTA model for music generation using attention networks, in Pytorch ( ? Could you provide any useful resources that can help me? Or share your process?

  • scenic

    Scenic: A Jax Library for Computer Vision Research and Beyond (by google-research)

    Project mention: Vid2Seq: A pretrained visual language model for describing multi-event videos | | 2023-03-17

    Anyone figured out how to run this against a video? has an example showing how to "train Vid2Seq on YouCook2" using "python -m scenic.projects.vid2seq.main", but I couldn't see the recipe for using it against a video to return a description.

  • argos-translate

    Open-source offline translation library written in Python

    Project mention: The state imposes Google (or Apple) on me | /r/StallmanWasRight | 2023-03-20

    Consider a free software alternative to this translation service, such as Argos Translate or its web based frontend LibreTranslate.

  • reformer-pytorch

    Reformer, the efficient Transformer, in Pytorch

  • scikit-llm

    Seamlessly integrate powerful language models like ChatGPT into scikit-learn for enhanced text analysis tasks.

    Project mention: FLaNK Stack Weekly 29 may 2023 | | 2023-05-30
  • adapter-transformers

    Huggingface Transformers + Adapters = ❤️

    Project mention: [D] NLP question: does fine-tuning train input embedding? | /r/MachineLearning | 2022-08-07

    Usually in computer vision resnets, people finetune only the last layers, but in NLP you tune the entire model. There are also plenty of instances where people try to not do this, such as in adapters, however.

  • SaaSHub

    SaaSHub - Software Alternatives and Reviews. SaaSHub helps you find the best software and product alternatives

NOTE: The open source projects on this list are ordered by number of github stars. The number of mentions indicates repo mentiontions in the last 12 Months or since we started tracking (Dec 2020). The latest post mention was on 2023-06-02.

Python Transformers related posts


What are some of the best open-source Transformer projects in Python? This list will help you:

Project Stars
1 vit-pytorch 14,271
2 PaddleNLP 9,259
3 haystack 8,976
4 nebuly 8,152
5 RWKV-LM 7,913
6 PaLM-rlhf-pytorch 6,911
7 peft 6,457
8 speechbrain 6,010
9 gpt-neox 5,538
10 bertviz 5,256
11 DALLE-pytorch 5,222
12 deep-daze 4,386
13 BERTopic 4,249
14 simpletransformers 3,681
15 yolov7_d2 3,061
16 x-transformers 2,975
17 courses 2,640
18 musiclm-pytorch 2,441
19 scenic 2,208
20 argos-translate 1,957
21 reformer-pytorch 1,906
22 scikit-llm 1,678
23 adapter-transformers 1,633
SaaSHub - Software Alternatives and Reviews
SaaSHub helps you find the best software and product alternatives