Top 14 Python attention-mechanism Projects

• PaLM-rlhf-pytorch

  25 7,593 4.6 Python

  Implementation of RLHF (Reinforcement Learning with Human Feedback) on top of the PaLM architecture. Basically ChatGPT but with PaLM

Project mention: How should I get an in-depth mathematical understanding of generative AI? | /r/datascience | 2023-05-18

ChatGPT isn't open sourced so we don't know what the actual implementation is. I think you can read Open Assistant's source code for application design. If that is too much, try Open Chat Toolkit's source code for developer tools . If you need very bare implementation, you should go for lucidrains/PaLM-rlhf-pytorch.

• musiclm-pytorch

  6 3,013 5.3 Python

  Implementation of MusicLM, Google's new SOTA model for music generation using attention networks, in Pytorch

Project mention: Training musiclm | /r/singularity | 2023-06-01

Has anyone tried to train this model : lucidrains/musiclm-pytorch: Implementation of MusicLM, Google's new SOTA model for music generation using attention networks, in Pytorch (github.com) ? Could you provide any useful resources that can help me? Or share your process?

• InfluxDB

  www.influxdata.com featured

  Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.

  

• audiolm-pytorch

  4 2,249 9.0 Python

  Implementation of AudioLM, a SOTA Language Modeling Approach to Audio Generation out of Google Research, in Pytorch

Project mention: Bark: A transformer based text to audio system | news.ycombinator.com | 2023-05-14

It’s mostly there in https://github.com/lucidrains/audiolm-pytorch#hierarchical-t....

• toolformer-pytorch

  2 1,889 7.2 Python

  Implementation of Toolformer, Language Models That Can Use Tools, by MetaAI

• make-a-video-pytorch

  6 1,840 4.8 Python

  Implementation of Make-A-Video, new SOTA text to video generator from Meta AI, in Pytorch

• muse-maskgit-pytorch

  5 816 5.6 Python

  Implementation of Muse: Text-to-Image Generation via Masked Generative Transformers, in Pytorch

Project mention: Google's StyleDrop can transfer style from a single image | /r/StableDiffusion | 2023-06-03

If google doesnt, someone like lucidrains probably would implement it, just like he did for imagen and muse.

• phenaki-pytorch

  2 715 3.6 Python

  Implementation of Phenaki Video, which uses Mask GIT to produce text guided videos of up to 2 minutes in length, in Pytorch

• SaaSHub

  www.saashub.com featured

  SaaSHub - Software Alternatives and Reviews. SaaSHub helps you find the best software and product alternatives

  

• LongNet

  16 652 9.0 Python

  Implementation of plug in and play Attention from "LongNet: Scaling Transformers to 1,000,000,000 Tokens"

Project mention: LongLlama | /r/LocalLLaMA | 2023-07-07

If you want to talk immature looking, longnet wouldn't even compile. That's a big oof, considering it's a python and usually nonworking code is good enough to generate byte code. (also it has hard-coded dtype and device)

• MEGABYTE-pytorch

  1 590 5.8 Python

  Implementation of MEGABYTE, Predicting Million-byte Sequences with Multiscale Transformers, in Pytorch

Project mention: [R] MEGABYTE: Predicting Million-byte Sequences with Multiscale Transformers | /r/MachineLearning | 2023-05-15

• recurrent-memory-transformer-pytorch

  1 380 7.4 Python

  Implementation of Recurrent Memory Transformer, Neurips 2022 paper, in Pytorch

• iTransformer

  1 334 8.1 Python

  Unofficial implementation of iTransformer - SOTA Time Series Forecasting using Attention networks, out of Tsinghua / Ant group

Project mention: Implementation of iTransformer – SOTA Time Series Forecasting Attention Networks | news.ycombinator.com | 2023-10-13

• q-transformer

  1 284 9.0 Python

  Implementation of Q-Transformer, Scalable Offline Reinforcement Learning via Autoregressive Q-Functions, out of Google Deepmind

Project mention: Q-Transformer | news.ycombinator.com | 2023-11-30

Being implemented as we speak, by the always impressive LucidRains [1]

[1]: https://github.com/lucidrains/q-transformer

• block-recurrent-transformer-pytorch

  1 204 5.0 Python

  Implementation of Block Recurrent Transformer - Pytorch

• flash-attention-jax

  1 175 2.0 Python

  Implementation of Flash Attention in Jax

• SaaSHub

  www.saashub.com featured

  SaaSHub - Software Alternatives and Reviews. SaaSHub helps you find the best software and product alternatives

  

Python attention-mechanisms related posts

• LongLlama

  2 projects | /r/LocalLLaMA | 7 Jul 2023

• Which features you wish that were added to Character Ai?

  1 project | /r/CharacterAI | 7 Jul 2023

• Why AI will not replace programmers.

  2 projects | /r/ChatGPT | 7 Jul 2023

• An open model that beats ChatGPT. We're seeing a real shift towards open source models that will accelerate in the coming weeks.

  1 project | /r/aipromptprogramming | 6 Jul 2023

• GitHub - kyegomez/LongNet: Implementation of plug in and play Attention from "LongNet: Scaling Transformers to 1,000,000,000 Tokens"

  1 project | /r/AITechTips | 6 Jul 2023

• GitHub - kyegomez/LongNet: Implementation of plug in and play Attention from "LongNet: Scaling Transformers to 1,000,000,000 Tokens"

  1 project | /r/mlscaling | 6 Jul 2023

• GitHub - kyegomez/LongNet: Implementation of plug in and play Attention from "LongNet: Scaling Transformers to 1,000,000,000 Tokens"

  1 project | /r/machinelearningnews | 6 Jul 2023
• A note from our sponsor - InfluxDB
  www.influxdata.com | 4 May 2024

  Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality. Learn more →

Index

Sponsored

SaaSHub - Software Alternatives and Reviews

SaaSHub helps you find the best software and product alternatives

www.saashub.com

Do not miss the trending Python projects with our weekly report!