Python attention-mechanisms

Open-source Python projects categorized as attention-mechanisms
Edit details
Topics: #Artificial intelligence #Deep Learning #Transformers #Memory #Recurrence
Learn any GitHub repo in 59 seconds
Onboard AI learns any GitHub repo in minutes and lets you chat with it to locate functionality, understand different parts, and generate new code. Use it for free at www.getonboard.dev.
getonboard.dev
Sponsored

Top 14 Python attention-mechanism Projects

  • PaLM-rlhf-pytorch

    Implementation of RLHF (Reinforcement Learning with Human Feedback) on top of the PaLM architecture. Basically ChatGPT but with PaLM

    Project mention: How should I get an in-depth mathematical understanding of generative AI? | /r/datascience | 2023-05-18

    ChatGPT isn't open sourced so we don't know what the actual implementation is. I think you can read Open Assistant's source code for application design. If that is too much, try Open Chat Toolkit's source code for developer tools . If you need very bare implementation, you should go for lucidrains/PaLM-rlhf-pytorch.

  • musiclm-pytorch

    Implementation of MusicLM, Google's new SOTA model for music generation using attention networks, in Pytorch

    Project mention: Training musiclm | /r/singularity | 2023-06-01

    Has anyone tried to train this model : lucidrains/musiclm-pytorch: Implementation of MusicLM, Google's new SOTA model for music generation using attention networks, in Pytorch (github.com) ? Could you provide any useful resources that can help me? Or share your process?

  • Onboard AI

    Learn any GitHub repo in 59 seconds. Onboard AI learns any GitHub repo in minutes and lets you chat with it to locate functionality, understand different parts, and generate new code. Use it for free at www.getonboard.dev.

  • audiolm-pytorch

    Implementation of AudioLM, a SOTA Language Modeling Approach to Audio Generation out of Google Research, in Pytorch

    Project mention: Bark: A transformer based text to audio system | news.ycombinator.com | 2023-05-14

    It’s mostly there in https://github.com/lucidrains/audiolm-pytorch#hierarchical-t....

  • toolformer-pytorch

    Implementation of Toolformer, Language Models That Can Use Tools, by MetaAI

    Project mention: CMV: AI like Chat GPT is really not as big of a problem as everyone is making it out to be, and people are vastly overestimating how powerful it is even now. | /r/changemyview | 2023-02-18

    Now consider Toolformer by MetaAI, which presents a language model capable of using external tools by modifying the text with API calls to wolfram alpha/google/internal database/compiler.

  • make-a-video-pytorch

    Implementation of Make-A-Video, new SOTA text to video generator from Meta AI, in Pytorch

    Project mention: How do I get this Python machine learning source code file to run? | /r/learnpython | 2023-02-01

  • muse-maskgit-pytorch

    Implementation of Muse: Text-to-Image Generation via Masked Generative Transformers, in Pytorch

    Project mention: Google's StyleDrop can transfer style from a single image | /r/StableDiffusion | 2023-06-03

    If google doesnt, someone like lucidrains probably would implement it, just like he did for imagen and muse.

  • phenaki-pytorch

    Implementation of Phenaki Video, which uses Mask GIT to produce text guided videos of up to 2 minutes in length, in Pytorch

    Project mention: How much CUDA memory to train this text-to-video model? | /r/learnmachinelearning | 2023-01-02

  • InfluxDB

    Collect and Analyze Billions of Data Points in Real Time. Manage all types of time series data in a single, purpose-built database. Run at any scale in any environment in the cloud, on-premises, or at the edge.

  • LongNet

    Implementation of plug in and play Attention from "LongNet: Scaling Transformers to 1,000,000,000 Tokens"

    Project mention: LongLlama | /r/LocalLLaMA | 2023-07-07

    If you want to talk immature looking, longnet wouldn't even compile. That's a big oof, considering it's a python and usually nonworking code is good enough to generate byte code. (also it has hard-coded dtype and device)

  • MEGABYTE-pytorch

    Implementation of MEGABYTE, Predicting Million-byte Sequences with Multiscale Transformers, in Pytorch

    Project mention: [R] MEGABYTE: Predicting Million-byte Sequences with Multiscale Transformers | /r/MachineLearning | 2023-05-15

  • recurrent-memory-transformer-pytorch

    Implementation of Recurrent Memory Transformer, Neurips 2022 paper, in Pytorch

    Project mention: The race is on | /r/LocalLLaMA | 2023-04-24

    Of course, lucidrains already has an empty repo up. So that's worth keeping an eye on.

  • iTransformer

    Unofficial implementation of iTransformer - SOTA Time Series Forecasting using Attention networks, out of Tsinghua / Ant group

    Project mention: Implementation of iTransformer – SOTA Time Series Forecasting Attention Networks | news.ycombinator.com | 2023-10-13

  • q-transformer

    Implementation of Q-Transformer, Scalable Offline Reinforcement Learning via Autoregressive Q-Functions, out of Google Deepmind

    Project mention: Q-Transformer | news.ycombinator.com | 2023-11-30

    Being implemented as we speak, by the always impressive LucidRains [1]

    [1]: https://github.com/lucidrains/q-transformer

  • block-recurrent-transformer-pytorch

    Implementation of Block Recurrent Transformer - Pytorch

    Project mention: From Deep to Long Learning | news.ycombinator.com | 2023-04-09

    that line of research is still going. https://github.com/lucidrains/block-recurrent-transformer-py... i think it is worth continuing research on both fronts.

  • flash-attention-jax

    Implementation of Flash Attention in Jax

  • SaaSHub

    SaaSHub - Software Alternatives and Reviews. SaaSHub helps you find the best software and product alternatives

NOTE: The open source projects on this list are ordered by number of github stars. The number of mentions indicates repo mentiontions in the last 12 Months or since we started tracking (Dec 2020). The latest post mention was on 2023-11-30.

Python attention-mechanisms related posts

Index

What are some of the best open-source attention-mechanism projects in Python? This list will help you:

Project Stars
1 PaLM-rlhf-pytorch 7,411
2 musiclm-pytorch 2,808
3 audiolm-pytorch 2,040
4 toolformer-pytorch 1,766
5 make-a-video-pytorch 1,747
6 muse-maskgit-pytorch 743
7 phenaki-pytorch 675
8 LongNet 609
9 MEGABYTE-pytorch 528
10 recurrent-memory-transformer-pytorch 356
11 iTransformer 238
12 q-transformer 209
13 block-recurrent-transformer-pytorch 185
14 flash-attention-jax 149

Sponsored
SaaSHub - Software Alternatives and Reviews
SaaSHub helps you find the best software and product alternatives
www.saashub.com