flash-attention VS memory-efficient-attention-pytorch

Compare flash-attention vs memory-efficient-attention-pytorch and see what are their differences.

flash-attention

Fast and memory-efficient exact attention (by Dao-AILab)

memory-efficient-attention-pytorch

Implementation of a memory efficient multi-head attention as proposed in the paper, "Self-attention Does Not Need O(n²) Memory" (by lucidrains)
Our great sponsors
  • WorkOS - The modern identity platform for B2B SaaS
  • InfluxDB - Power Real-Time Data Analytics at Scale
  • SaaSHub - Software Alternatives and Reviews
flash-attention memory-efficient-attention-pytorch
25 2
10,263 227
8.8% -
9.4 6.1
3 days ago about 1 year ago
Python Python
BSD 3-clause "New" or "Revised" License MIT License
The number of mentions indicates the total number of mentions that we've tracked plus the number of user suggested alternatives.
Stars - the number of stars that a project has on GitHub. Growth - month over month growth in stars.
Activity is a relative number indicating how actively a project is being developed. Recent commits have higher weight than older ones.
For example, an activity of 9.0 indicates that a project is amongst the top 10% of the most actively developed projects that we are tracking.

flash-attention

Posts with mentions or reviews of flash-attention. We have used some of these posts to build our list of alternatives and similar projects. The last one was on 2023-12-10.

memory-efficient-attention-pytorch

Posts with mentions or reviews of memory-efficient-attention-pytorch. We have used some of these posts to build our list of alternatives and similar projects. The last one was on 2022-06-09.

What are some alternatives?

When comparing flash-attention and memory-efficient-attention-pytorch you can also consider the following projects:

xformers - Hackable and optimized Transformers building blocks, supporting a composable construction.

TensorRT - NVIDIA® TensorRT™ is an SDK for high-performance deep learning inference on NVIDIA GPUs. This repository contains the open source components of TensorRT.

DeepSpeed - DeepSpeed is a deep learning optimization library that makes distributed training and inference easy, efficient, and effective.

RWKV-LM - RWKV is an RNN with transformer-level LLM performance. It can be directly trained like a GPT (parallelizable). So it's combining the best of RNN and transformer - great performance, fast inference, saves VRAM, fast training, "infinite" ctx_len, and free sentence embedding.

alpaca_lora_4bit

performer-pytorch - An implementation of Performer, a linear attention-based transformer, in Pytorch

x-transformers - A simple but complete full-attention transformer with a set of promising experimental features from various papers

XMem - [ECCV 2022] XMem: Long-Term Video Object Segmentation with an Atkinson-Shiffrin Memory Model

StableLM - StableLM: Stability AI Language Models

RWKV-v2-RNN-Pile - RWKV-v2-RNN trained on the Pile. See https://github.com/BlinkDL/RWKV-LM for details.

quality

memory-efficient-attention-pyt