memory-efficient-attention-pytorch VS Compact-Transformers

Compare memory-efficient-attention-pytorch vs Compact-Transformers and see what are their differences.

memory-efficient-attention-pytorch

Implementation of a memory efficient multi-head attention as proposed in the paper, "Self-attention Does Not Need O(n²) Memory" (by lucidrains)

Compact-Transformers

Escaping the Big Data Paradigm with Compact Transformers, 2021 (Train your Vision Transformers in 30 mins on CIFAR-10 with a single GPU!) (by SHI-Labs)
CodeRabbit: AI Code Reviews for Developers
Revolutionize your code reviews with AI. CodeRabbit offers PR summaries, code walkthroughs, 1-click suggestions, and AST-based analysis. Boost productivity and code quality across all major languages with each PR.
coderabbit.ai
featured
SaaSHub - Software Alternatives and Reviews
SaaSHub helps you find the best software and product alternatives
www.saashub.com
featured
memory-efficient-attention-pytorch Compact-Transformers
2 4
227 504
- 3.4%
6.1 3.4
over 1 year ago about 1 month ago
Python Python
MIT License Apache License 2.0
The number of mentions indicates the total number of mentions that we've tracked plus the number of user suggested alternatives.
Stars - the number of stars that a project has on GitHub. Growth - month over month growth in stars.
Activity is a relative number indicating how actively a project is being developed. Recent commits have higher weight than older ones.
For example, an activity of 9.0 indicates that a project is amongst the top 10% of the most actively developed projects that we are tracking.

memory-efficient-attention-pytorch

Posts with mentions or reviews of memory-efficient-attention-pytorch. We have used some of these posts to build our list of alternatives and similar projects. The last one was on 2022-06-09.

Compact-Transformers

Posts with mentions or reviews of Compact-Transformers. We have used some of these posts to build our list of alternatives and similar projects. The last one was on 2023-12-01.
  • Deep Dive into the Vision Transformers Paper (ViT)
    3 projects | news.ycombinator.com | 1 Dec 2023
    Yeah so I can get how that might be confusing. Sometimes code is clearer. So in the vanilla transformer you do a patch and then embed operation, right? A quick way to do that is actually with non-overlapping convolutions. Your strides are the same size as your kernel sizes. Look closely at Figure 2 (you can also see a visual representation in Figure 1 but I'll admit there is some artistic liberty there because we wanted to stress the combined patch and embed operation. Those are real outputs though. But basically yeah, change the stride so you overlap. Those create patches, then you embed. So we don't really call it a hybrid the same way you may call a 1x1 cov a channel wise linear.

    ViT https://github.com/SHI-Labs/Compact-Transformers/blob/main/s...

    CCT: https://github.com/SHI-Labs/Compact-Transformers/blob/main/s...

  • [D] Different input image size when using Visual Transformers
    2 projects | /r/MachineLearning | 26 May 2022
    After looking around for this type of architecture, it looks someone had this idea before me: https://github.com/SHI-Labs/Compact-Transformers
  • Will Transformers Take over Artificial Intelligence?
    5 projects | news.ycombinator.com | 10 Mar 2022
    > Are transformers competitive with (for example) CNNs on vision-related tasks when there's less data available?

    Typically no, but in some cases yes. Pure vision transformers suffer from too much global inductive bias and not enough local inductive bias. For vision problems LIB tends to be a pretty important part to learning vision problems. That's what made convolutions helpful in the first place. But the good news is that you don't need much. CCT[0] showed that early convs (1 or 2) was enough to get good performance. In fact, CCT gets ResNet50 level performance on CIFAR-10 with 15% the number of parameters[1]. This was done _without_ pre-training, which is what most papers with transformers do when reporting CIFAR numbers. We also have SOTA on Flowers102 which is a small dataset (~6.5k) but larger images (99.76% with pre-training, 97.19% without). So we can definitely have transformers work with datasets on the order of magnitude you're talking about. But keep in mind that transformers still like a lot of data augmentation.

    > I'm in an industry (building energy consumption prediction) where we can only generate around 10,000 to 100,000 datapoints (from simulation engines) for DL. Are transformers ever used with that scale of data?

    In short, yes. Feel free to open issues on our GH if you experience problems[2]. We've been trying to help people use our network for various types of problems.

    Disclosure: I'm one of the lead authors on CCT.

    [0] https://arxiv.org/abs/2104.05704

    [1] https://paperswithcode.com/sota/image-classification-on-cifa...

    [2] https://github.com/SHI-Labs/Compact-Transformers

What are some alternatives?

When comparing memory-efficient-attention-pytorch and Compact-Transformers you can also consider the following projects:

flash-attention - Fast and memory-efficient exact attention

vit-pytorch - Implementation of Vision Transformer, a simple way to achieve SOTA in vision classification with only a single transformer encoder, in Pytorch

performer-pytorch - An implementation of Performer, a linear attention-based transformer, in Pytorch

hlb-CIFAR10 - Train CIFAR-10 in <7 seconds on an A100, the current world record.

x-transformers - A concise but complete full-attention transformer with a set of promising experimental features from various papers

routing-transformer - Fully featured implementation of Routing Transformer

memory-efficient-attention-pyt

mae - PyTorch implementation of MAE https//arxiv.org/abs/2111.06377

DALLE-pytorch - Implementation / replication of DALL-E, OpenAI's Text to Image Transformer, in Pytorch

CodeRabbit: AI Code Reviews for Developers
Revolutionize your code reviews with AI. CodeRabbit offers PR summaries, code walkthroughs, 1-click suggestions, and AST-based analysis. Boost productivity and code quality across all major languages with each PR.
coderabbit.ai
featured
SaaSHub - Software Alternatives and Reviews
SaaSHub helps you find the best software and product alternatives
www.saashub.com
featured

Did you konow that Python is
the 2nd most popular programming language
based on number of metions?