Our great sponsors
-
memory-efficient-attention-pytorch
Discontinued Implementation of a memory efficient multi-head attention as proposed in the paper, "Self-attention Does Not Need O(n²) Memory"
-
WorkOS
The modern identity platform for B2B SaaS. The APIs are flexible and easy-to-use, supporting authentication, user identity, and complex enterprise features like SSO and SCIM provisioning.
-
Compact-Transformers
Escaping the Big Data Paradigm with Compact Transformers, 2021 (Train your Vision Transformers in 30 mins on CIFAR-10 with a single GPU!)
-
vit-pytorch
Implementation of Vision Transformer, a simple way to achieve SOTA in vision classification with only a single transformer encoder, in Pytorch
-
InfluxDB
Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.
I would recommend Routing Transformer https://github.com/lucidrains/routing-transformer but the real truth is nothing beats full attention. Luckily, someone recently figured out how to get past the memory bottleneck. https://github.com/lucidrains/memory-efficient-attention-pyt...
I would recommend Routing Transformer https://github.com/lucidrains/routing-transformer but the real truth is nothing beats full attention. Luckily, someone recently figured out how to get past the memory bottleneck. https://github.com/lucidrains/memory-efficient-attention-pyt...
I would recommend Routing Transformer https://github.com/lucidrains/routing-transformer but the real truth is nothing beats full attention. Luckily, someone recently figured out how to get past the memory bottleneck. https://github.com/lucidrains/memory-efficient-attention-pyt...
> Are transformers competitive with (for example) CNNs on vision-related tasks when there's less data available?
Typically no, but in some cases yes. Pure vision transformers suffer from too much global inductive bias and not enough local inductive bias. For vision problems LIB tends to be a pretty important part to learning vision problems. That's what made convolutions helpful in the first place. But the good news is that you don't need much. CCT[0] showed that early convs (1 or 2) was enough to get good performance. In fact, CCT gets ResNet50 level performance on CIFAR-10 with 15% the number of parameters[1]. This was done _without_ pre-training, which is what most papers with transformers do when reporting CIFAR numbers. We also have SOTA on Flowers102 which is a small dataset (~6.5k) but larger images (99.76% with pre-training, 97.19% without). So we can definitely have transformers work with datasets on the order of magnitude you're talking about. But keep in mind that transformers still like a lot of data augmentation.
> I'm in an industry (building energy consumption prediction) where we can only generate around 10,000 to 100,000 datapoints (from simulation engines) for DL. Are transformers ever used with that scale of data?
In short, yes. Feel free to open issues on our GH if you experience problems[2]. We've been trying to help people use our network for various types of problems.
Disclosure: I'm one of the lead authors on CCT.
[0] https://arxiv.org/abs/2104.05704
[1] https://paperswithcode.com/sota/image-classification-on-cifa...
[2] https://github.com/SHI-Labs/Compact-Transformers
Sure thing. Also if you're getting into transformers I'd recommend lucidrains's GitHub[0] since it has a large collection of them with links to papers. It's nice that things are consolidated.
[0] https://github.com/lucidrains/vit-pytorch
Related posts
- [R] Rotary Positional Embeddings - a new relative positional embedding for Transformers that significantly improves convergence (20-30%) and works for both regular and efficient attention
- x-transformers
- Is it easier to go from Pytorch to TF and Keras than the other way around?
- A single API call using almost the whole 32k context window costs around 2$.
- GPT-4 architecture: what we can deduce from research literature