vit-pytorch VS Compact-Transformers

I also need to learn Pyspark so right now I am going to download the Fashion Mnist dataset, use Pyspark to downsize each image and put the into separate folders according to their labels (just to show employers I can do some basic ETL with Pyspark, not sure how I am going to load for training in Pytorch yet though). Then I am going to write the simplest Le Net to try to categorize the fashion MNIST dataset (results will most likely be bad but it's okay). Next, try to learn transfer learning in Pytorch for both CNN or maybe skip ahead to ViT. Ideally at this point I want to study the Attention mechanism a bit more and try to implement Simple Vit which I saw here: https://github.com/lucidrains/vit-pytorch/blob/main/vit_pytorch/simple_vit.py

For code implementation, check out this git repo. It contains fairly straightforward PyTorch implementations for various ViT papers with references.

For ViT’s specifically, there’s been a good amount of research trying to extend ViT’s to work on small datasets without a large amount of pre-training (which comes with its own host of issues such as the best way to fine tune such a huge model). One paper which comes to mind is ViT’s for small datasets (https://arxiv.org/abs/2112.13492), which has an implementation in lucidrain’s repo here: https://github.com/lucidrains/vit-pytorch

Here's a pytorch implementation of ViT https://github.com/lucidrains/vit-pytorch

looks great Junho! i've linked to it from https://github.com/lucidrains/vit-pytorch like you asked :)

Sure thing. Also if you're getting into transformers I'd recommend lucidrains's GitHub[0] since it has a large collection of them with links to papers. It's nice that things are consolidated.

[0] https://github.com/lucidrains/vit-pytorch

nah, it is really simple. here is the code https://github.com/lucidrains/vit-pytorch/blob/main/vit_pytorch/mae.py

lucid rains VI has all of what you may need in a clean API

I've attempted it here https://github.com/lucidrains/vit-pytorch/blob/main/vit_pytorch/rvt.py but those who have tried it haven't seen knock out results as 1d. Perhaps the axial lengths are too small to see a benefit

Yeah so I can get how that might be confusing. Sometimes code is clearer. So in the vanilla transformer you do a patch and then embed operation, right? A quick way to do that is actually with non-overlapping convolutions. Your strides are the same size as your kernel sizes. Look closely at Figure 2 (you can also see a visual representation in Figure 1 but I'll admit there is some artistic liberty there because we wanted to stress the combined patch and embed operation. Those are real outputs though. But basically yeah, change the stride so you overlap. Those create patches, then you embed. So we don't really call it a hybrid the same way you may call a 1x1 cov a channel wise linear.

ViT https://github.com/SHI-Labs/Compact-Transformers/blob/main/s...

CCT: https://github.com/SHI-Labs/Compact-Transformers/blob/main/s...

After looking around for this type of architecture, it looks someone had this idea before me: https://github.com/SHI-Labs/Compact-Transformers

> Are transformers competitive with (for example) CNNs on vision-related tasks when there's less data available?

Typically no, but in some cases yes. Pure vision transformers suffer from too much global inductive bias and not enough local inductive bias. For vision problems LIB tends to be a pretty important part to learning vision problems. That's what made convolutions helpful in the first place. But the good news is that you don't need much. CCT[0] showed that early convs (1 or 2) was enough to get good performance. In fact, CCT gets ResNet50 level performance on CIFAR-10 with 15% the number of parameters[1]. This was done _without_ pre-training, which is what most papers with transformers do when reporting CIFAR numbers. We also have SOTA on Flowers102 which is a small dataset (~6.5k) but larger images (99.76% with pre-training, 97.19% without). So we can definitely have transformers work with datasets on the order of magnitude you're talking about. But keep in mind that transformers still like a lot of data augmentation.

> I'm in an industry (building energy consumption prediction) where we can only generate around 10,000 to 100,000 datapoints (from simulation engines) for DL. Are transformers ever used with that scale of data?

In short, yes. Feel free to open issues on our GH if you experience problems[2]. We've been trying to help people use our network for various types of problems.

Disclosure: I'm one of the lead authors on CCT.

[0] https://arxiv.org/abs/2104.05704

[1] https://paperswithcode.com/sota/image-classification-on-cifa...

[2] https://github.com/SHI-Labs/Compact-Transformers