Compact-Transformers VS hlb-CIFAR10

Yeah so I can get how that might be confusing. Sometimes code is clearer. So in the vanilla transformer you do a patch and then embed operation, right? A quick way to do that is actually with non-overlapping convolutions. Your strides are the same size as your kernel sizes. Look closely at Figure 2 (you can also see a visual representation in Figure 1 but I'll admit there is some artistic liberty there because we wanted to stress the combined patch and embed operation. Those are real outputs though. But basically yeah, change the stride so you overlap. Those create patches, then you embed. So we don't really call it a hybrid the same way you may call a 1x1 cov a channel wise linear.

ViT https://github.com/SHI-Labs/Compact-Transformers/blob/main/s...

CCT: https://github.com/SHI-Labs/Compact-Transformers/blob/main/s...

After looking around for this type of architecture, it looks someone had this idea before me: https://github.com/SHI-Labs/Compact-Transformers

> Are transformers competitive with (for example) CNNs on vision-related tasks when there's less data available?

Typically no, but in some cases yes. Pure vision transformers suffer from too much global inductive bias and not enough local inductive bias. For vision problems LIB tends to be a pretty important part to learning vision problems. That's what made convolutions helpful in the first place. But the good news is that you don't need much. CCT[0] showed that early convs (1 or 2) was enough to get good performance. In fact, CCT gets ResNet50 level performance on CIFAR-10 with 15% the number of parameters[1]. This was done _without_ pre-training, which is what most papers with transformers do when reporting CIFAR numbers. We also have SOTA on Flowers102 which is a small dataset (~6.5k) but larger images (99.76% with pre-training, 97.19% without). So we can definitely have transformers work with datasets on the order of magnitude you're talking about. But keep in mind that transformers still like a lot of data augmentation.

> I'm in an industry (building energy consumption prediction) where we can only generate around 10,000 to 100,000 datapoints (from simulation engines) for DL. Are transformers ever used with that scale of data?

In short, yes. Feel free to open issues on our GH if you experience problems[2]. We've been trying to help people use our network for various types of problems.

Disclosure: I'm one of the lead authors on CCT.

[0] https://arxiv.org/abs/2104.05704

[1] https://paperswithcode.com/sota/image-classification-on-cifa...

[2] https://github.com/SHI-Labs/Compact-Transformers

A training speed project building on https://github.com/tysam-code/hlb-CIFAR10 to reach faster times

Logged into my personal account for this one! I'm a lead author on a paper that explored exactly. It does enable faster training and smaller model sizes. For reference, you can get 80% accuracy on CIFAR-10 in ~30 minutes of CPU (not using crazy optimizations). There are open questions about scaling but at the time we did not have access to big compute (really still don't) and our goals were focused on addressing the original ViT's claims of data constraints and necessities of pretraining for smaller datasets (spoiler, augmentation + overlapping patches plays a huge role). Basically we wanted to make a network that allowed people to train transformers from scratch for their data projects because pretrained models aren't always the best solutions or practical.

Paper: https://arxiv.org/abs/2104.05704

Blog: https://medium.com/pytorch/training-compact-transformers-fro...

CPU compute: https://twitter.com/WaltonStevenj/status/1382045610283397120

Crazy optimizations (no affiliation): 94% on CIFAR-10 in <6.3 seconds on a single A100 : https://github.com/tysam-code/hlb-CIFAR10

I also want to give maybe some better information about ViTs in general. Lucas Beyer is a good source and has some lectures as well as Hila Chefer and Sayak Paul's tutorials.

Lucas Beyer: https://twitter.com/giffmana/status/1570152923233144832

Chefer & Paul's All Things ViT: https://all-things-vits.github.io/atv/

If you'd like to play around with MNIST yourself, I wrote a PyTorch training implementation that gets ~95.45%+ in <13.6 seconds on a V100, est. < 6.5 seconds on an A100. Made to be edited/run in Colab: https://github.com/tysam-code/hlb-CIFAR10

It's originally kitted for CIFAR10, but I've found the parameters to be quite general. The code is very easy to read and well-commented, and is a great starting place for exploration.

Min-cut deltas to run MNIST:

`.datasets.CIFAR10('` -> `.datasets.MNIST('` (both occurences)

Sure. Basically everything in https://github.com/tysam-code/hlb-CIFAR10 was directly founded on the concepts in the paper, down to the coding, commenting, and layout styles (hence why I advocate so strongly for it as a requirement for ML. The empirical benefits are clear to me).

Before I sat down and wrote my first line, I spent a very long time thinking about how to optimize the repo. Not just in terms of information flow during training, but how the code was laid out (minimize the expected value of deltas for changes from a superset of possible code changes), and comments (ratio of space vs mental effort to decode the repo for experienced vs inexperienced developers).

It's not perfect, but I've used info theory as a strong guiding light for that repo. There's more to say here, but it's a long conversation about the expected utility of doing research a few different kinds of ways.

I think this is a good casual introduction to the marketplace dynamics of how ML will impact the market. I do, however, disagree as this version of things assumes a more open-information set of competitive strategies among potentially ideal agents from a game theoretic perspective, and we can see this is absolutely not the case 'in real life'. To one of his examples -- Exxon-Mobil.

An updated version: There will be a log-normally distributed set of winners and losers from the exponential effects of ML and 'AI', and the flatness of this curve will be almost entirely solely determined by the governance of the various countries in the world over different economic and/or informational policies. Other than that, the information asymmetry is going to make it a power-bloodbath as we go through our informational-industrial revolution.

While I'm here, I think Hotz does contribute a lot of good to the field, though I do have a bit of a minor personal beef with him. He said he was going to reimplement https://github.com/tysam-code/hlb-CIFAR10 in tinygrad, bashed a few parts of the code for a while on stream, and then gave up a few hours later because of the empirical speed/occupancy numbers. >:( I want my fast reimplementation, George.

In a related but different vein (w/ hardcoded hyperparameters), if you'd like to have a research toolbench that trains rapidly on CIFAR10 (94% in <7 seconds on an A100), I made https://github.com/tysam-code/hlb-CIFAR10. It's also very breadboard-ized, for lack of a better term, so you can reclone and hack stuff in quickly to see if it works or doesn't. Most things I tested took 5 minutes or less, some a few seconds, and just a few more involved ones maybe half an hour to an hour or so, maybe a little more or less with debugging (depending upon how involved it was). I'm definitely curious about the software in this post though, as there was a lot of painful tuning involved (the reward space is, er, quite noisy).

Karpathy's zero to hero series is excellent, and I really recommend it.

I also made a few repos that are geared around readability and being a good 'working code demonstration' of certain best-practices in neural networks. If you're like me and you grok code better than symbols, this could be a helpful adjunct as well if you're wanting to dig deep a bit.

https://github.com/tysam-code/hlb-CIFAR10