hlb-CIFAR10 VS cifar10-fast-simple

A training speed project building on https://github.com/tysam-code/hlb-CIFAR10 to reach faster times

Logged into my personal account for this one! I'm a lead author on a paper that explored exactly. It does enable faster training and smaller model sizes. For reference, you can get 80% accuracy on CIFAR-10 in ~30 minutes of CPU (not using crazy optimizations). There are open questions about scaling but at the time we did not have access to big compute (really still don't) and our goals were focused on addressing the original ViT's claims of data constraints and necessities of pretraining for smaller datasets (spoiler, augmentation + overlapping patches plays a huge role). Basically we wanted to make a network that allowed people to train transformers from scratch for their data projects because pretrained models aren't always the best solutions or practical.

Paper: https://arxiv.org/abs/2104.05704

Blog: https://medium.com/pytorch/training-compact-transformers-fro...

CPU compute: https://twitter.com/WaltonStevenj/status/1382045610283397120

Crazy optimizations (no affiliation): 94% on CIFAR-10 in <6.3 seconds on a single A100 : https://github.com/tysam-code/hlb-CIFAR10

I also want to give maybe some better information about ViTs in general. Lucas Beyer is a good source and has some lectures as well as Hila Chefer and Sayak Paul's tutorials.

Lucas Beyer: https://twitter.com/giffmana/status/1570152923233144832

Chefer & Paul's All Things ViT: https://all-things-vits.github.io/atv/

If you'd like to play around with MNIST yourself, I wrote a PyTorch training implementation that gets ~95.45%+ in <13.6 seconds on a V100, est. < 6.5 seconds on an A100. Made to be edited/run in Colab: https://github.com/tysam-code/hlb-CIFAR10

It's originally kitted for CIFAR10, but I've found the parameters to be quite general. The code is very easy to read and well-commented, and is a great starting place for exploration.

Min-cut deltas to run MNIST:

`.datasets.CIFAR10('` -> `.datasets.MNIST('` (both occurences)

Sure. Basically everything in https://github.com/tysam-code/hlb-CIFAR10 was directly founded on the concepts in the paper, down to the coding, commenting, and layout styles (hence why I advocate so strongly for it as a requirement for ML. The empirical benefits are clear to me).

Before I sat down and wrote my first line, I spent a very long time thinking about how to optimize the repo. Not just in terms of information flow during training, but how the code was laid out (minimize the expected value of deltas for changes from a superset of possible code changes), and comments (ratio of space vs mental effort to decode the repo for experienced vs inexperienced developers).

It's not perfect, but I've used info theory as a strong guiding light for that repo. There's more to say here, but it's a long conversation about the expected utility of doing research a few different kinds of ways.

I think this is a good casual introduction to the marketplace dynamics of how ML will impact the market. I do, however, disagree as this version of things assumes a more open-information set of competitive strategies among potentially ideal agents from a game theoretic perspective, and we can see this is absolutely not the case 'in real life'. To one of his examples -- Exxon-Mobil.

An updated version: There will be a log-normally distributed set of winners and losers from the exponential effects of ML and 'AI', and the flatness of this curve will be almost entirely solely determined by the governance of the various countries in the world over different economic and/or informational policies. Other than that, the information asymmetry is going to make it a power-bloodbath as we go through our informational-industrial revolution.

While I'm here, I think Hotz does contribute a lot of good to the field, though I do have a bit of a minor personal beef with him. He said he was going to reimplement https://github.com/tysam-code/hlb-CIFAR10 in tinygrad, bashed a few parts of the code for a while on stream, and then gave up a few hours later because of the empirical speed/occupancy numbers. >:( I want my fast reimplementation, George.

As a practitioner specializing in extremely fast-training neural networks, seeing a paper in 2023 considering fp32 as a gold standard over pure non-mixed fp16/bp16 is a bit shocking to me and feels dated/distracting from the discussion. They make good points but unless I am hopelessly misinformed, it's been pretty well established at this point in a number of circles that fp32 is overkill for the majority of uses for many modern-day practitioners. Loads of networks train directly in bfloat16 as the standard -- a lot of the modern LLMs among them. Mixed precision is very much no longer needed, not even with fp16 if you're willing to tolerate some range hacks. If you don't want the range hacks, just use bfloat16 directly. The complexity is not worth it, adds not much at all, and the dynamic loss scaler a lot of people use is just begging for more issues.

Both of the main repos that I've published in terms of speed benchmarks train directly in pure fp16 and bf16 respectively without any fp32 frippery, if you want to see an example of both paradigms successfully feel free to take a look (I'll note that bf16 is simpler on the whole for a few reasons, generally seamless): https://github.com/tysam-code/hlb-CIFAR10 [for fp16] and https://github.com/tysam-code/hlb-gpt [for bf16]

Personally from my experience, I think fp16/bf16 is honestly a bit too expressive for what we need, fp8 seems to do just fine and I think will be quite alright with some accommodations, just as with pure fp16. The what and the how of that is a story for a different day (and at this point, the max pooling operation is basically one of the slowest now).

You'll have to excuse my frustration a bit, it just is a bit jarring to see a streetsign from way in the past fly forward in the wind to hit you in the face before tumbling on its merry way. And additionally in the comment section the general discussion doesn't seem to talk about what seems to be a pretty clearly-established consensus in certain research circles. It's not really too much of a debate anymore, it works and we're off to bigger and better problems that I think we should talk about. I guess in one sense it does justify the paper's utility, but also a bit frustrating because it normalizes the conversation as a few notches back from where I personally feel that it actually is at the moment.

We've got to move out of the past, this fp32 business to me personally is like writing a Relu-activated VGG network in Keras on Tensorflow. Phew.

And while we're at it, if I shall throw my frumpy-grumpy hat right back into the ring, this is an information-theoretic problem! Not enough discussion of Shannon and co. Let's please fix that too. See my other rants for x-references to that, should you be so-inclined to punish yourself in that manner.

I really love small neural networks. They have some nice properties that people overlook. The training speed record (warning, self promo) for CIFAR10 to 94% uses a very tiny neural network (<10 MB if just saved raw out to disk as a definition file). That's located at https://github.com/tysam-code/hlb-CIFAR10.

You could make that even smaller if you wanted to, though at least this network is already pushing maybe even a little further down the diminishing returns spectrum in some areas than I'd like.

I think a really fun challenge would be to find the fastest network that infers at 94% in under 1 MB. I certainly believe it's possible, but with pareto laws the way they are, it would take a whole lot longer to train and might not be as fast on a GPU during inference as the main net (despite having fewer parameters). That might not be true, however.

There's a few NP-hard problems that actually exist in this space that not a lot of people talk about but I feel will be considered a core part of the theory of training neural networks at some point in the future. The size of the network is a very interesting tradeoff that opens up certain mathematically interesting properties on either end of the spectrum. Bigger is not always better, though it is simpler and simple oftentimes survives.

One of the common threads (might be a "common", I'm not sure to be honest as I live in my own personal bubble of research interests and community and etc) is the dimensionality of the problem at hand. That plays into the scale of the network used to solve a problem. I remember some discussion being sparked a while back from some Uber research about the inherent dimensionality of a neural network on a particular problem (though of course it's naturally linked to your inductive bias so please take that as you will). As you noted, some networks do quite well with very few neurons, 15 is a record however from what I've heard (and I'd love to see that -- I have a guess as to which particular method, or, at least, method family, it is... ;P I'm...casually interested in that arena of research).

In any case, as you can see I am quite interested and passionate about this topic and am happy to discuss it at length further.

In a related but different vein (w/ hardcoded hyperparameters), if you'd like to have a research toolbench that trains rapidly on CIFAR10 (94% in <7 seconds on an A100), I made https://github.com/tysam-code/hlb-CIFAR10. It's also very breadboard-ized, for lack of a better term, so you can reclone and hack stuff in quickly to see if it works or doesn't. Most things I tested took 5 minutes or less, some a few seconds, and just a few more involved ones maybe half an hour to an hour or so, maybe a little more or less with debugging (depending upon how involved it was). I'm definitely curious about the software in this post though, as there was a lot of painful tuning involved (the reward space is, er, quite noisy).

Karpathy's zero to hero series is excellent, and I really recommend it.

I also made a few repos that are geared around readability and being a good 'working code demonstration' of certain best-practices in neural networks. If you're like me and you grok code better than symbols, this could be a helpful adjunct as well if you're wanting to dig deep a bit.

https://github.com/tysam-code/hlb-CIFAR10

Great point! If I recall correctly, this team (well, nearly all the top teams from DawnBench) took Page's code and wrestled it into the multi-GPU realm. I'm a sucker for simplicity, as much as is reasonable (this codebase currently does not use JIT or any custom kernels! (!!!)), and also making sure that the average practitioner (like me) could do something workable without having to pay tons of money. My computing costs are $50 a month currently, i.e. the cost of Pro Colab. And we were able to break the single-GPU WR, and we're really close to pushing past any of the official multi-GPU submissions (old as they may be!).

I took David's work in a different direction and just kept it true I think to the original spirit of things. Cycle times for experimentation are king in ML when it comes to the speed of research progress, regardless of what anyone else might tell you. Having tons of hardware may be really flashy and useful for the end product, but it's certainly not needed for much of the lo-fi, day-to-day stuff.

That said, the A100 is definitely a step up. It is under 2x, though, as we are basically only memory-and-slow-backprop-kernel limited now, not as much by the convolutions (which now are among the shorter operations). Running https://github.com/99991/cifar10-fast-simple on my end gave me 17.2 seconds, vs the 24 seconds that Dave reported on the V100 (though the lovely author of that repo, @99991, was able to get faster speeds on their personal A100 setup). So we're definitely in that weird regime where moving everything to massively scaled matrix multiplies when possible is preferred, and sometimes that's...tricky for a few of these operations.

Hello everyone,

After recreating the accuracy/rough speed from David Page's implementation in hlb-CIFAR10 0.1.0 (18.1s on an A100, SXM4, Colab), it was down to some basic NVIDIA kernel profiling to figure out which operations were the long poles in the tent. Perhaps (somewhat?) unsurprisingly, the NCHW <-> NHWC thrash was the worst part, but unfortunately the GhostBatchNorm was a barrier even using the faster-on-Ampere channels_last memory format.

A quick note before continuing -- some may find the use of a convolutional network and on CIFAR10 to be curious. A quick answer to that would be that in doing the research that optimizes well-known problems (especially if the testing path is incredibly rapid), we get much clearer pictures of what certain fundamental information learning limits are for systems like this, as well as stable prototypes that can then be translated (potentially somewhat analogously) into other modalities. You can see this practice with a few researchers, Hinton comes to mind though his work is much more fundamental and experimental than this is. Back to the release notes.

Ultimately, however, we were able to get a similar level of regularization to the original GhostBatchNorm (called GhostNorm) in the code, which allowed us to remove it and a bunch of tensor allocation/contiguous tensor calls, saving us nearly exactly 5 seconds or so (!!!!).

Replacing the call for nn.AdaptiveMaxPooling(1,1) with a torch.amax(dim=2,3), added an additional .5 seconds off the clock, bringing us down below Thomas Germer (@99991)'s excellently quick implementation of the same base method (https://github.com/99991/cifar10-fast-simple) and giving us the new world record.

This work is pretty simple on its own -- though the various ways to use the nvidia profiler(s) can be very daunting to use and I can post snippets of the simplest way that I've found (via the torch.profiler route) if someone asks/is curious. That said, looking at kernel execution order and times can really and truly do a lot to quickly improve a network in conjunction with good research engineering practices.

This is what I'm pretty good at doing so getting to flex a bit on a spare time project is fun. I'm consistently storing up time saves into a draft bin of sorts and plan on keeping releasing them in related/clustered releases as I'm able to appropriately polish them to whatever their capabilities seem to be. There is a lot of room to grow, and I think we now definitely have a good chance at making it within that 94% accuracy under ~<2s mark within a few years!

This work is meant to be a living resume for me, feel free to check out my README.md for more info. I love a lot of aspects of the technical/nitty gritty side of the fusion of neural network engineering and the edge of research, particularly when it comes to speed, so this is my strong area. I'm certainly happy to answer whatever reasonable questions anyone might have, let me help with getting this project going for you (or other related stuff -- feel free to ask! <3 :)))) )

Replacing the call for nn.AdaptiveMaxPooling(1,1) with a torch.amax(dim=2,3), added an additional .5 seconds off the clock, bringing us down below Thomas Germer (@99991)'s excellently quick implementation of the same base method (https://github.com/99991/cifar10-fast-simple) and giving us the new world record.