falkon VS pytorch-lightning

As a satisfied customer (thanks!), was about to recommend KeOps as well. It might also be worth looking into falkon which builds on KeOps and leverages Nystrom approximation and conjugate gradient optimisation to further scale kernel operations.

On the computational side, it is also important to note that kernel methods are now 100-1,000 faster than they were just three years ago. You may be interested by the KeOps library, which is to kernels and geometric ML what cuDNN is to convolutions. You could also have a look at GPyTorch and the Falkon solvers: the software bottlenecks that were holding back kernel methods are progressively being lifted. Million-scale datasets are now routinely processed in minutes/hours and billion-scale problems are starting to become tractable.

On this note, you may be interested by the KeOps library (which is to kernels/geometric ML what cuDNN is to CNNs) and the Falkon solvers: the software bottlenecks that were holding back kernel methods are progressively being lifted. Million-scale datasets are now routinely processed in minutes/hours and billion-scale problems are starting to become tractable. This opens up quite a few possibilities :-)

It's very easy to get started, right in your Terminal, no fees! No credit card at all.

And there are cloud providers like https://replicate.com/ and https://lightning.ai/ that will let you use your LLM via an API key just like you did with OpenAI if you need that.

You don't need OpenAI - nobody does.

https://see.stanford.edu/Course/CS229 https://lightning.ai/ https://www.youtube.com/watch?v=00s9ireCnCw&t=57s https://towardsdatascience.com/

I've been wondering on what is the recommended method of saving logits/activations using PyTorch Lightning. I've looked at Callbacks, Loggers and ModelHooks but none of the use-cases seem to be for this kind of activity (even if I were to create my own custom variants of each utility). The ModelCheckpoint Callback in its utility makes me feel like custom Callbacks would be the way to go but I'm not quite sure. This closed GitHub issue does address my issue to some extent.

Our codebase for the diffusion models builds heavily on OpenAI's ADM codebase , lucidrains, Stable Diffusion, Lightning and Hugging Face. Thanks for open-sourcing!

For lightweight use, pytorch-lightning is too heavy, and its source code will be very difficult for beginners to read, at least for me.

For the A100 GPU, theoretical performance is the same for FP16/BF16 and both rely on the same number of bits, meaning memory should be the same. However since it's quite newly added to PyTorch, performance seems to still be dependent on underlying operators used (pytorch lightning debugging in progress here).