pytorch-lightning VS lightning-hydra-template

It's very easy to get started, right in your Terminal, no fees! No credit card at all.

And there are cloud providers like https://replicate.com/ and https://lightning.ai/ that will let you use your LLM via an API key just like you did with OpenAI if you need that.

You don't need OpenAI - nobody does.

https://see.stanford.edu/Course/CS229 https://lightning.ai/ https://www.youtube.com/watch?v=00s9ireCnCw&t=57s https://towardsdatascience.com/

I've been wondering on what is the recommended method of saving logits/activations using PyTorch Lightning. I've looked at Callbacks, Loggers and ModelHooks but none of the use-cases seem to be for this kind of activity (even if I were to create my own custom variants of each utility). The ModelCheckpoint Callback in its utility makes me feel like custom Callbacks would be the way to go but I'm not quite sure. This closed GitHub issue does address my issue to some extent.

Our codebase for the diffusion models builds heavily on OpenAI's ADM codebase , lucidrains, Stable Diffusion, Lightning and Hugging Face. Thanks for open-sourcing!

For lightweight use, pytorch-lightning is too heavy, and its source code will be very difficult for beginners to read, at least for me.

For the A100 GPU, theoretical performance is the same for FP16/BF16 and both rely on the same number of bits, meaning memory should be the same. However since it's quite newly added to PyTorch, performance seems to still be dependent on underlying operators used (pytorch lightning debugging in progress here).

I've been trying to learn PyTorch Lightning and Hydra in order to use/create my own custom deep learning template (e.g. like this) as it would greatly help with my research workflow. A lot of the work I do requires me to analyse metrics based on the logits/activations of the model.

I can't say for sure whether it is the best combination for research in the long run, but if you do go down that route I have found this template very useful

lightning-hydra-template

This lightning + hydra template is quite complete. Great for learning best practices.

A good example is this project template https://github.com/ashleve/lightning-hydra-template. It uses a lot of cool things such as

and many more! (checkout the #Your Superpowers section of the readme)

I write a LightningDatamodule. I found it to be an intuitive way to encapsulate any dataset. LightningDatamodule is a simple abstraction providing methods for data download, split, transforms and exposing dataloaders. Would love to see more researchers try out this concept, even in projects which don't use pytorch lightning. Reading LightningDatamodule makes me immedietely see how the dataset is prepared, while it seems like most data science projects throw around data logic across different parts of the pipeline, making it hard to understand what's going on. You can see example of such datamodule here

I feel like most ML people don't use those tools because they simply don't realize all the advantages (especially Hydra seems like a very useful addition to any deep learning project). I focused on structuring the readme in a way, which (I hope) will give you a quick overview - my hope is it can help to spread the word about those frameworks in a broaded community. It incorporates best practices and tricks I gathered over the last couple of months of playing around with it.