denoising-diffusion-pytorch VS pytorch-lightning

I’m trying to train a diffusion model using this code (https://github.com/lucidrains/denoising-diffusion-pytorch). My idea is to take a short audio segment, transform it into a spectrogram and train the model on these images then have it generate spectrograms then go back to audio. However the model requires square images. I cannot for the life of me figure out how to make a square spectrogram. Also is a regular spectrogram or a mel spectrogram better for this application?

Generally it's without weights, but MusicLM is also a WIP more mature implementations have descriptions on how to train them and follow ups on small scale/crowd-sourced experiments & research[1].

[1]: https://github.com/lucidrains/denoising-diffusion-pytorch

[1] https://colab.research.google.com/drive/1sjy9odlSSy0RBVgMTgP7s99NXsqglsUL?usp=sharing#scrollTo=KOYPSxPf_LL7 [2] https://github.com/lucidrains/denoising-diffusion-pytorch/blob/main/denoising_diffusion_pytorch/denoising_diffusion_pytorch.py

Sure. I am using: https://github.com/lucidrains/denoising-diffusion-pytorch

Lucidrains GitHub is always my go to repo for understandable paper implementations https://github.com/lucidrains/denoising-diffusion-pytorch

Thanks for the reply. Is there any suggestion if I wanted to train a model to generate half cat and half butterfly images what I should do? I git cloned the code from https://github.com/lucidrains/denoising-diffusion-pytorch and trained from scratch.

DDIM/DDPM are the same model to train, they only differ at inference time. To start I would recommend building from lucidrains' MIT licenced version (https://github.com/lucidrains/denoising-diffusion-pytorch). Just play around with the models until you gain an intuition.

Our codebase for the diffusion models builds heavily on OpenAI's ADM codebase , lucidrains, Stable Diffusion, Lightning and Hugging Face. Thanks for open-sourcing!

Once you understand these papers you can begin to understand Palette, and from there I would start with an open-source diffusion implementation like this one and then modify it to suit your needs!

https://see.stanford.edu/Course/CS229 https://lightning.ai/ https://www.youtube.com/watch?v=00s9ireCnCw&t=57s https://towardsdatascience.com/

I've been wondering on what is the recommended method of saving logits/activations using PyTorch Lightning. I've looked at Callbacks, Loggers and ModelHooks but none of the use-cases seem to be for this kind of activity (even if I were to create my own custom variants of each utility). The ModelCheckpoint Callback in its utility makes me feel like custom Callbacks would be the way to go but I'm not quite sure. This closed GitHub issue does address my issue to some extent.

Our codebase for the diffusion models builds heavily on OpenAI's ADM codebase , lucidrains, Stable Diffusion, Lightning and Hugging Face. Thanks for open-sourcing!

For lightweight use, pytorch-lightning is too heavy, and its source code will be very difficult for beginners to read, at least for me.

For the A100 GPU, theoretical performance is the same for FP16/BF16 and both rely on the same number of bits, meaning memory should be the same. However since it's quite newly added to PyTorch, performance seems to still be dependent on underlying operators used (pytorch lightning debugging in progress here).