denoising-diffusion-pytorch VS RAVE

I’m trying to train a diffusion model using this code (https://github.com/lucidrains/denoising-diffusion-pytorch). My idea is to take a short audio segment, transform it into a spectrogram and train the model on these images then have it generate spectrograms then go back to audio. However the model requires square images. I cannot for the life of me figure out how to make a square spectrogram. Also is a regular spectrogram or a mel spectrogram better for this application?

Generally it's without weights, but MusicLM is also a WIP more mature implementations have descriptions on how to train them and follow ups on small scale/crowd-sourced experiments & research[1].

[1]: https://github.com/lucidrains/denoising-diffusion-pytorch

[1] https://colab.research.google.com/drive/1sjy9odlSSy0RBVgMTgP7s99NXsqglsUL?usp=sharing#scrollTo=KOYPSxPf_LL7 [2] https://github.com/lucidrains/denoising-diffusion-pytorch/blob/main/denoising_diffusion_pytorch/denoising_diffusion_pytorch.py

Sure. I am using: https://github.com/lucidrains/denoising-diffusion-pytorch

Lucidrains GitHub is always my go to repo for understandable paper implementations https://github.com/lucidrains/denoising-diffusion-pytorch

Thanks for the reply. Is there any suggestion if I wanted to train a model to generate half cat and half butterfly images what I should do? I git cloned the code from https://github.com/lucidrains/denoising-diffusion-pytorch and trained from scratch.

DDIM/DDPM are the same model to train, they only differ at inference time. To start I would recommend building from lucidrains' MIT licenced version (https://github.com/lucidrains/denoising-diffusion-pytorch). Just play around with the models until you gain an intuition.

Our codebase for the diffusion models builds heavily on OpenAI's ADM codebase , lucidrains, Stable Diffusion, Lightning and Hugging Face. Thanks for open-sourcing!

Once you understand these papers you can begin to understand Palette, and from there I would start with an open-source diffusion implementation like this one and then modify it to suit your needs!

Also, theres new tools like https://github.com/acids-ircam/RAVE which can be used to create new sounds, i feel like the more granular you work with AI on art, the better can help you out. Magentajs is pretty cool! Im still just playing with it but is easy to use so far.

https://github.com/acids-ircam/RAVE hope this wasn't too much off topic, i'm just super enthusiastic about RAVE and have been trying to squeeze a sonification in with it with no good applications so far. this feels kinda awesome tho. maybe it fits?

I'm in the deep learning music scene, which is due for its stable diffusion moment in the next year or two. The (primarily) timbre transfer system called RAVE is where I'm starting, and my contribution is to optimize the system to improve training time.

[] https://github.com/acids-ircam/RAVE/tree/master/rave

I'm obsessed with generative audio models, particularly RAVE[0].

Music is set to have its GPT-3 / Stable Diffusion moment within a couple years.

I believe in 10 years the venn diagram of music made with computers and music made with neural nets will be a circle, and that now is a great time to jump in.

Would LOVE to swap notes with anyone else here into this. Email in bio.

[0] https://github.com/acids-ircam/RAVE

Source code is here: https://github.com/caillonantoine/RAVE