stylegan2-flax-tpu VS denoising-diffusion-pytorch

I’ll take this opportunity to mention our research scaling StyleGAN 2 to larger datasets (using LAION) on food images, leveraging free TPU compute through the TRC program.

We trained for 36 days on a v4-8 on 558k images.

https://nyx-ai.github.io/stylegan2-flax-tpu/

We were hopeful GANs would beat diffusion models when trained on specific domains. But we’ve now switched to Stable Diffusion and Dreambooth training which has proven more much efficient for this purpose.

I still have hopes for GANs! I miss their insane inference speed.

https://nyx-ai.github.io/stylegan2-flax-tpu

https://nyx-ai.github.io/stylegan2-flax-tpu

This is an incremental update on the work we shared 3 months ago: https://news.ycombinator.com/item?id=32167704

We keep scaling up StyleGAN2 training using more data, larger models, and more compute. In this release we open-source a 5-class model able to generate images of burgers, cheesecakes, cocktails, cookies, and sushis at resolution 512x512.

This research is part of the technology underlying our AI-generated photography platform Nyx.gallery that we shared here 2 weeks ago: https://news.ycombinator.com/item?id=33179730

It is also part of an academic research into data augmentation using synthetic methods in partnership with the Food & You project (https://www.foodandyou.org/).

This is a continuation of our work on "This Food Does Not Exist" (Reddit discussion, Github, checkpoints).

Models for the 256px food images were previously released here:

https://github.com/nyx-ai/stylegan2-flax-tpu

This is the continuation of our work on "This Food Does Not Exist", using old-school StyleGAN2 with more data and larger models to generate bigger images (now 512x512 instead of 256x256).

I believe you had seen it before on HN: https://nyx-ai.github.io/stylegan2-flax-tpu/

📘 Repo: https://github.com/nyx-ai/stylegan2-flax-tpu

I’m trying to train a diffusion model using this code (https://github.com/lucidrains/denoising-diffusion-pytorch). My idea is to take a short audio segment, transform it into a spectrogram and train the model on these images then have it generate spectrograms then go back to audio. However the model requires square images. I cannot for the life of me figure out how to make a square spectrogram. Also is a regular spectrogram or a mel spectrogram better for this application?

Generally it's without weights, but MusicLM is also a WIP more mature implementations have descriptions on how to train them and follow ups on small scale/crowd-sourced experiments & research[1].

[1]: https://github.com/lucidrains/denoising-diffusion-pytorch

[1] https://colab.research.google.com/drive/1sjy9odlSSy0RBVgMTgP7s99NXsqglsUL?usp=sharing#scrollTo=KOYPSxPf_LL7 [2] https://github.com/lucidrains/denoising-diffusion-pytorch/blob/main/denoising_diffusion_pytorch/denoising_diffusion_pytorch.py

Sure. I am using: https://github.com/lucidrains/denoising-diffusion-pytorch

Lucidrains GitHub is always my go to repo for understandable paper implementations https://github.com/lucidrains/denoising-diffusion-pytorch

Thanks for the reply. Is there any suggestion if I wanted to train a model to generate half cat and half butterfly images what I should do? I git cloned the code from https://github.com/lucidrains/denoising-diffusion-pytorch and trained from scratch.

DDIM/DDPM are the same model to train, they only differ at inference time. To start I would recommend building from lucidrains' MIT licenced version (https://github.com/lucidrains/denoising-diffusion-pytorch). Just play around with the models until you gain an intuition.

Our codebase for the diffusion models builds heavily on OpenAI's ADM codebase , lucidrains, Stable Diffusion, Lightning and Hugging Face. Thanks for open-sourcing!

Once you understand these papers you can begin to understand Palette, and from there I would start with an open-source diffusion implementation like this one and then modify it to suit your needs!