lightweight-gan VS stylegan2

GAN details: I trained using the code from https://github.com/lucidrains/lightweight-gan, image size is 256, attn-res-layers is [32,64], disc_output_size is 5 and I trained with AMP.

Long short story, i trained AI called 'Lightweight' GAN with some image from the game (Steam version). The result isn't satisfying (either too similar with original image or distorted), but i decided to share it since it took 3-4 days.

Code for training your own [original] [simple] [light]

Trained lightweight-gan on public domain nebulae images. The blogpost covers how it was assembled end-to-end, from data scraping, to model training, and finally deploying it as a web app using free cloud services only

https://github.com/lucidrains/stylegan2-pytorch is the simplest implementation I know of, and https://github.com/lucidrains/lightweight-gan is similar but designed for relatively lower performance machines.

I also regret using https://github.com/lucidrains/lightweight-gan over the standard StyleGAN2. The training time wasn't an issue for 128x128 images and I should have gone for the higher quality model.

It's hard to tell if the medicore results are due to the small dataset (~305 semi-aligned images, yes there's that many of these) or my own inexperience in GAN generation (i used https://github.com/lucidrains/lightweight-gan and the only changes i made from the defaults were to decrease the batch size and add color augmentation)

I don't know. If you're really curious, you can just try it: https://github.com/NVlabs/stylegan2

StyleGAN2 (Dec 2019) - Karras et al. and Nvidia

> The denoising part of a denoising autoencoder refers to the noise applied to its input

Agree, it converts a noisy image to a denoised image. But the odd thing is, when you put a noisy image into a StyleGAN2 encoder, you get latents which the decoder will turn into a de-noised image. So in practical use, you can take a trained StyleGAN2 encoder/decoder pair and use it as if it was a denoiser.

> These differences lead to learned distributions in the latent space that are entirely different

I also agree there. The training for a denoising auto-encoder and for a GAN network is different, leading to different distributions which are sampled for generating the images. But the architecture is still very similar, meaning the limits of what can be learned should be the same.

> Beyond that the comparison just doesn't work, yes there are two networks but the discriminator doesn't play the role of the AE's encoder at all

Yes, the discriminator in a GAN won't work like an encoder. But if you look at how StyleGAN 1/2 are used in practice, people combine it with a so-called "projection", which is effectively an encoder to convert images to latents. So people use a pipeline of "image to latent encoder" + "latent to image decoder".

That whole pipeline is very similar to an auto-encoder. For example, here's an NVIDIA paper about how they round-trip from image to latent to image with StyleGAN: https://arxiv.org/abs/1912.04958 My interpretation of what they did in that paper is that they effectively trained a StyleGAN-like model with the image L2 loss typically used for training a denoising auto-encoder.

These problems are usually done with CNN Encoder-Decoder frameworks. Usually GAN (Generative Adversarial Networks see StyleGan2).

This is the result of two different models. The first looks like a latent space interpolation of StyleGan2 and the mouth movements are without a doubt from wav2lip.

OP: if you want to run this at higher resolution, you should probably look at running it yourself, using something like this: https://github.com/NVlabs/stylegan2

StyleGAN2 (Dec 2019) - Karras et al. and Nvidia

A few of you might've seen an earlier post of mine about this project (Or the repost that got more upvotes 🙃), and I've improved the code and network since then after more thoroughly reading and understanding the official StyleGAN2 implementation.

So I've been a Pro+ subscriber since around November which I mainly use to train GANs. I have multiple Google accounts, let's call them Account 1, 2, and 3. Accounts 1 and 2 are normal Google accounts and Account 3 is an account I got from my university after I graduated which has unlimited storage.