stylegan VS ffhq-dataset

Been following generative AI since 2017 when nvidia released their first GAN paper & the results always fascinated me. Trained my own models with their repo then experimented with other open source projects. went thru the pain of assembling my own data set, tweaking code parameters to achieve what i'm looking for, had to deal with all kinds of hardware/software issues. I know it's not easy. (screenshot of a motorbike GAN model i was training in 2018 https://imgur.com/a/SIULFhR, was taken after 5 hours of training on a gtx 1080) or this, cinema camera output from another locally trained model. So yeah i have a couple ideas of how generative AI works. yup things were that bad few years ago, that technology has come a long way. Using & setting up something like stable diffusion with automatic1111 webui isn't really a complex process. Though generating AI art locally is always gonna feel more rewarding than using a cloud based service.

Their algorithm is public, you could do it yourself if you have the proper hardware: https://github.com/NVlabs/stylegan

Umm, StyleGAN was the first decent image generation model, and it was producing great images from random seeds 5 years ago. Now, that's with the obvious caveat that each model was trained to produce one specific type of image and it helped immensely if the training images were all aligned the same. Diffusion models are certainly the trendy current architecture for image generation, but AFAIK there's no fundamental theoretical limitation to the output quality of any architecture except the general rule that more parameters is better.

Jokes aside, ML moves extremely fast and our field is quickly advancing. The honest truth is that no researcher can even keep up other than their extremely niche corner. I'll show you an example. Here's what state of the art image generation looked like in 2014, 2018, and here is today (which now is highly controllable using text prompts instead of data prompts).

The fundamental technology we will use in this work is a generative adversarial network. Specifically, the Style GAN variant.

Based on my findings, we don't really need FP64 unless it's for certain medical applications. But The Best GPUs for Deep Learning in 2020 — An In-depth Analysis is suggesting A100 outperforms A6000 ~50% in DL. Also the Stylegan project  GitHub - NVlabs/stylegan: StyleGAN - Official TensorFlow Implementation uses NVIDIA DGX-1 with 8 Tesla V100 16G(Fp32=15TFLOPS) to train dataset of  high-res 1024*1024 images, I'm getting a bit uncertain if my specific tasks would require FP64 since my dataset is also high-res images. If not, can I assume A6000*5(total 120G) could provide similar results for StyleGan?

Yes that's what I thought. But StyleGan https://github.com/NVlabs/stylegan uses NVIDIA DGX-1 with 8 Tesla V100 16G GPUs(FP32=15) to do the training, not sure if it's related to its high-res training images or something else.

Datasets like FFHQ consist of face images crawled from the Internet. While those images are published under CC licenses, the authors usually have not obtained consent from each person depicted in those images. I guess that's why they are taking takedown requests: People can send requests to remove their faces from the dataset.

The real ones were taken from this dataset.

NVIDIA's FFHQ says "Only images under permissive licenses were collected." https://github.com/NVlabs/ffhq-dataset

The FFHQ dataset https://github.com/NVlabs/ffhq-dataset is a high quality, high resolution, and extremely well curated dataset that is used in many recent SOTA GAN papers and also has applications in many other areas.

I trained on FFHQ and Danbooru2019 Portraits with resolution 512x512.

> Which makes it an inexcusable mistake to make in 2021 - how are you not testing for this?

They probably are, but not good enough. These things can be surprisingly hard to detect. Post hoc it is easy to see the bias, but it isn't so easy before you deploy the models.

If we take racial connotations out of it then we could say that the algorithm is doing quite well because it got the larger hierarchical class correct, primate. The algorithm doesn't know the racial connotations, it just knows the data and what metric you were seeking. BUT considering the racial and historical context this is NOT an acceptable answer (not even close).

I've made a few comments in the past about bias and how many machine learning people are deploying models without understanding them. This is what happens when you don't try to understand statistics and particularly long tail distributions. gumboshoes mentioned that Google just removed the primate type labels. That's a solution, but honestly not a great one (technically speaking). But this solution is far easier than technically fixing the problem (I'd wager that putting a strong loss penalty for misclassifiying a black person as an ape is not enough). If you follow the links from jcims then you might notice that a lot of those faces are white. Would it be all that surprising if Google trained from the FFHQ (Flickr) Dataset?[0] A dataset known to have a strong bias towards white faces. We actually saw that when Pulse[1] turned Obama white (do note that if you didn't know the left picture was a black person and who they were that this is a decent (key word) representation). So it is pretty likely that _some_ problems could simply be fixed by better datasets (This part of the LeCunn controversy last year).

Though datasets aren't the only problems here. ML can algorithmically highlight bias in datasets. Often research papers are metric hacking, or going for the highest accuracy that they can get[2]. This leaderboardism undermines some of the usage and often there's a disconnect between researchers and those in production. With large and complex datasets we might be targeting leaderboard scores until we have a sufficient accuracy on that dataset before we start focusing on bias on that dataset (or more often we, sadly, just move to a more complex dataset and start the whole process over again). There's not many people working on the biased aspects of ML systems (both in data bias and algorithmic bias), but as more people are putting these tools into production we're running into walls. Many of these people are not thinking about how these models are trained or the bias that they contain. They go to the leaderboard and pick the best pre-trained model and hit go, maybe tuning on their dataset. Tuning doesn't eliminate the bias in the pre-training (it can actually amplify it!). ~~Money~~Scale is NOT all you need, as GAMF often tries to sell. (or some try to sell augmentation as all you need)

These problems won't be solved without significant research into both data and algorithmic bias. They won't be solved until those in production also understand these principles and robust testing methods are created to find these biases. Until people understand that a good ImageNet (or even JFT-300M) score doesn't mean your model will generalize well to real world data (though there is a correlation).

So with that in mind, I'll make a prediction that rather than seeing fewer cases of these mistakes rather we're going to see more (I'd actually argue that there's a lot of this currently happening that you just don't see). The AI hype isn't dying down and more people are entering that don't want to learn the math. "Throw a neural net at it" is not and never will be the answer. Anyone saying that is selling snake oil.

I don't want people to think I'm anti-ML. In fact I'm a ML researcher. But there's a hard reality we need to face in our field. We've made a lot of progress in the last decade that is very exciting, but we've got a long way to go as well. We can't just have everyone focusing on leaderboard scores and expect to solve our problems.

[0] https://github.com/NVlabs/ffhq-dataset

[1] https://twitter.com/Chicken3gg/status/1274314622447820801

[2] https://twitter.com/emilymbender/status/1434874728682901507