StyleDomain VS StyleGAN-nada

Abstract: Domain adaptation of GANs is a problem of fine-tuning GAN models pretrained on a large dataset (e.g., StyleGAN) to a specific domain with few samples (e.g., painting faces, sketches, etc.). While there are many methods that tackle this problem in different ways, there are still many important questions that remain unanswered. In this paper, we provide a systematic and in-depth analysis of the domain adaptation problem of GANs, focusing on the StyleGAN model. We perform a detailed exploration of the most important parts of StyleGAN that are responsible for adapting the generator to a new domain depending on the similarity between the source and target domains. As a result of this study, we propose new efficient and lightweight parameterizations of StyleGAN for domain adaptation. Particularly, we show that there exist directions in StyleSpace (StyleDomain directions) that are sufficient for adapting to similar domains. For dissimilar domains, we propose Affine+ and AffineLight+ parameterizations that allow us to outperform existing baselines in few-shot adaptation while having significantly fewer training parameters. Finally, we examine StyleDomain directions and discover their many surprising properties that we apply for domain mixing and cross-domain image morphing. Source code can be found at GitHub.

Here's a paper about adding the ability to guide outputs with text a full year before Stable Diffusion was published.

StyleGAN-NADA: CLIP-Guided Domain Adaptation of Image Generators

project page: https://stylegan-nada.github.io/

Sooo I retrained the FFHQ model to be Danny using StyleGAN-NADA via this Colab notebook.

Haha thanks for trying it out :) It was actually really challenging to get it working (especially all in the browser without processing on a server). A lot of help came from stylegan-nada https://github.com/rinongal/StyleGAN-nada (and a custom lightweight model basically distilling pairs from it and ffhq).

As for usage on non-face images most of NVidia's pre-trained models were face based (animal, humans, and paintings). Which was the aim of releasing our WikiArt model so the community would have something that could generate a greater variety of images. However these models are still constrained to the dataset that they were trained on. So without some tricks you can't generate "novel" images (like mashups of different objects)

I think the directional loss concepts in https://github.com/rinongal/StyleGAN-nada have real potential for artistic work, as they can go beyond the filter and paint effects that traditional style transfer applies well, while maintaining recognisable equivalency between the resulting images.

5 Min Read | Paper | Project | Code