StyleGAN-nada VS prompt-to-prompt

Here's a paper about adding the ability to guide outputs with text a full year before Stable Diffusion was published.

StyleGAN-NADA: CLIP-Guided Domain Adaptation of Image Generators

project page: https://stylegan-nada.github.io/

Sooo I retrained the FFHQ model to be Danny using StyleGAN-NADA via this Colab notebook.

Haha thanks for trying it out :) It was actually really challenging to get it working (especially all in the browser without processing on a server). A lot of help came from stylegan-nada https://github.com/rinongal/StyleGAN-nada (and a custom lightweight model basically distilling pairs from it and ffhq).

As for usage on non-face images most of NVidia's pre-trained models were face based (animal, humans, and paintings). Which was the aim of releasing our WikiArt model so the community would have something that could generate a greater variety of images. However these models are still constrained to the dataset that they were trained on. So without some tricks you can't generate "novel" images (like mashups of different objects)

I think the directional loss concepts in https://github.com/rinongal/StyleGAN-nada have real potential for artistic work, as they can go beyond the filter and paint effects that traditional style transfer applies well, while maintaining recognisable equivalency between the resulting images.

5 Min Read | Paper | Project | Code

Found relevant code at https://github.com/google/prompt-to-prompt + all code implementations here

Null text inversion produces almost a perfect textual inversion, and then allows you to edit it with a prompt, like instruct2pix. https://github.com/google/prompt-to-prompt

First we had Google's prompt to prompt https://github.com/google/prompt-to-prompt

Narratives can also work, like "walking down forest path". However, it'll be difficult to keep the character positioned the way you want with that. If you're a techie somewhat, you can try to use https://github.com/google/prompt-to-prompt to generate someone you like and then see if you can get a better background without changing the character.

Google has a nice paper on text-guided image2image translation by inferring the (random) init image and changing the prompt: https://github.com/google/prompt-to-prompt