StyleGAN-nada VS artistic-videos

Here's a paper about adding the ability to guide outputs with text a full year before Stable Diffusion was published.

StyleGAN-NADA: CLIP-Guided Domain Adaptation of Image Generators

project page: https://stylegan-nada.github.io/

Sooo I retrained the FFHQ model to be Danny using StyleGAN-NADA via this Colab notebook.

Haha thanks for trying it out :) It was actually really challenging to get it working (especially all in the browser without processing on a server). A lot of help came from stylegan-nada https://github.com/rinongal/StyleGAN-nada (and a custom lightweight model basically distilling pairs from it and ffhq).

As for usage on non-face images most of NVidia's pre-trained models were face based (animal, humans, and paintings). Which was the aim of releasing our WikiArt model so the community would have something that could generate a greater variety of images. However these models are still constrained to the dataset that they were trained on. So without some tricks you can't generate "novel" images (like mashups of different objects)

I think the directional loss concepts in https://github.com/rinongal/StyleGAN-nada have real potential for artistic work, as they can go beyond the filter and paint effects that traditional style transfer applies well, while maintaining recognisable equivalency between the resulting images.

5 Min Read | Paper | Project | Code

It's been figured out back in the GAN days, then applied to disco diffusion, and then finally stable warp diffusion, although locked behind a patreon paywall. There are also extensions for A1111 Webui like this temporal kit but it's mostly based on ebsynth and doesn't do true temporal warping that I have in mind with these other links.

What you are thinking of is called "temporal coherence", and it was used all the way back in 2016 to create videos with neural style transfer. Example: https://github.com/manuelruder/artistic-videos

Mostly using this repo: https://github.com/manuelruder/artistic-videos

You should definitely have the option to toggle optical flow on/off in your plugin if this is what it looks like with it off. I've come across it before while using this old beauty, but I'm guessing that is the old and messy version you mentioned further up the thread.

There are a lot of works in video style transfer (ex: https://github.com/manuelruder/artistic-videos, https://github.com/manuelruder/fast-artistic-videos, https://github.com/sunshineatnoon/LinearStyleTransfer) but with any of this you wont achive such quality out of the box. Video above is a commercial product with a lot of tricks hidden inside it, witch only creators are aware of.