stable-diffusion-videos VS sd-dynamic-thresholding

Main Code: https://github.com/nateraw/stable-diffusion-videos/

So I've continued to experiment with how many papers I can fit into a single pipe and have them play nicely together. The images below were created by combining the panorama code from omerbt/MultiDiffusion with the ideas from albarji/mixture-of-diffusers. Also turns out nateraw/stable-diffusion-videos can be seen as a special case of a panorama (in latent space rather than prompt space).

https://huggingface.co/spaces/tomg-group-umd/pez-dispenser https://huggingface.co/spaces/AIML-TUDA/safe-stable-diffusion https://huggingface.co/spaces/AIML-TUDA/semantic-diffusion https://github.com/nateraw/stable-diffusion-videos

The goal is to make a (short) video out of a given first and last frame. It is similar to what this guy does (https://github.com/nateraw/stable-diffusion-videos (7sec example video half way down page)). But instead of starting and ending with a prompt, I want to start and end with 2 different frames.

Hey Y'all! We've been working on a tool that extends Nate Raw's Stable Diffusion Videos repo and makes it as easy as possible to use for artists and are having a competition this week to stress test the beta and see who can use it to make the most compelling short video (40 seconds max)

Of course! There might be a better process but mainly used: 1.) Nate Raw's repo for morphing between prompts https://github.com/nateraw/stable-diffusion-videos 2.) Google FILM interpolation to smooth out transitions https://github.com/google-research/frame-interpolation

There are directions in the readme and an example script.

This does exactly that - https://github.com/nateraw/stable-diffusion-videos

For outputs to look right, you will need some form of CFG rescale or dynamic thresholding in order to correct for overexposure (A1111 extensions are linked -- I am told that ComfyUI has nodes available for these functions). A good starting point for CFG rescale is 0.7, as recommended in the paper. I strongly suspect that CFG rescale is not an ideal solution and leaves a substantial training-inference gap, and when using zero terminal SNR models I find that Dynamic Thresholding can give better outputs that are closer to what I expect from the data without the brownout often caused by CFG rescale. A potential starting point for Dynamic Thresholding would be: Restart sampler, 15 CFG scale, Mimic CFG scale 15 7.5, Sawtooth on both scale schedulers, 6 for both minimum values, scheduler value 4, do not separate feature channels, ZERO, STD. You will likely have to experiment a lot with Dynamic Thresholding. (edit: small correction to DT settings)

Recently switched from A1111 and i love it so far, flexibility to orchestrate complex workflows automatically instead of manual operations is a life changer. Anyhow, one extension i like on A1111 was this one: https://github.com/mcmonkeyprojects/sd-dynamic-thresholding

In the Automatic1111 webui, there is a Dynamic Thresholding (CFG scale fix) extension that:

Ive found using ADetailer (https://github.com/Bing-su/adetailer, using their reccomended advanced settings and face_yolov8n.pt) and Dynamic Thresholding (CFG set to 12 and Mimic to 7) has vastly improved my face renders. (https://github.com/mcmonkeyprojects/sd-dynamic-thresholding) GL!

The output LoRA works best with CFG at 4, because at 7 it gets that gasoline colors and contrast of overbaking, but I guess this is a tradeoff of that many steps in total (5200) since the earlier snapshots were not that good in style and with character details. You can use a workaround like the Dynamic Trescholding extention: https://github.com/mcmonkeyprojects/sd-dynamic-thresholding.git - helps a lot in many cases when you want a high CFG but the model/lora overbakes them (it mimics a lower CFG while keeping the high CFG details and prompt alignment).

Use sd-dynamic-thresholding extension (set CFG scale to 12 or more and mimic CFG scale to 7): https://github.com/mcmonkeyprojects/sd-dynamic-thresholding

Tweak this up or down for small changes. Too far and you’ll get a different image. Extensions like Dynamic Thresholding can let you go much higher without the overexposed look.

Turn CFG scale down or maybe use this extension, I've never used Dynamic Thresholding before but I think its what you want