k-diffusion VS Fooocus

Here's my attempt at an explanation without jargon, you can just read the last paragraph, the first 4 are just context.

These image models are trained on 1000 steps of noise, where at 0 no noise is added to the training image and at 1000 the image is pure noise. The model's goal it to denoise the image, and it does this knowing how much noise the image has, this makes the model learn how much it should change the image, for example at high noise it changes a lot of pixels and starts building the overall "structure" of the image, and a low noise it changes less pixels and focuses on adding details.

To use the model you start with pure noise, then the model iteratively denoises that noise until a clean image shows up. A naive approach would take 1000 steps, this means you run the model 1000 times, each time feeding the previous result and telling the model that the noise decreased by 1 until it reaches 0 noise. This takes a long time, up to 15 minutes to generate an image on a mid-range consumer GPU.

Turns out when you give the model pure noise and tell it there's 1000 steps of noise, the result is not an image that has 999 steps of noise, but an image that looks like it has much less, this means that you can probably skip 50-100 steps of denoising per iteration and still get a very good picture, the issue is: what steps to pick? You could again take a naive approach and just skip every 50 steps for a total of 20 steps, but turns out there's better ways.

This is where samplers come in, essentially a sampler takes the number of steps you want to take to denoise an image (usually ~20 steps) and it will--among other things--pick which steps to choose each iteration. The most popular samplers are the samplers in the k-diffusion repo[1] or k-samplers for short. Do note that samplers do much more than just pick the steps, they are actually responsible for doing the denoising process itself, some of them even add a small noise after a denoising step among other things.

The newest open source model, SDXL, is actually 2 models. A base model that can generate images as normal, and a refiner model that is specialized on adding details to images. A typical workflow is to ask the base model for 25 steps of denoise, but only run the first 20, then use the refiner model to do the rest. According to the OP, this was being done without keeping the state of the sampler, that is they were running 2 samplers separately, one for the base model and then start one over for the refiner model. Since the samplers use historical data for optimization, the end result was not ideal.

[1] https://github.com/crowsonkb/k-diffusion

I believe all the advanced samplers are defined upstream in this repo by crowsonkb. As for "loading them" into invoke, you would need to modify the invokeAI source code to define new samplers. The good news is since it's all in python, you don't need to do any compiling.

The K Diffusion samplers are probably the most advanced currently.

I don't know of any Sampler that is not already in A1111, and this is the closest thing to a "list of Samplers for SD".

This is the main source of all the Samplers we see in the various SD UI's. The source code has references to published papers behind the samplers. Aside from this, I haven't found a wiki for them.

git clone https://github.com/crowsonkb/k-diffusion.git

Am curious in seeing what drives these choices. I think LMS was the default in Dreamstudio when Stable Diffusion was released. Then Euler A became the default in AUTOMATIC1111 which I think explained a lot. But now that many people are more literate about samplers it looks like these decision are more deliberate. With a lot more samplers implemented in https://github.com/crowsonkb/k-diffusion and added to AUTOMATIC1111, is speed the main driver (DPM++ is a lot about speed https://arxiv.org/abs/2211.01095), what about image quality? What are your thoughts?

Almost all other samplers come from work done by @RiversHaveWings or Katherine Crowson, which is mostly contained in her work at this repository. She is listed as the principal researcher at Stability AI. Her notes for those samplers are as follows:

But we can come to a pretty realistic (although not as accurate) conclusion if we put our minds to it. I chose Fooocus for this example, which is the most straightforward (and I believe popular) stable diffusion GUI out there. Let's start simple:

# Install the JuiceFS client. !curl -sSL https://d.juicefs.com/install | sh - # Mount the JuiceFS file system. !juicefs mount rediss://:[email protected]/1 myjfs -d # Create the directory structure for Fooocus models in JuiceFS. !mkdir -p myjfs/models/{checkpoints,loras,embeddings,vae_approx,upscale_models,inpaint,controlnet,clip_vision} # Clone the Fooocus repository. !git clone https://github.com/lllyasviel/Fooocus.git

That looks very impressive unless the demo is cherrypicked, would be great if this could be implemented into a frontend like Fooocus https://github.com/lllyasviel/Fooocus

You might be thinking of Fooocus: https://github.com/lllyasviel/Fooocus

The Stable Diffusion web interface that got a lot of people's attention originally was Automatic1111: https://github.com/AUTOMATIC1111/stable-diffusion-webui

Fooocus is definitely more beginner friendly. It does a lot of the prompt engineering for you. Automatic1111 has a ton of plugins, most notably ControlNet which gives you fine grained control over the images, but there is a learning curve.

I just installed this last night on my laptop:

https://github.com/lllyasviel/Fooocus

Highly recommend:

>"Looking up from the deck of golden gate bridge at the towers and metal work, the towers rise and arch back in an ominous and foreboding manner. more artistic, like an alphonse mucha propaganda poster - slightly fish-eye feeling" -- https://i.imgur.com/vyNg79f.jpg

the local UI and 1.27.0.0.1 - https://i.imgur.com/wRwghuN.jpg

Why not use the Google Colab notebook while it's still a free option at: https://github.com/lllyasviel/Fooocus It's not bad. I've been using the Colab Fooocus notebook and A1111 on Sage with the 4 free hours of GPU time. The Colab has the Juggernaut model preloaded, but I combined some code from another notebook to add other models and loras.

i'm trying to test the abilities of image generators and the risks that can comes with it. and i'm looking for an image generator that work locally and has no limits. i used the Fooocus project from github and the juggernut model and it's capable of generating nude pictures but not fully nude pictures. and it doesn't work will with bloody scenes. any recommendation for a better model

I recently saw this video from Fireship and I started wondering about licensing of SD models and frameworks. Fireship shows Fooocus and advertises it as a cool solution. What I started wondering about is, Fooocus downloads a couple of models: Juggernaut XL, some control nets, some loras. What licensing is tied to all of this? One I am most insterested in is JuggernautXL, on civitai it's listed as having CreativeML Open RAIL++-M license but in the description there's remark: For business inquires, commercial licensing, custom models, and consultation contact me under [email protected] There's a lot of separate parts going on in AI frameworks and it's a bit unclear to understand if I can use this commercialy.

For running locally w/o a lot of "hazzle" i would recommend using https://github.com/lllyasviel/Fooocus and the "Sticker" style, which is available within the UI (Advanced tab). If you need lot of text directly in you images, you may have difficulties with SD and other AI models. In this case LoRas could help. Example: https://civitai.com/posts/880523 (check details for used prompt) In this case i used the following LoRa (a LoRa is kind of specialized submodel for style, concept or person) .