k-diffusion
instruct-pix2pix
k-diffusion | instruct-pix2pix | |
---|---|---|
20 | 21 | |
2,078 | 5,649 | |
- | - | |
8.4 | 0.0 | |
6 days ago | about 1 year ago | |
Python | Python | |
MIT License | GNU General Public License v3.0 or later |
Stars - the number of stars that a project has on GitHub. Growth - month over month growth in stars.
Activity is a relative number indicating how actively a project is being developed. Recent commits have higher weight than older ones.
For example, an activity of 9.0 indicates that a project is amongst the top 10% of the most actively developed projects that we are tracking.
k-diffusion
-
Fooocus: OSS of prompts and generations based on A111 and ComfyUI
Here's my attempt at an explanation without jargon, you can just read the last paragraph, the first 4 are just context.
These image models are trained on 1000 steps of noise, where at 0 no noise is added to the training image and at 1000 the image is pure noise. The model's goal it to denoise the image, and it does this knowing how much noise the image has, this makes the model learn how much it should change the image, for example at high noise it changes a lot of pixels and starts building the overall "structure" of the image, and a low noise it changes less pixels and focuses on adding details.
To use the model you start with pure noise, then the model iteratively denoises that noise until a clean image shows up. A naive approach would take 1000 steps, this means you run the model 1000 times, each time feeding the previous result and telling the model that the noise decreased by 1 until it reaches 0 noise. This takes a long time, up to 15 minutes to generate an image on a mid-range consumer GPU.
Turns out when you give the model pure noise and tell it there's 1000 steps of noise, the result is not an image that has 999 steps of noise, but an image that looks like it has much less, this means that you can probably skip 50-100 steps of denoising per iteration and still get a very good picture, the issue is: what steps to pick? You could again take a naive approach and just skip every 50 steps for a total of 20 steps, but turns out there's better ways.
This is where samplers come in, essentially a sampler takes the number of steps you want to take to denoise an image (usually ~20 steps) and it will--among other things--pick which steps to choose each iteration. The most popular samplers are the samplers in the k-diffusion repo[1] or k-samplers for short. Do note that samplers do much more than just pick the steps, they are actually responsible for doing the denoising process itself, some of them even add a small noise after a denoising step among other things.
The newest open source model, SDXL, is actually 2 models. A base model that can generate images as normal, and a refiner model that is specialized on adding details to images. A typical workflow is to ask the base model for 25 steps of denoise, but only run the first 20, then use the refiner model to do the rest. According to the OP, this was being done without keeping the state of the sampler, that is they were running 2 samplers separately, one for the base model and then start one over for the refiner model. Since the samplers use historical data for optimization, the end result was not ideal.
[1] https://github.com/crowsonkb/k-diffusion
-
Is it possible to install dpm++ 2s a karras on InvokeAI? 🙏
I believe all the advanced samplers are defined upstream in this repo by crowsonkb. As for "loading them" into invoke, you would need to modify the invokeAI source code to define new samplers. The good news is since it's all in python, you don't need to do any compiling.
- Why does UniPC sampler use DDIM for Hires Fix?
-
Can someone ELI5 the differences between samplers?
The K Diffusion samplers are probably the most advanced currently.
-
Is there a resource that has list of samplers for SD? Like https://upscale.wiki/wiki/Model_Database for upscalers?
I don't know of any Sampler that is not already in A1111, and this is the closest thing to a "list of Samplers for SD".
-
Different Samplers?
This is the main source of all the Samplers we see in the various SD UI's. The source code has references to published papers behind the samplers. Aside from this, I haven't found a wiki for them.
-
Image editing with just text prompt. New Instruct2Pix2Pix paper. Demo link in comments
git clone https://github.com/crowsonkb/k-diffusion.git
-
The sampler vibe started with LMS, then there was a big migration to using EULER A. Are many now moving to DPM++ e.g. DPM++ 2S a Karras and why?
Am curious in seeing what drives these choices. I think LMS was the default in Dreamstudio when Stable Diffusion was released. Then Euler A became the default in AUTOMATIC1111 which I think explained a lot. But now that many people are more literate about samplers it looks like these decision are more deliberate. With a lot more samplers implemented in https://github.com/crowsonkb/k-diffusion and added to AUTOMATIC1111, is speed the main driver (DPM++ is a lot about speed https://arxiv.org/abs/2211.01095), what about image quality? What are your thoughts?
-
Can anyone explain differences between sampling methods and their uses to me in simple terms, because all the info I've found so far is either very contradicting or complex and goes over my head
Almost all other samplers come from work done by @RiversHaveWings or Katherine Crowson, which is mostly contained in her work at this repository. She is listed as the principal researcher at Stability AI. Her notes for those samplers are as follows:
- K-diffusion: Karras et al. (2022) diffusion models for PyTorch
instruct-pix2pix
-
Stable Video Diffusion
My guess is you're thinking of InstructPix2Pix[1], with prompts like "make the sky green" or "replace the fruits with cake"
[1] https://github.com/timothybrooks/instruct-pix2pix
-
AI image editors with “text to filter” function?
This comes from https://github.com/timothybrooks/instruct-pix2pix, there is also an extension to use it in Automatic1111 Stable diffusion webui.
- [D] NeRF, LeRF, Prolific Dreamer, Neuralangelo, and a lot of other cool NeRF research
-
Was it SD that had the ability to edit a photo using prompts?
InstructPix2Pix
-
Alternate download location for instruct-pix2pix-00-22000.ckpt?
Is there another place I can download the model? I tried downloading the file using the instructions on this page:
-
Using our photoshop plugin for some cool image editing! :D
It comes from https://github.com/timothybrooks/instruct-pix2pix, you can try it out https://huggingface.co/spaces/timbrooks/instruct-pix2pix
-
instruct pix2pix faces always come out messed up. The rest is really good. Any idea how to fix this?
interesting, I've been running it using this: https://github.com/timothybrooks/instruct-pix2pix/blob/main/LICENSE
-
Everybody is always talking about AGI. I'm more curious about using the tools that we have now.
This is already done and it's already been implemented in the most popular web-ui for stable diffusion too. Granted the results aren't perfect yet.
-
gif2gif: Quick and easy webui extension for dropping animated GIFs into img2img
Select the script, drop in a GIF, use img2img as normal to process it. Supports quick non-ffmpeg interpolation, and works surprisingly well with InstructPix2Pix. Intended to be a fun no-nonsense GIF pipeline.
-
NMKD Stable Diffusion GUI 1.9.0 is out now, featuring InstructPix2Pix - Edit images simply by using instructions! Link and details in comments.
Github Issue - Closed
What are some alternatives?
stable-diffusion - k_diffusion wrapper included for k_lms sampling. fixed for notebook.
stable-diffusion-webui - Stable Diffusion web UI