paint-with-words-sd
daam
paint-with-words-sd | daam | |
---|---|---|
13 | 4 | |
618 | 607 | |
- | 3.6% | |
5.2 | 5.5 | |
about 1 year ago | about 1 month ago | |
Jupyter Notebook | Jupyter Notebook | |
MIT License | MIT License |
Stars - the number of stars that a project has on GitHub. Growth - month over month growth in stars.
Activity is a relative number indicating how actively a project is being developed. Recent commits have higher weight than older ones.
For example, an activity of 9.0 indicates that a project is amongst the top 10% of the most actively developed projects that we are tracking.
paint-with-words-sd
-
paint with words with loras and multicontrolnet (will pay if needed)
I am refering to this btw: https://github.com/cloneofsimo/paint-with-words-sd
-
More control than ControlNet - code is out for MultiDiffusion Region Control, a prompt on each mask
This essentially supercharges the Nvidia eDiffi / SD paint-with-words attempts done for the same thing previously.
- "Segmentation" ControlNet preprocessor options
-
I figured out a way to apply different prompts to different sections of the image with regular Stable Diffusion models and it works pretty well.
There is stable diffusion paint with words GitHub which probably does exactly this, but no UI ever: https://github.com/cloneofsimo/paint-with-words-sd
-
What do you think will be added/created next?
personally i want to see the ediffi paint w/words stable diffusion extension https://github.com/cloneofsimo/paint-with-words-sd/commit/789419e3a34f43a1454df5a940020cfa531fbc63 that clonesofimo was working on before he stopped
- Will models have to be retrained for when this feature is eventually added into SD?
-
Paint with words (aka NVIDIA eDiff-I)
Just found there is a repo for an NVIDIA eDiff-I style img2img workflow for Stable Diffusion. For those unfamiliar, this lets you specify where parts of your text prompt should be placed in the image giving you much greater control on the composition e.g.
-
Different Models = Different prompts?
Paint-with-Words might eventually allow something along those lines, but it's a bit awkward to use now, and AFAIK you still get bleedthrough between multiple human subjects.
-
eDiff-I: A new Text-to-Image Diffusion Model with Ensemble of Expert Denoisers
someone attempted something like paint with words but I think Nvidia's version is better looking.
- Paint with words? What is next? Hope this gets to be a module in automatic 1111 soon.
daam
- q: can I somehow validate my prompt as meaningful?
-
Patchouli is kind of a vibe, tbh
I did see this recently, where it can generate a heatmap of what an addition to a prompt changed in the image, but I haven't tried it yet, and that's not exactly much of an an explanation anyways
-
[R] What the DAAM: Interpreting Stable Diffusion and Uncovering Generation Entanglement
Paper: What the DAAM: Interpreting Stable Diffusion Using Cross Attention (arXiv paper, codebase)
-
[Discussion] Approach for applying Explainable AI on Diffusion Models
Found relevant code at https://github.com/castorini/daam + all code implementations here
What are some alternatives?
ComfyUI - The most powerful and modular stable diffusion GUI, api and backend with a graph/nodes interface.
Rerender_A_Video - [SIGGRAPH Asia 2023] Rerender A Video: Zero-Shot Text-Guided Video-to-Video Translation
openOutpaint - local offline javascript and html canvas outpainting gizmo for stable diffusion webUI API 🐠
infinite-zoom-stable-diffusion - resources for creating Ininite zoom video using Stable Diffiusion, you can use multiple prompts and it is easy to use.
LECO - Low-rank adaptation for Erasing COncepts from diffusion models.
stable-diffusion-docker - Run the official Stable Diffusion releases in a Docker container with txt2img, img2img, depth2img, pix2pix, upscale4x, and inpaint.
openOutpaint-webUI-extension - direct A1111 webUI extension for openOutpaint
lora - Using Low-rank adaptation to quickly fine-tune diffusion models.
stable-diffusion-webui-two-shot - Latent Couple extension (two shot diffusion port)
blended-latent-diffusion - Official implementation for "Blended Latent Diffusion" [SIGGRAPH 2023]