examples VS artbot-for-stable-diffusion

Hello everyone, I am trying to fine tune a Stable Diffusion 1.4 model to work on specific images which requires specific descriptions. I am following this github, which is a fork of the original one, I have12000 images and I am at the 20th epoch with a 0.199 loss: https://github.com/LambdaLabsML/examples/tree/main/stable-diffusion-finetuning

It seems there are two ways: 1) Use Dreambooth technique (joepenna, Shivam's, lastben repos) 2) Train on top of original stable-diffusion model (as described for example here https://github.com/LambdaLabsML/examples/tree/main/stable-diffusion-finetuning, on XavierXiao repo)

If you want to make or customize a model: - there's fine-tuning a model (not Dreambooth). You're essentially continuing the training process that the SD authors used. It requires professional-grade AI hardware and takes a while. People seem to not even know this exists. You start with some base model (usually plain SD, but it could be any model) and fine-tune it. You should assume the process of fine-tuning it will make it unsuitable for anything else -- for instance, if you tune on one person's face, expect it to never generate anyone else's face, and if you find-tune on one art style, any other art style may suck. - Dreambooth is a different method for fine-tuning a model, needing a fraction of the power and time "real" fine-tuning does. But it still takes a lot of power, the most optimized Dreambooth tools take 12GB of VRAM and most graphics cards don't even have that. - There's several competitors to the Dreambooth method, such as EveryDream, which claim better results and sometimes claim only needing one photo. I'm not sure how things have really played out, especially since you can't tell the difference between "this method sucks" and "this method is great but everyone is using it wrong". - Hypernetworks take less time and power than DB. some testify it's better for style training than DreamBooth - TI's take the least time and power, I recently saw a training method for 6 GB VRAM cards. - Aesthetic gradients don't need training! :)

I BLIP captioned the images to try and retrain using this approach: https://github.com/LambdaLabsML/examples/blob/main/stable-diffusion-finetuning/pokemon_finetune.ipynb. I used the BLIP captions, and then put "D&D character {race}" in front, where race was the manually annotated race I did. After that, for dreambooth (followed this roughly: https://www.youtube.com/watch?v=7bVZDeGPv6I) You don't need to rename the images, just put them in the same folder, which you specify in a json file that Dreambooth reads to know how to handle each class

Trained a Dreambooth model from v1.5 checkpoint. I tried finetuning the model using this approach: https://github.com/LambdaLabsML/examples/blob/main/stable-diffusion-finetuning/pokemon_finetune.ipynb, but I didn't achieve results I liked

Thank you! This model training did not use dreambooth. Here is the reference repo I used, it is based on the original training repo for stable diffusion. Dreambooth is a more sophisticated framework and I am very interested in doing a side-by-side comparison against this model as a follow-up.

Code and details: https://github.com/LambdaLabsML/examples/tree/main/stable-diffusion-finetuning

I am going to plug artbot, an actually free SD image generator: https://tinybots.net/artbot

The front end is all local, and the backend image generation run on volunteer hosts running on the AI Hoard. I donate my own 3090/2060 sometimes (albeit for text generation, not imagegen).

No mention of the AI Horde anywhere?

https://tinybots.net/artbot

https://lite.koboldai.net/

This AI Horde UI has, IMO, some really good templates and suggestions:

https://tinybots.net/artbot

Tried just now and this is the result, i haven't clear what is the LSDR 3.75 parameter, im using https://tinybots.net/artbot

Try either of these, they in theory would work on your phone or PC https://aqualxx.github.io/stable-ui/ https://tinybots.net/artbot

ArtBot