Dreambooth-SD-optimized VS Stable-Diffusion-Regularization-Images

Hey guys im quite new to dream booth i tried the one in stabel diffusion but wasnt really satisfied with the output. Im lloking for a external dreambooth that could be started with anaconda but doesnt need 24 gb Vram i only have 12 Gb. I tried gammagec / Dreambooth-SD-optimizedbut he says you need at least 24 GB

Using this GitHub https://github.com/gammagec/Dreambooth-SD-optimized from this guide https://pastebin.com/xcFpp9Mr

I had pretty good results with 20 images of myself, 200 reg images and 6000 step using https://github.com/gammagec/Dreambooth-SD-optimized.

I use Automatic1111 for SD and then Dreambooth Optimized https://github.com/gammagec/Dreambooth-SD-optimized to do custom models.

Gammagec Dreambooth-SD-optimized - https://github.com/gammagec/Dreambooth-SD-optimized

I use this dreambooth repo to train the SD model. https://github.com/gammagec/Dreambooth-SD-optimized Here's the video that shows how to install it in very good detail. https://youtu.be/TwhqmkzdH3s He uses it to train in a face but you can use it to train in a style as well. I suggest taking about 15 or 20 detailed frames of the video and training it in as a style for the class name. You'll have to experiment with how many training steps to take. I suggest doing 1,000 steps at a time and testing out the model. Also don't leave the default "sks", the researchers forgot that that's a common acronym if you know what I mean. Do something like my_style1 so the model doesn't get confused with something else.

I'm using this guide: https://www.reddit.com/r/StableDiffusion/comments/xpoexy/yet_another_dreambooth_post_how_to_train_an_image/ to train locally with this repo https://github.com/gammagec/Dreambooth-SD-optimized on windows. Needs a 24gb card though.

However, when I look at regularization dataset that people have created, a lot of them are composed by bad quality AI generated pictures, for instance disfigured humans, or images full or artifacts. For instance, this image of train, or this one of a woman.

BTW, you can find the regularization images (ready to use class images) here.

Have you compared results to using regularization images from an existing repo such as https://github.com/JoePenna/Stable-Diffusion-Regularization-Images

For subjects/people, paste this into the github downloader https://github.com/JoePenna/Stable-Diffusion-Regularization-Images/tree/main/person_ddim

If you are using person, man or woman as class, you don't need to generate the images as there are a some github repos that have a bunch of them already generated for you to use. Nitrosocke also shared some, check my initial post for the link.

I did some experiments with dreambooth training. Overall better results were when I have used 1500 "person" class and about 50 training images. It is vital to have different background and different clothes otherwise it will "bake it" into your token (e.g. same sweater will influence all the rendering with color or pattern as it will be "part of your token"). Now I need to test textual inversion and see the difference.

As for what kind of images to use I've tried actual photos of people and images generated with Stable Diffusion and I've had pretty good results with both. I also tried using exclusively pictures of the person I'm training for everything and even that worked pretty well. All I can really say is that it seems to pay off if you keep an eye on the framing of your images - if the majority of your reference images cut off the upper 10% of the head for example then your model will tend to also produce images that cut off the upper 10% of the head. Oh, and I haven't tried it myself but this Github repository apparently has a ton of images specifically for use in DreamBooth.

I've made sure all my images are only me, and clean images. I have tried using the unsplash regularization images from https://github.com/JoePenna/Stable-Diffusion-Regularization-Images. I've tried generating my own images from SD itself. I've tried 1k, 2k, 3k, 4k steps. I've tried more images of myself and fewer. I've tried using "man", "person", "face" as the class. All of it results in absolute garbage. I get outputs that consistently look like I'm 80 years old or a different ethnicity. Or just wrong... so wrong.