Dreambooth-Stable-Diffusion VS Stable-Diffusion-Regularization-Images

Tons of stuff here, no? https://github.com/JoePenna/Dreambooth-Stable-Diffusion/

Joe Penna's Dreambooth (Tutorial|24GB) Most popular DB repo with great results.

1) The https://github.com/JoePenna/Dreambooth-Stable-Diffusion instructions say to use the 1.5 checkpoints - is that the latest? Can I use the 2+ models or?

I quickly turned to the second library, https://github.com/JoePenna/Dreambooth-Stable-Diffusion, because its readme was very encouraging, and its results were the best. Unfortunately, to use it on Colab, you need to sign up for Colab Pro to use advanced GPUs (at least 24GB of VRAM), and training a model requires at least 14 compute units. As a poor Chinese person, I could only buy Colab Pro from a proxy. The results from JoePenna/Dreambooth-Stable-Diffusion were fantastic, and the preparation was straightforward, requiring only <=20 512*512 photos without writing captions. I used it to create many beautiful photos.

I've found EveryDream to be brilliant and have switched from JoePenna's Dreambooth because I've found I get better results so long as I provide good captions for all the images, even if preparing the dataset takes 3x as long (took me 2 hours to crop and label the 54 images).

From what I know you can train with whatever size you want. But you need software that will support it. For example, ShivamShrirao/diffusers repo seems to allow a change of dimension. Also, you need HW that would support the training, because bigger images need more VRAM, for example,Joe Penna repo is using ~23GB with 512x512px so probably it's not a valid option. But the ShivamShrirao repo has optimizations that allow to run it with less VRAM.

This is a good starting place: https://github.com/JoePenna/Dreambooth-Stable-Diffusion

However, when I look at regularization dataset that people have created, a lot of them are composed by bad quality AI generated pictures, for instance disfigured humans, or images full or artifacts. For instance, this image of train, or this one of a woman.

BTW, you can find the regularization images (ready to use class images) here.

Have you compared results to using regularization images from an existing repo such as https://github.com/JoePenna/Stable-Diffusion-Regularization-Images

For subjects/people, paste this into the github downloader https://github.com/JoePenna/Stable-Diffusion-Regularization-Images/tree/main/person_ddim

If you are using person, man or woman as class, you don't need to generate the images as there are a some github repos that have a bunch of them already generated for you to use. Nitrosocke also shared some, check my initial post for the link.

I did some experiments with dreambooth training. Overall better results were when I have used 1500 "person" class and about 50 training images. It is vital to have different background and different clothes otherwise it will "bake it" into your token (e.g. same sweater will influence all the rendering with color or pattern as it will be "part of your token"). Now I need to test textual inversion and see the difference.

As for what kind of images to use I've tried actual photos of people and images generated with Stable Diffusion and I've had pretty good results with both. I also tried using exclusively pictures of the person I'm training for everything and even that worked pretty well. All I can really say is that it seems to pay off if you keep an eye on the framing of your images - if the majority of your reference images cut off the upper 10% of the head for example then your model will tend to also produce images that cut off the upper 10% of the head. Oh, and I haven't tried it myself but this Github repository apparently has a ton of images specifically for use in DreamBooth.

I've made sure all my images are only me, and clean images. I have tried using the unsplash regularization images from https://github.com/JoePenna/Stable-Diffusion-Regularization-Images. I've tried generating my own images from SD itself. I've tried 1k, 2k, 3k, 4k steps. I've tried more images of myself and fewer. I've tried using "man", "person", "face" as the class. All of it results in absolute garbage. I get outputs that consistently look like I'm 80 years old or a different ethnicity. Or just wrong... so wrong.