lora VS sd-scripts

Diffusion unet has an "extended" version nowadays that applies to the resnet part as well as the cross-attention: https://github.com/cloneofsimo/lora

Absolutely. But that doesn't matter because you only have to train it at scale, once. There are papers released already that show it's possible to update weights in small sections. You won't have to wait for the next monolithic LLM to drop to get up to date information. It will start to learn in bits and pieces.

No, it's a deep learning thing

Low Rank Adaptation of Large Language Models.

Recently, I have seen the LoRA technique (Low-Rank Adaptation of Large Language Models) as a popular method for fine-tuning LLMs and other models.

Enter LoRA, a method proposed for adapting pre-trained models to specific tasks[2]. By freezing pre-trained model weights and injecting trainable rank decomposition matrices into the transformer architecture, LoRA can reduce the number of trainable parameters and the GPU memory requirement, making the adaptation of LLMs for downstream tasks more feasible.

Open-source LLM projects have largely solved this using Low-Rank Adaptation of Large Language Models (LoRA): https://arxiv.org/abs/2106.09685

Apparently an RTX 4090 running overnight is sufficient to produce a fine-tuned model that can spit out new Harry Potter stories, or whatever...

Alpaca was trained for $600 ($100 for the smaller model) and offers outputs competitive with ChatGTP. https://arxiv.org/abs/2106.09685

This repository mostly provides a Windows-focused Gradio GUI for Kohya's Stable Diffusion trainers... but support for Linux OS is also provided through community contributions.

Leaving this here because i'm very tired, so this is the file of the ipynb that uses the sdxl_train.py from the https://github.com/kohya-ss/sd-scripts/tree/sdxl repo, if anybody find out why when getting to the training i get this very empty error : " [00:09:11] WARNING The following values were not passed to "

You can try and test training LoRAs now https://github.com/kohya-ss/sd-scripts/tree/sdxl

This might help.

Kohya_ss sd-scripts Seems to be the standard for lora training. The linked page has an English translation, but doesn't really have system specific tips. Someone else has a popular gui for it, but it's designed with windows in mind. There's another, simpler gui, but its still in development and the dev doesn't do any testing on Linux. With any of these, I run into dependency conflicts like crazy.

"Direct training" is already feasible with masking in kohya-ss: https://github.com/kohya-ss/sd-scripts/pull/589