LoRA-EXTRACTOR VS LoRA-EXTRACTOR-Colab

Regardless of the method used to fine tune the model it wouldn't exceed the sizes I listed above. (fun fact, the maximum dimensions for LoRA is 320 (which limits the LoRA models to ~500mb, because that is the size of the smallest tensor) https://github.com/sashaok123/LoRA-EXTRACTOR/issues/8 (with LoCons it might be slightly larger, but I have not check out the theoretical limit of LoCons yet)

I changed sashaok123's code to work on Google Colab. https://github.com/AlirezaF80/LoRA-EXTRACTOR-Colab

Can you extract a "LoRA transform" by comparing the weights from 2 checkpoints? Yes you can. https://github.com/sashaok123/LoRA-EXTRACTOR/blob/main/lib/extract_lora_from_models.py is a script that does exactly that, and if you look at the code, all it is is just applying singular value decomposition (SVD) (the underlying method behind principal component analysis (PCA), for those who are more Statistics/ML minded) to calculate the transformation between the two. Will you lose information when you "compress" the network using SVD? yes, the amount of "information" (in stats this is measured by variance) explained by the "lower ranked" component omitted by the new compressed representation, however the more similar the networks are, the less components will be needed to represent the differences.

Instead of using kohya_ss, I used https://github.com/sashaok123/LoRA-EXTRACTOR

Here is the link to the LORA-extraction script on github by sashaok123: https://github.com/sashaok123/LoRA-EXTRACTOR

Link to GitHub, where you can download the script.

I changed sashaok123's code to work on Google Colab. https://github.com/AlirezaF80/LoRA-EXTRACTOR-Colab