-
LoRA-EXTRACTOR
Discontinued A small script to facilitate the extraction of LoRA models from custom checkpoints.
-
InfluxDB
Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.
You can extract Loras from any model using this web-ui. Most of the ones I extract are from trained checkpoints(that I also use in merges) rather than from merges. Loras extracted from merges work though. Under the Utilities tab, there's a subtab called Extract Lora. You select a model and subtract a base model(usually the one it was trained on) to get the Lora. SD 1.5 pruned will work as a base model most cases, but if, for example, you have model that was trained on top of NovelAI or AnythingV3, selecting that as the base will provide better, and more potent, results.
Can you extract a "LoRA transform" by comparing the weights from 2 checkpoints? Yes you can. https://github.com/sashaok123/LoRA-EXTRACTOR/blob/main/lib/extract_lora_from_models.py is a script that does exactly that, and if you look at the code, all it is is just applying singular value decomposition (SVD) (the underlying method behind principal component analysis (PCA), for those who are more Statistics/ML minded) to calculate the transformation between the two. Will you lose information when you "compress" the network using SVD? yes, the amount of "information" (in stats this is measured by variance) explained by the "lower ranked" component omitted by the new compressed representation, however the more similar the networks are, the less components will be needed to represent the differences.