dream-factory VS alpaca_lora_4bit

Good stuff! For anyone else that uses Dream Factory, I've converted this to a .prompts file for use with it here:

BTW if you aren't already using dream factory to manage your GPUs in stable diffusion you should be. It works with automatic1111 too so you won't be losing any abilities.

Take a look at Dream Factory - it's a multi-GPU capable front-end for Auto1111. It'll let you run as many GPUs as you want in parallel, and adds a bunch of automation and remote management features as well.

I compiled these into a single Dream Factory .prompt file here: https://github.com/rbbrdckybk/dream-factory-prompt-files/tree/main/prompts

Correct and you can use Dream Factory to help manage your queue across GPUs. But keep in mind you are still only using individual cards for individual prompts meaning multiple cards can't improve the speeds of individual prompt generation nor allow for larger images.

Wow 24 instances! Just in case you weren't aware, Dream Factory is a multi-GPU frontend for Auto1111. It'll let you manage sending work to all those GPUs from one place!

Dream Factory will let you use multiple GPUs in parallel. So essentially adding a 2nd identical GPU will double your output.

No worries - Dream Factory is essentially a front-end for the popular Automatic1111 SD repo. It adds a bunch of prompt automation and remote management features, along with a multi-GPU engine (works fine with just 1 GPU too) to Auto1111.

For training there is also https://github.com/johnsmith0031/alpaca_lora_4bit

I've been trying to fine tune an erotica model on some large context chat history (reverse proxy logs) and a literotica-instruct dataset I made, with a max context of 8k. The large context size eats a lot of VRAM so I've been trying to find the most efficient way to experiment considering I'd like to do multiple runs to test some ideas. So I'm going to try and use https://github.com/johnsmith0031/alpaca_lora_4bit, which is supposed to train faster and use less memory than qlora.

Follow up the popular work of u/tloen alpaca-lora, I wrapped the setup of alpaca_lora_4bit to add support for GPTQ training in form of installable pip packages. You can perform training and inference with multiple quantizations method to compare the results.

" Using LoRAs with GPTQ-for-LLaMa This requires using a monkey patch that is supported by this web UI: https://github.com/johnsmith0031/alpaca_lora_4bit"

4-bit GPTQ LoRA training was available since early April. I did not see any comparison to it in the QLoRA paper or even a mention, so it makes me think they were not aware it already existed.

https://github.com/johnsmith0031/alpaca_lora_4bit is still the SOTA - Faster than qlora, trains on a GPTQ base.

I would also like to know how one would finetune this in 4 bit? I think one could take the merged 8K PEFT with the LLaMA weights, and then quantize it to 4 bit, and then train with https://github.com/johnsmith0031/alpaca_lora_4bit ?

I was under the impression that you just git clone this repo into text-generation-webui/repositories (so you would have GPTQ_for_Llama and alpaca_lora_4bit in the folder), and then just load with monkey patch. Is that not correct? I also tried just downloading alpaca_lora_4bit on its own, git cloning text-gen-webui within it, and installing requirements.txt for both and running with monkey patch. I was following the sections of alpaca_lora_4bit, "Text Generation Webui Monkey Patch" and "monkey patch inside webui"

I dont have any familiarity with esxi, but I can say that there are quite a few posts about people doing it on proxmox. I've currently got a machine with 2x3090 passing through to VM's. When I'm training, I pass them both through to the same VM and can do lora 4-bit training on llama33 using https://github.com/johnsmith0031/alpaca_lora_4bit. Then, at inference time, I run a single card into a different VM, and have an extra card available for experimentation.