alpaca_lora_4bit VS text-generation-webui-testing

For training there is also https://github.com/johnsmith0031/alpaca_lora_4bit

I've been trying to fine tune an erotica model on some large context chat history (reverse proxy logs) and a literotica-instruct dataset I made, with a max context of 8k. The large context size eats a lot of VRAM so I've been trying to find the most efficient way to experiment considering I'd like to do multiple runs to test some ideas. So I'm going to try and use https://github.com/johnsmith0031/alpaca_lora_4bit, which is supposed to train faster and use less memory than qlora.

Follow up the popular work of u/tloen alpaca-lora, I wrapped the setup of alpaca_lora_4bit to add support for GPTQ training in form of installable pip packages. You can perform training and inference with multiple quantizations method to compare the results.

" Using LoRAs with GPTQ-for-LLaMa This requires using a monkey patch that is supported by this web UI: https://github.com/johnsmith0031/alpaca_lora_4bit"

4-bit GPTQ LoRA training was available since early April. I did not see any comparison to it in the QLoRA paper or even a mention, so it makes me think they were not aware it already existed.

https://github.com/johnsmith0031/alpaca_lora_4bit is still the SOTA - Faster than qlora, trains on a GPTQ base.

I would also like to know how one would finetune this in 4 bit? I think one could take the merged 8K PEFT with the LLaMA weights, and then quantize it to 4 bit, and then train with https://github.com/johnsmith0031/alpaca_lora_4bit ?

I was under the impression that you just git clone this repo into text-generation-webui/repositories (so you would have GPTQ_for_Llama and alpaca_lora_4bit in the folder), and then just load with monkey patch. Is that not correct? I also tried just downloading alpaca_lora_4bit on its own, git cloning text-gen-webui within it, and installing requirements.txt for both and running with monkey patch. I was following the sections of alpaca_lora_4bit, "Text Generation Webui Monkey Patch" and "monkey patch inside webui"

I dont have any familiarity with esxi, but I can say that there are quite a few posts about people doing it on proxmox. I've currently got a machine with 2x3090 passing through to VM's. When I'm training, I pass them both through to the same VM and can do lora 4-bit training on llama33 using https://github.com/johnsmith0031/alpaca_lora_4bit. Then, at inference time, I run a single card into a different VM, and have an extra card available for experimentation.

Also autograd from here: https://github.com/Ph0rk0z/text-generation-webui-testing/ and it's matching GPTQ: https://github.com/Ph0rk0z/GPTQ-Merged/tree/dual-model

Some of the novel-ai ones work but the first one I tried to make it not spazz out was: https://github.com/Ph0rk0z/text-generation-webui-testing/blob/DualModel/presets/MOSS.yaml

i've never tried that particular one. everything else I threw at it trained through : https://github.com/Ph0rk0z/text-generation-webui-testing/ successfully.

as integrated in https://github.com/Ph0rk0z/text-generation-webui-testing/

if you like the "monkeypatch" https://github.com/Ph0rk0z/text-generation-webui-testing/ is better. I think in the discussion someone got it running on windows.

https://github.com/Ph0rk0z/text-generation-webui-testing < 4bit lora use from the UI on old GPTQ

well im running it with oobabooga/text-generation-webui and 8 bit now works after i did this fix Add 8bit threshold for my Pascal card. I use 1.5 or 1.0, otherwise NaN · Ph0rk0z/text-generation-webui-testing@ecad08f (github.com)