GPTQ-for-LLaMa VS starcoder

git clone https://github.com/oobabooga/GPTQ-for-LLaMa

If you're using the model directly from ehartford, that one hasn't been quantized. Try using the GPTQ quantized version here, and use this fork of GPTQ-for-LLaMa. Load in 4-bit with --wbits 4

This worked for me on Ubuntu. If you want to use the CUDA branch instead of triton, do the same steps except clone this GPTQ-for-LLaMa fork and run python setup_cuda.py install

If you don't have triton and you use AutoGPTQ you're gonna notice a huge slow down compared to the old GPTQ-for-LLaMA cuda branch. For me AutoGPTQ gives me a whopping 1 token per second compared to the old GPTQ that gives me a decent 9 tokens per second.. both times I used a same sized model. (I think the slowdown is due to AutoGPTQ using the newer cuda branch which is much slower than the old one)

Are you using a later version of GPTQ-for-LLaMa? If so, go to ooba's CUDA fork (https://github.com/oobabooga/GPTQ-for-LLaMa). That's what I made it in and it definitely works with that. And that's what's included in the one-click-installers.

This usually happens when using models that conflict with your GPTQ installation. You should be using this fork: https://github.com/oobabooga/GPTQ-for-LLaMa. If you did the manual installation wrong, use the one click installer instead.

cd text-generation-webui # wherever you have it installed mkdir -p repositories cd repositories git clone https://github.com/oobabooga/GPTQ-for-LLaMa -b cuda GPTQ-for-LLaMa cd GPTQ-for-LLaMa python setup_cuda install

I'm on GPTQ for LLaMA folder under repositories says it's pointed at https://github.com/oobabooga/GPTQ-for-LLaMa.git. But I've run through the instructions and also applied the monkey patch to train and apply 4 bit lora which may come into play. No idea.

git clone https://github.com/oobabooga/GPTQ-for-LLaMa.git -b cuda

Use local LLM like: https://github.com/bigcode-project/starcoder

Here's the script I use to merge a LoRA onto a base model: https://gist.github.com/TheBloke/d31d289d3198c24e0ca68aaf37a19032 (a slightly modified version of https://github.com/bigcode-project/starcoder/blob/main/finetune/merge_peft_adapters.py)

consult: https://github.com/bigcode-project/starcoder/issues/6 https://huggingface.co/bigcode/starcoder/discussions/12

StarCoder - The biggest open source code LLM. It’s a free VS code extension. Looks great for coding, makes you wonder how long things like Github Copilot and Ghostwriter can afford to charge when we have open source building things like this. Link to github [Link] Link to HF [Link]

Outside of just using GPT4, which works well, this is supposedly the solution, though I haven't tried it just yet. starcoder/README.md at main · bigcode-project/starcoder · GitHub

GitHub link: https://github.com/bigcode-project/starcoder/tree/main