GPTQ-for-LLaMa VS bitsandbytes-rocm

git clone https://github.com/oobabooga/GPTQ-for-LLaMa

If you're using the model directly from ehartford, that one hasn't been quantized. Try using the GPTQ quantized version here, and use this fork of GPTQ-for-LLaMa. Load in 4-bit with --wbits 4

This worked for me on Ubuntu. If you want to use the CUDA branch instead of triton, do the same steps except clone this GPTQ-for-LLaMa fork and run python setup_cuda.py install

If you don't have triton and you use AutoGPTQ you're gonna notice a huge slow down compared to the old GPTQ-for-LLaMA cuda branch. For me AutoGPTQ gives me a whopping 1 token per second compared to the old GPTQ that gives me a decent 9 tokens per second.. both times I used a same sized model. (I think the slowdown is due to AutoGPTQ using the newer cuda branch which is much slower than the old one)

Are you using a later version of GPTQ-for-LLaMa? If so, go to ooba's CUDA fork (https://github.com/oobabooga/GPTQ-for-LLaMa). That's what I made it in and it definitely works with that. And that's what's included in the one-click-installers.

This usually happens when using models that conflict with your GPTQ installation. You should be using this fork: https://github.com/oobabooga/GPTQ-for-LLaMa. If you did the manual installation wrong, use the one click installer instead.

cd text-generation-webui # wherever you have it installed mkdir -p repositories cd repositories git clone https://github.com/oobabooga/GPTQ-for-LLaMa -b cuda GPTQ-for-LLaMa cd GPTQ-for-LLaMa python setup_cuda install

I'm on GPTQ for LLaMA folder under repositories says it's pointed at https://github.com/oobabooga/GPTQ-for-LLaMa.git. But I've run through the instructions and also applied the monkey patch to train and apply 4 bit lora which may come into play. No idea.

git clone https://github.com/oobabooga/GPTQ-for-LLaMa.git -b cuda

Install dependencies like hipblas-devel hipsparse-devel hipcub-devel git python3.10 make libstdc++-devel accelerate, rocm and hip git clone https://github.com/bmaltais/kohya_ss && cd kohya_ss python3.10 -m venv venv source venv/bin/activate pip3 install torch==1.13.1 torchvision==0.14.1 torchtext==0.14.1 torchaudio==0.13.1 --index-url https://download.pytorch.org/whl/rocm5.2 # problems on 2.0.0 last I tried, but kohya gotten updates since then pip3 install --upgrade -r requirements.txt pip3 uninstall tensorflow && pip3 install tensorflow-rocm pip uninstall bitsandbytes && git clone https://github.com/broncotc/bitsandbytes-rocm # bitsandbytes not required if not using adam8? cd bitesandbytes-rocm && nano Makefile # Replace all 3 instances of 5.3.0 with 5.4.3 make hip python3 setup.py install

There are a lot of ROCm versions of bitsandbytes. For example this one: https://github.com/broncotc/bitsandbytes-rocm The problem is compatibility with most of the requirements. Kobold does a better job than ooba in offering a more streamlined approach for AMD users.