GPTQ-for-LLaMa VS SillyTavern

git clone https://github.com/oobabooga/GPTQ-for-LLaMa

If you're using the model directly from ehartford, that one hasn't been quantized. Try using the GPTQ quantized version here, and use this fork of GPTQ-for-LLaMa. Load in 4-bit with --wbits 4

This worked for me on Ubuntu. If you want to use the CUDA branch instead of triton, do the same steps except clone this GPTQ-for-LLaMa fork and run python setup_cuda.py install

If you don't have triton and you use AutoGPTQ you're gonna notice a huge slow down compared to the old GPTQ-for-LLaMA cuda branch. For me AutoGPTQ gives me a whopping 1 token per second compared to the old GPTQ that gives me a decent 9 tokens per second.. both times I used a same sized model. (I think the slowdown is due to AutoGPTQ using the newer cuda branch which is much slower than the old one)

Are you using a later version of GPTQ-for-LLaMa? If so, go to ooba's CUDA fork (https://github.com/oobabooga/GPTQ-for-LLaMa). That's what I made it in and it definitely works with that. And that's what's included in the one-click-installers.

This usually happens when using models that conflict with your GPTQ installation. You should be using this fork: https://github.com/oobabooga/GPTQ-for-LLaMa. If you did the manual installation wrong, use the one click installer instead.

cd text-generation-webui # wherever you have it installed mkdir -p repositories cd repositories git clone https://github.com/oobabooga/GPTQ-for-LLaMa -b cuda GPTQ-for-LLaMa cd GPTQ-for-LLaMa python setup_cuda install

I'm on GPTQ for LLaMA folder under repositories says it's pointed at https://github.com/oobabooga/GPTQ-for-LLaMa.git. But I've run through the instructions and also applied the monkey patch to train and apply 4 bit lora which may come into play. No idea.

git clone https://github.com/oobabooga/GPTQ-for-LLaMa.git -b cuda

Go to Termix and click Exit. Then go to Termux and code 1. Apk update 2. Apk upgrade 3. git clone https://github.com/Cohee1207/SillyTavern 4. cd SillyTavern 5. Install nodejs 6. Npm install 7. Node server

If you want the best roleplaying experience, I can only recommend SillyTavern with SillyTavern/SillyTavern-extras. The extras include summarization and ChromaDB, both helping to get longer and more coherent chats.

Really? Then we definitely have different experiences (or different ways to interact) with Guanaco. It's been the most unrestricted model I've tried, and I tried them all, but I'm using SillyTavern and the simple-proxy-for-tavern which combined with a little prompting liberates basically any model.

Why reinvent the wheel? Just use SillyTavern, ideally with the simple-proxy-for-tavern. That does it all, and more.

I tested this today in an hours-long direct roleplay comparison between q3_K_M quants of TheBloke/airoboros-65B-gpt4-1.2-GGML and TheBloke/guanaco-65B-GGML, using koboldcpp as backend together with simple-proxy-for-tavern and SillyTavern as frontend.

I'm using SillyTavern frontend and simple-proxy-for-tavern with koboldcpp backend.

I'm using SillyTavern frontend and simple-proxy-for-tavern with koboldcpp. Not sure which of these has solved the prompt-reprocessing problem, but I no longer have these slowdowns.

WizardLM 30B V1.0 is not only smarter and follows instructions better than the others, it's even uncensored when used with an uncensoring character card (I use SillyTavern as my GUI/frontend) - more so than any other model I tested. Probably because it follows instructions so well, thus roleplaying an uncensored character properly (and not breaking character or going "as an AI" even once during my tests).

curl -o- https://raw.githubusercontent.com/nvm-sh/nvm/v0.39.3/install.sh | bash export NVM_DIR="$([ -z "${XDG_CONFIG_HOME-}" ] && printf %s "${HOME}/.nvm" || printf %s "${XDG_CONFIG_HOME}/nvm")" [ -s "$NVM_DIR/nvm.sh" ] && \. "$NVM_DIR/nvm.sh" nvm install node git clone -b dev https://github.com/Cohee1207/SillyTavern && cd SillyTavern npm i && node server.js

For Android (Termux users): 1. apt update 2. apt upgrade 3. Type "y" to everything and hit enter 4. pkg install git 5. git clone -b dev https://github.com/Cohee1207/SillyTavern 6. cd SillyTavern 7. pkg install nodejs 8. npm install 9. bash start.sh