AutoGPTQ VS Dependencies

Deploying the 60B version is a challenge though and you might need to apply 4-bit quantization with something like https://github.com/PanQiWei/AutoGPTQ or https://github.com/qwopqwop200/GPTQ-for-LLaMa . Then you can improve the inference speed by using https://github.com/turboderp/exllama .

If you prefer to use an "instruct" model à la ChatGPT (i.e. that does not need few-shot learning to output good results) you can use something like this: https://huggingface.co/TheBloke/Wizard-Vicuna-30B-Uncensored...

AutoGPTQ: an attempt at standardizing GPTQ-for-LLaMa and turning it into a library that is easier to install and use, and that supports more models. https://github.com/PanQiWei/AutoGPTQ

Just use the script: https://github.com/PanQiWei/AutoGPTQ/blob/main/examples/quantization/quant_with_alpaca.py

In the wild, people tend to use GTPQ quantization for pure GPU inference: https://github.com/PanQiWei/AutoGPTQ

And ggml's quant for CPU inference with some offload, which just got updated to a more GPTQ-like method days ago: https://github.com/ggerganov/llama.cpp/pull/1684

Some other runtimes like Apache TVM also have their own quant implementations: https://github.com/mlc-ai/mlc-llm

For training, 4-bit bitsandbytes is SOTA, as far as I know.

TBH I'm not sure why this November paper is being linked. Few are running 8 bit models when they could fit a better 3-5 bit model in the same memory pool.

Instead of integrating GPTQ-for-Lllama, use AutoGPTQ instead.

I did some digging with Dependencies and found that issue is with libstdc++-6.dll

Maybe Dependency Walker can shed some light on that 🤔

I did that, basically. Problem is there is a clblast.dll (on windows) that llama.dll depends on, and it llama-cpp-python always failed dependency resolve to find it. I copied the dll to the right folder, loading it manually via CDLL worked fine, and https://github.com/lucasg/Dependencies also confirmed the dll was findable. When loading DLL's in windows, it checks the same folder for dependency dll's (and a few other places).

You can uses dependencies walker to detect the exact version of MS c++ runtime required.

If that doesn't work you will have to do it the hard way like i did. Using Dependencies to find the missing dlls

("Dependencies" is Dependencies.exe from https://github.com/lucasg/Dependencies)

If you're using MinGW or Cygwin and link in an arbitrary number of system libraries, then you need to ship those files as well. You can use Dependencies to list all DLLs your program is using, including transient dependencies.

A couple of them, yes. Someone else linked Dependencies which is much more modern and doesn't have some of the issues these older applications have. Thank you for the suggestions regardless.