llama-cpu VS bitsandbytes-win-prebuilt

You've probably heard, but LLaMA just released, and its 13B parameter model outperforms GPT-3 on most metrics (because they trained it on a lot more data). Someone's already quantized it to 4 and 3 bits and it performs virtually the same. It also apparently performs well on CPUs (several words per second on a 7900X). Running something equivalent to GPT3.5 on a phone is not out that far out.

I tried to port the llama-cpu version to a gpu-accelerated mps version for macs, it runs, but the outputs are not as good as expected and it often gives "-1" tokens. Any help and contributions on fixing it are welcome!

You can run it with only a CPU and 32 gigs of RAM: https://github.com/markasoftware/llama-cpu

I was able to run 7B on a CPU, inferring several words per second: https://github.com/markasoftware/llama-cpu

So there used to be a compiled version from https://github.com/DeXtmL/bitsandbytes-win-prebuilt but now I see there is a new version (from last week) at https://github.com/acpopescu/bitsandbytes/releases which appears to maybe become the start of Windows support in the official repo?

put libbitsandbytes_cuda116.dll in C:\Users\xxx\miniconda3\envs\textgen\lib\site-packages\bitsandbytes\

Download these 2 dll files from here. then you move those files into "installer_files\env\lib\site-packages\bitsandbytes\" under your oobabooga root folder (where you've extracted the oneclick installer)