How to install LLaMA: 8-bit and 4-bit

• text-generation-webui

  876 36,552 9.9 Python

  A Gradio web UI for Large Language Models. Supports transformers, GPTQ, AWQ, EXL2, llama.cpp (GGUF), Llama models.

I see the GPTQ-for-LLaMA repo received an update today that people are saying is causing several problems, but it shouldn't have caused that error from my own testing. Even on the nonworking repo update, I was able to at least complete setup. Can you share the more of the log, for example, are you seeing "No CUDA runtime is found"?

• GPTQ-for-LLaMa

  75 2,916 8.6 Python

  4 bits quantization of LLaMA using GPTQ

git clone https://github.com/qwopqwop200/GPTQ-for-LLaMa cd GPTQ-for-LLaMa python setup_cuda.py install

• InfluxDB

  www.influxdata.com featured

  Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.

  

• bitsandbytes

  61 5,447 9.4 Python

  Accessible large language models via k-bit quantization for PyTorch.

Welcome to bitsandbytes. For bug reports, please submit your error trace to: https://github.com/TimDettmers/bitsandbytes/issues

• text-generation-webui

  3 5 9.0 Python

  A gradio web UI for running Large Language Models like GPT-J 6B, OPT, GALACTICA, GPT-Neo, and Pygmalion. (by TheTerrasque)

I've put together a more automated setup, maybe you'll have more luck with that: https://github.com/TheTerrasque/text-generation-webui

• alpaca.cpp

  94 9,878 9.4 C

  Discontinued Locally run an Instruction-Tuned Chat-Style LLM

Alternatively, if you want a quick and easy way to test Alpaca on Mac, you can use alpaca.cpp here.

• SaaSHub

  www.saashub.com featured

  SaaSHub - Software Alternatives and Reviews. SaaSHub helps you find the best software and product alternatives

  

Suggest a related project