Anyone actually running 30b/65b at reasonably high speed? What's your rig?

• llama.cpp

  1 1 9.4 C

  Port of Facebook's LLaMA model in C/C++ (by sw)

It might be worth trying out 2/3 bit quantization on llama.cpp. Currently sitting in an unmerged pr, but it works. I doubt you’ll be getting 5+ tokens/second though. link

• GPTQ-for-LLaMa

  19 129 7.7 Python

  4 bits quantization of LLaMa using GPTQ (by oobabooga)

I'm on GPTQ for LLaMA folder under repositories says it's pointed at https://github.com/oobabooga/GPTQ-for-LLaMa.git. But I've run through the instructions and also applied the monkey patch to train and apply 4 bit lora which may come into play. No idea.

• InfluxDB

  www.influxdata.com sponsored

  Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.

  

Suggest a related project