[Project] Alpaca-30B: Facebook's 30b parameter LLaMa fine-tuned on the Alpaca dataset

• llama.cpp

  766 55,117 9.9 C++

  LLM inference in C/C++

  You can run llama-30B on a CPU using llama.cpp, it's just slow. The alpaca models I've seen are the same size as the llama model they are trained on, so I would expect running the alpaca-30B models will be possible on any system capable of running llama-30B.

• alpaca.cpp

  94 9,878 9.4 C

  Discontinued Locally run an Instruction-Tuned Chat-Style LLM

  LLaMa/Alpaca work just fine on CPU with llama.cpp/alpaca.cpp. Not very snappy but fast enough for me.

• InfluxDB

  www.influxdata.com

  sponsored

  Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.

  

• text-generation-webui

  876 35,583 9.9 Python

  A Gradio web UI for Large Language Models. Supports transformers, GPTQ, AWQ, EXL2, llama.cpp (GGUF), Llama models.

  You are absolutely correct. text-gen-webui offers "streaming" via paging models in and out of VRAM. Using this your CPU no longer gets bogged down with running the model, but you don't see much improvement in generation speed as the GPU is churning with loading and unloading model data from main RAM all the time.

Suggest a related project