[P] rwkv.cpp: FP16 & INT4 inference on CPU for RWKV language model

• rwkv.cpp

  12 1,111 6.8 C++

  INT4/INT5/INT8 and FP16 inference on CPU for RWKV language model

  

• cformers

  4 315 6.7 C

  SoTA Transformers with C-backend for fast inference on your CPU.

  

it's a combination of things, and removing python from the loop isn't essential to achieving most of these performance gains. the main trick is quantizing the weights and compiling the model. concrete example that builds on top of ggml with python APIs: https://github.com/NolanoOrg/cformers

• InfluxDB

  www.influxdata.com featured

  Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.

  

• RWKV-LM

  84 11,747 8.8 Python

  RWKV is an RNN with transformer-level LLM performance. It can be directly trained like a GPT (parallelizable). So it's combining the best of RNN and transformer - great performance, fast inference, saves VRAM, fast training, "infinite" ctx_len, and free sentence embedding.

Author of RWKV shows that an X billion model is comparable to an X billion GPT model: https://github.com/BlinkDL/RWKV-LM/blob/main/RWKV-eval2.png

• whisper.cpp

  187 31,649 9.8 C

  Port of OpenAI's Whisper model in C/C++

• llama.cpp

  780 57,984 10.0 C++

  LLM inference in C/C++

• llm

  41 5,954 9.4 Rust

  An ecosystem of Rust libraries for working with large language models

  

• alpaca.cpp

  94 9,878 9.4 C

  Discontinued Locally run an Instruction-Tuned Chat-Style LLM

• SaaSHub

  www.saashub.com featured

  SaaSHub - Software Alternatives and Reviews. SaaSHub helps you find the best software and product alternatives

  

• gpt4all.cpp

  2 510 6.5 C

  Locally run an Assistant-Tuned Chat-Style LLM

Suggest a related project