High-Speed Large Language Model Serving on PCs with Consumer-Grade GPUs

• PowerInfer

  4 6,978 9.8 C++

  High-speed Large Language Model Serving on PCs with Consumer-grade GPUs

  

• Cgml

  22 39 8.6 C++

  GPU-targeted vendor-agnostic AI library for Windows, and Mistral model implementation.

Since they mentioned they’re working on Mistral-7B, I’d like to note that my GPU-only implementation of Mistral uses slightly over 5GB of VRAM: https://github.com/Const-me/Cgml Runs pretty good on most consumer-grade GPUs.

• InfluxDB

  www.influxdata.com featured

  Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.

  

Suggest a related project