rustformers/llm: Run inference for Large Language Models on CPU, with Rust 🦀🚀🦙

• llm

  41 5,885 9.4 Rust

  An ecosystem of Rust libraries for working with large language models

  

wonnx has done some fantastic work in this regard, so that's where we plan to start once we get there. In terms of general discussion of alternate backends, see this issue.

• wonnx

  18 1,487 6.5 Rust

  A WebGPU-accelerated ONNX inference run-time written 100% in Rust, ready for native and the web

  

wonnx has done some fantastic work in this regard, so that's where we plan to start once we get there. In terms of general discussion of alternate backends, see this issue.

• InfluxDB

  www.influxdata.com sponsored

  Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.

  

• llama-dfdx

  2 94 7.3 Rust

  LLaMa 7b with CUDA acceleration implemented in rust. Minimal GPU memory needed!

Not a maintainer, but dfdx can run llama with CUDA!

Suggest a related project