Pure Rust CPU and OpenCL implementation of LLaMA language model

• rllama

  7 519 6.2 Rust

  Rust+OpenCL+AVX2 implementation of LLaMA inference code

• llama

  184 53,053 8.1 Python

  Inference code for Llama models

  

I noticed that Facebook's reference code https://github.com/facebookresearch/llama/blob/main/llama/model.py is surprisingly short and easy to understand. So I decided to try replicate it in Rust.

• InfluxDB

  www.influxdata.com featured

  Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.

  

• llama.cpp

  773 56,891 10.0 C++

  LLM inference in C/C++

The project that inspired me was this one: https://github.com/ggerganov/llama.cpp although I decided to be rebel and try different technologies since the ML community is all about using PyTorch or Cuda or Apple Silicon or something.

• ggllama

  2 6 10.0 Rust

  Discontinued `ggllama` is a Rust port of ggerganov's llama.cpp.

Here's my implementation if you're interested: https://github.com/philpax/ggllama

• rust-gpu

  82 6,952 7.7 Rust

  🐉 Making Rust a first-class language and ecosystem for GPU shaders 🚧

  

OPs implementation runs OpenCL kernels on the GPU not Rust. You could use rust-gpu to re-implement the kernels in Rust which are converted to SPIR-V and execute via Vulkan.

• llm

  41 5,911 9.4 Rust

  An ecosystem of Rust libraries for working with large language models

  

I also noticed someone shared https://github.com/setzer22/llama-rs/ on this Reddit. I predict there will be more implementations; I followed StableDiffusion developments back when it was released and it didn't take very long for the community to optimize the crap out of it.

Suggest a related project