Rust llm-inference

Open-source Rust projects categorized as llm-inference
Edit details
Rust llms AI LLM Cuda
Monitoring that respects your time & budget
APM, error tracking, and dashboards for modern web apps. Ten-minute setup, transparent flat pricing, and support from engineers who actually use the product.
www.appsignal.com
sponsored

Top 12 Rust llm-inference Projects

Rust logo
llm-inference

  1. plano

    Plano is an AI-native proxy server and data plane for agentic apps. Smart LLM routing, observability, agent orchestration, and guardrails so you stay focused on your agents core logic.

    Project mention: Show HN: Signals – finding the most informative agent traces without LLM judges | news.ycombinator.com | 2026-04-04

    Project where Signals are already implemented: https://github.com/katanemo/plano

    Happy to answer questions on the taxonomy, implementation details, or where this breaks down.

  2. AppSignal

    Monitoring that respects your time & budget. APM, error tracking, and dashboards for modern web apps. Ten-minute setup, transparent flat pricing, and support from engineers who actually use the product.

    AppSignal logo

  3. shimmy

    ⚡ Pure-Rust WebGPU inference engine — OpenAI-API compatible, GGUF native, runs on any GPU. No Python. No llama.cpp. Single binary.

    Project mention: Shimmy v1.7.0: Running 42B Moe Models on Consumer GPUs with 99.9% VRAM Reduction | news.ycombinator.com | 2025-10-08

  4. RuVector

    RuVector is a High Performance, Real-Time, Self-Learning Ai, Vector GNN, Memory DB built in Rust.

    Project mention: High Performance, Real-Time, Self-Learning, Vector GNN and DB Built in Rust | news.ycombinator.com | 2026-02-26

  5. aici

    AICI: Prompts as (Wasm) Programs

  6. blast

    Open-source VMs-as-a-service (by stanford-mast)

  7. hipfire

    RDNA-native LLM inference engine in Rust.

    Project mention: Qwen 3.6 27B is the sweet spot for local development | news.ycombinator.com | 2026-06-29

    I usually doubt the 'small dataset tuned' variants. B/c ages ago (in the NN prehistory) I've done some NN training, and appreciate how hard it is to improve in general, and how easy it is to ruin a model in general while targeting a small dataset (LoRA-s are ok, that's different). That model/quant was the most recent one I was trying. But could not really use any of them, as the combo model + llama-server ground to a halt even at small context depth sizes on the amd gpu.

    Yesterday I finally found a good combo! So writing this for the benefit for anyone that may have the same h/w. Got around to /GOAL search for something better for the h/w (amd 7900xtx), and pi agent found a new best that actually seems it will be useful for real. As the 40 tok/s speed starts dropping only at 260K context depth?? Served by hipfire from this repo https://github.com/Kaden-Schutt/hipfire, that worked the best got on llama-benchy:

    | Context | Wall Time | PP (t/s) | TG (t/s) |

  8. pmetal

    PMetal: high-performance Apple Silicon framework for local LLM inference, LoRA/QLoRA fine-tuning, serving, quantization, and MLX/Metal acceleration.

    Project mention: PMetal – (Powdered Metal) LLM Fine-Tuning Framework for Apple Silicon | news.ycombinator.com | 2026-03-16

  9. Kargo

    Stop Scripting Promotions. Start Shipping with Kargo. Kargo automates promotion across dev, staging, and prod with approval gates and verification. Open source, built by the team behind Argo CD. Download now.

    Kargo logo

  10. qwen3-rs

    An educational Rust project for exporting and running inference on Qwen3 LLM family

  11. atlas

    Pure Rust Inference Engine (by Avarok-Cybersecurity)

    Project mention: Atlas: An LLM inference engine written from scratch in Rust and CUDA | news.ycombinator.com | 2026-05-12

  12. saient-quartz

    Quartz — custom CUDA/Vulkan GGUF inference engine + SD1.5/SDXL image and video generation (Rust, no llama.cpp)

    Project mention: Open-sourced a local WAN 2.2 mobile pipeline for on-device video generation | news.ycombinator.com | 2026-08-01

  13. quaynor

    Embed local LLMs in your app

    Project mention: Easiest way to create agents with local LLMs | news.ycombinator.com | 2026-05-04

  14. gguf-switchboard

    Run GGUF through llama.cpp and SafeTensors through vLLM behind one OpenAI-compatible endpoint. Your coding tools select a model; the switchboard manages the local runtime, process, and resident-model change.

    Project mention: My RTX 3060 Can Run Many LLMs — Just Not at the Same Time | dev.to | 2026-09-03

    GGUF Switchboard is open source: GitHub — GGUF Switchboard

NOTE: The open source projects on this list are ordered by number of github stars. The number of mentions indicates repo mentiontions in the last 12 Months or since we started tracking (Dec 2020).

Rust llm-inference discussion

Log in or Post with

Rust llm-inference related posts

Index

What are some of the best open-source llm-inference projects in Rust? This list will help you:

# Project Stars
1 plano 7,037
2 shimmy 5,839
3 RuVector 4,474
4 aici 2,077
5 blast 799
6 hipfire 599
7 pmetal 310
8 qwen3-rs 43
9 atlas 12
10 saient-quartz 5
11 quaynor 3
12 gguf-switchboard 1

Sponsored
Monitoring that respects your time & budget
APM, error tracking, and dashboards for modern web apps. Ten-minute setup, transparent flat pricing, and support from engineers who actually use the product.
www.appsignal.com

Did you know that Rust is
the 3rd most popular programming language
based on number of references?

Loading...