Rust Inference

Open-source Rust projects categorized as Inference
Edit details
Rust Machine Learning LLM AI Onnx
SaaSHub - Software Alternatives and Reviews
SaaSHub helps you find the best software and product alternatives
www.saashub.com
featured

Top 22 Rust Inference Projects

Rust logo
Inference

  1. typedb

    TypeDB: Built for systems, not records

  2. SaaSHub

    SaaSHub - Software Alternatives and Reviews. SaaSHub helps you find the best software and product alternatives

    SaaSHub logo

  3. ort

    Fast ML inference & training for ONNX models in Rust

    Project mention: I added AI background removal to my image converter in a week, in Rust, no Python | dev.to | 2026-06-29

    Steps 1, 4, 5 are libvips, which I already use everywhere. Step 3 is ONNX Runtime via the ort crate. Step 2 is a tight Rust loop. No Python anywhere.

  4. aici

    AICI: Prompts as (Wasm) Programs

    Project mention: MCP Doesn't Need 30 Tools: It Needs Code | news.ycombinator.com | 2025-08-18

    Oh yeah, it can certainly work, sounds like cool stuff! The question is not whether it can work, but whether it works better than an edit tool using textual search/replace blocks. Not to throw shade on the idea, there's certainly tons of variations that I'm sure haven't yet been tried.

    Yes agentic loop with diagnostic feedback is quite powerful. I'd love to have more controllable structured decode from the big llm providers to skip some sources of needing to loop - something like https://github.com/microsoft/aici

  5. uzu

    A high-performance inference engine for AI models

    Project mention: All Data and AI Weekly #199 - July 21, 2025 | dev.to | 2025-07-21

    Run Fast on Mac local https://github.com/trymirai/uzu

  6. EmbedAnything

    Highly Performant, Modular, Memory Safe and Production-ready Inference, Ingestion and Indexing built in Rust 🦀

  7. pipeless

    An open-source computer vision framework to build and deploy apps in minutes

  8. blindai

    Confidential AI deployment with secure enclaves :lock:

  9. snips-nlu-rs

    Snips NLU rust implementation

    Project mention: Show HN: Open-Source Voice AI Badge Powered by ESP32+WebRTC | news.ycombinator.com | 2025-10-09

    There was one a startup called Snips https://snips.ai/ which made an open-source voice recognition engine running on an RPi.

  10. onnxruntime-rs

    Rust wrapper for Microsoft's ONNX Runtime (version 1.8)

  11. inferx

    InferX: Inference as a Service Platform

    Project mention: We Replaced Our RAG Pipeline With Persistent KV Cache. Here's What We Found. | dev.to | 2026-05-23

    inferx.net

  12. infera

    A DuckDB extension for in-database inference

    Project mention: Show HN: Infera – A DuckDB extension for in-database inference, written in Rust | news.ycombinator.com | 2025-10-01

  13. llama-dfdx

    LLaMa 7b with CUDA acceleration implemented in rust. Minimal GPU memory needed!

  14. Kyanite

    A neural network inference library, written in Rust.

  15. profile

    A physics-grounded, cost-aware optimizer for vLLM. (by jungledesh)

    Project mention: Profile(v2.1.4) physics-aware optimizer for vLLM (31→470 tok/s on A100) | news.ycombinator.com | 2026-06-18

  16. kjarni

    Native and Private ML inference engine, embeddings, classification, reranking, search, and text generation. Rust core with C# bindings. No Python, no ONNX, no CUDA.

    Project mention: Build a Document Search Engine in C# Without Python | dev.to | 2026-02-15

    NuGet: https://www.nuget.org/packages/Kjarni GitHub: https://github.com/olafurjohannsson/kjarni

  17. Isartor

    Open-source Prompt Firewall — deflect up to 95% of redundant LLM traffic before it leaves your infrastructure. Documents: https://isartor-ai.github.io/Isartor/index.html

    Project mention: Isartor – Pure-Rust prompt firewall, deflects 60-95% of LLM traffic | news.ycombinator.com | 2026-03-27

    one I forgot, please visit the benchmark of Isartor and see the deflection rate to reduce LLM tokens: https://github.com/isartor-ai/Isartor/tree/main/benchmarks

  18. llmnop

    A tool for measuring LLM performance metrics.

    Project mention: Show HN: llmnop – Rust CLI for benchmarking LLM endpoints | news.ycombinator.com | 2026-01-04

  19. openuma

    Unified Memory Abstraction Layer for AI Inference on AMD APUs and Intel iGPUs

    Project mention: OpenUMA – bring Apple-style unified memory to x86 AI inference (Rust, Linux) | news.ycombinator.com | 2026-04-03

  20. llmterm

    Your friendly local LLM terminal companion

  21. meteorite

    A fast and simple web server for machine learning models (by deploifai)

  22. mediapipe-rs

    A Rust library for mediapipe tasks for WasmEdge WASI-NN.

  23. kenosis

    Pure Rust static INT8 ONNX quantizer with ReLU-aware QDQ placement for optimal ORT inference fusion

    Project mention: Kenosis – Quantize ONNX models without breaking kernel fusion | news.ycombinator.com | 2026-05-28
NOTE: The open source projects on this list are ordered by number of github stars. The number of mentions indicates repo mentiontions in the last 12 Months or since we started tracking (Dec 2020).

Rust Inference discussion

Log in or Post with

Rust Inference related posts

  • I added AI background removal to my image converter in a week, in Rust, no Python

    1 project | dev.to | 29 Jun 2026

  • We Replaced Our RAG Pipeline With Persistent KV Cache. Here's What We Found.

    1 project | dev.to | 23 May 2026

  • Show HN: We Run 60 Hugging Face Models on 2 GPUs

    2 projects | news.ycombinator.com | 31 Jan 2026

  • Show HN: llmnop – Rust CLI for benchmarking LLM endpoints

    1 project | news.ycombinator.com | 4 Jan 2026

  • Show HN: Digger Solo – Semantic Search and Maps for Local Files

    3 projects | news.ycombinator.com | 25 Jun 2025

  • Show HN: Digger Solo – Semantic File Search and Maps

    3 projects | news.ycombinator.com | 17 Jun 2025

  • Show HN: GPU snapshotting for sub-2s LLM inference – the InferX runtime

    1 project | news.ycombinator.com | 16 May 2025
  • A note from our sponsor - SaaSHub
    www.saashub.com | 20 Jul 2026
    SaaSHub helps you find the best software and product alternatives Learn more →

Index

What are some of the best open-source Inference projects in Rust? This list will help you:

# Project Stars
1 typedb 4,383
2 ort 2,406
3 aici 2,077
4 uzu 1,657
5 EmbedAnything 1,283
6 pipeless 850
7 blindai 511
8 snips-nlu-rs 345
9 onnxruntime-rs 326
10 inferx 230
11 infera 133
12 llama-dfdx 111
13 Kyanite 74
14 profile 49
15 kjarni 33
16 Isartor 21
17 llmnop 12
18 openuma 12
19 llmterm 11
20 meteorite 7
21 mediapipe-rs 0
22 kenosis 0

Sponsored
SaaSHub - Software Alternatives and Reviews
SaaSHub helps you find the best software and product alternatives
www.saashub.com
Loading...