Rust Inference

Open-source Rust projects categorized as Inference

Top 12 Rust Inference Projects

  1. typedb

    TypeDB: the power of programming, in your database

    Project mention: TypeDB: A Next-Gen Database | news.ycombinator.com | 2024-12-22
  2. CodeRabbit

    CodeRabbit: AI Code Reviews for Developers. Revolutionize your code reviews with AI. CodeRabbit offers PR summaries, code walkthroughs, 1-click suggestions, and AST-based analysis. Boost productivity and code quality across all major languages with each PR.

    CodeRabbit logo
  3. aici

    AICI: Prompts as (Wasm) Programs

    Project mention: What Is ChatGPT Doing and Why Does It Work? | news.ycombinator.com | 2024-06-18

    That’s right, if LLMs were really thinking/forming world models etc. we would expect them to be robust against word choice or phrasing. But in practice anyone using RAG can tell you that that is not the case.

    I’m just a practitioner so my language might be imprecise but when I say similarly structured sentences what I mean is, and this is my interpretation based on my experience with using Agents and LLMs, that the shape of the context as in the phrasing and the word choice highly bias the outputs of LLMs.

    In my own observations at work, those who interpret LLMs to be thinking often produce bad agents. LLM are not good at open ended questions, if you ask an LLM “improve this code” you will often get bad results that just look passable. But if you interpret LLMs as probabilistic models highly biased by their context then you would add a lot more context and specific instructions in the prompt in order to get the Agent to produce the right output.

    Side note, this is also why I like the AICI approach: https://github.com/microsoft/aici

  4. ort

    Fast ML inference & training for ONNX models in Rust (by pykeio)

    Project mention: AI Inference now available in Supabase Edge Functions | dev.to | 2024-04-16

    To solve this, we built a native extension in Edge Runtime that enables using ONNX runtime via the Rust interface. This was made possible thanks to an excellent Rust wrapper called Ort:

  5. pipeless

    An open-source computer vision framework to build and deploy apps in minutes

    Project mention: Vision AI agents for any task | dev.to | 2024-04-30

    After spending some months working on the Pipeless open-source framework, today I bring something new and really cool: Pipeless Agents

  6. blindai

    Confidential AI deployment with secure enclaves :lock:

  7. snips-nlu-rs

    Snips NLU rust implementation

  8. onnxruntime-rs

    Rust wrapper for Microsoft's ONNX Runtime (version 1.8)

  9. Nutrient

    Nutrient - The #1 PDF SDK Library. Bad PDFs = bad UX. Slow load times, broken annotations, clunky UX frustrates users. Nutrient’s PDF SDKs gives seamless document experiences, fast rendering, annotations, real-time collaboration, 100+ features. Used by 10K+ devs, serving ~half a billion users worldwide. Explore the SDK for free.

    Nutrient logo
  10. llama-dfdx

    LLaMa 7b with CUDA acceleration implemented in rust. Minimal GPU memory needed!

  11. Kyanite

    A neural network inference library, written in Rust.

  12. meteorite

    A fast and simple web server for machine learning models (by deploifai)

  13. fastllm

    FastLLM - Rust based LLM Inference API

    Project mention: Floating Point Precision: Understanding FP64, FP32, and FP16 in Large Language Models | dev.to | 2025-02-08

    Recently, I started working on my own inference API as a side project and found myself grappling with torch_dtype , which determines the data type of a Tensor. Each dtype is a floating point and values are usually defined in a model's config.json, but I wanted to dig deeper into how they work. I figured it was worth learning more and sharing my findings with others.

  14. mediapipe-rs

    A Rust library for mediapipe tasks for WasmEdge WASI-NN.

NOTE: The open source projects on this list are ordered by number of github stars. The number of mentions indicates repo mentiontions in the last 12 Months or since we started tracking (Dec 2020).

Rust Inference discussion

Log in or Post with

Rust Inference related posts

  • What Is ChatGPT Doing and Why Does It Work?

    1 project | news.ycombinator.com | 18 Jun 2024
  • Google Gemini: Context Caching

    1 project | news.ycombinator.com | 16 May 2024
  • AI Inference Now Available in Supabase Edge Functions

    1 project | news.ycombinator.com | 16 Apr 2024
  • Meta's Segment Anything written with C++ / GGML

    4 projects | news.ycombinator.com | 5 Sep 2023
  • Machine Learning Inference Server in Rust?

    4 projects | /r/rust | 21 Mar 2023
  • A note from our sponsor - CodeRabbit
    coderabbit.ai | 16 Feb 2025
    Revolutionize your code reviews with AI. CodeRabbit offers PR summaries, code walkthroughs, 1-click suggestions, and AST-based analysis. Boost productivity and code quality across all major languages with each PR. Learn more →

Index

What are some of the best open-source Inference projects in Rust? This list will help you:

# Project Stars
1 typedb 3,921
2 aici 1,998
3 ort 1,119
4 pipeless 736
5 blindai 505
6 snips-nlu-rs 341
7 onnxruntime-rs 287
8 llama-dfdx 102
9 Kyanite 59
10 meteorite 8
11 fastllm 5
12 mediapipe-rs 0

Sponsored
CodeRabbit: AI Code Reviews for Developers
Revolutionize your code reviews with AI. CodeRabbit offers PR summaries, code walkthroughs, 1-click suggestions, and AST-based analysis. Boost productivity and code quality across all major languages with each PR.
coderabbit.ai

Did you know that Rust is
the 5th most popular programming language
based on number of references?