Rust Inference

Open-source Rust projects categorized as Inference

Top 7 Rust Inference Projects

  • aici

    AICI: Prompts as (Wasm) Programs

    Project mention: LLM4Decompile: Decompiling Binary Code with LLM | | 2024-03-17

    I have been planning to work on something like this. I think that eventually, someone will crack the "binary in -> good source code out of LLM" pipeline but we are probably a few years away from that still. I say a few years because I don't think there's a huge pile of money sitting at the end of this problem, but maybe I'm wrong.

    A really good "stop-gap" approach would be to build a decompilation pipeline using Ghidra in headless mode and then combine the strict syntax correctness of a decompiler with the "intuition/system 1 skills" of an LLM. My inspiration for this setup comes from two recent advancements, both shared here on HN:

    1. AlphaGeometry: The Decompiler and the LLM should complement each other, covering each other's weaknesses.

    2. AICI: We need a better way of "hacking" on top of these models, and being able to use something like AICI as the "glue" to coordinate the generation of C source. I don't really want the weights of my LLM to be used to generate syntactically correct C source, I want the LLM to think in terms of variable names, "snippet patterns" and architectural choices while other tools (Ghidra, LLVM) worry about the rest.

    Obviously this is all hand-wavey armchair commentary from a former grad student who just thinks this stuff is cool. Huge props to these researchers for diving into this. I know the authors already mentioned incorporating Ghidra into their future work, so I know they're on the right track.

  • ort

    A Rust wrapper for ONNX Runtime (by pykeio)

    Project mention: Moving from Typescript and Langchain to Rust and Loops | | 2023-09-07

    In the quest for more efficient solutions, the ONNX runtime emerged as a beacon of performance. The decision to transition from Typescript to Rust was an unconventional yet pivotal one. Driven by Rust's robust parallel processing capabilities using Rayon and seamless integration with ONNX through the ort crate, Repo-Query unlocked a realm of unparalleled efficiency. The result? A transformation from sluggish processing to, I have to say it, blazing-fast performance.

  • InfluxDB

    Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.

  • blindai

    Confidential AI deployment with secure enclaves :lock:

  • snips-nlu-rs

    Snips NLU rust implementation

  • onnxruntime-rs

    Rust wrapper for Microsoft's ONNX Runtime (version 1.8)

  • meteorite

    A fast and simple web server for machine learning models (by deploifai)

  • mediapipe-rs

    A Rust library for mediapipe tasks for WasmEdge WASI-NN.

    Project mention: Release: WasmEdge 0.12 and 0.12.1 | | 2023-05-28

    Some examples:

    WasmEdge’s Tensorflow Lite plugin enables lightweight (1/10 of Linux containers) and fast (native GPU) AI inference apps for Tensorflow Lite models.

    WasmEdge’s PyTorch plugin enables lightweight (1/10 of Linux containers) and fast (native GPU) AI inference apps for PyTorch models.

    Through the new WasmEdge plugin system, the community are adding support for libraries like zlib, OpenCV, tesseract and ffmpeg etc. Putting these together, we are supporting complex AI libraries such as Mediapipe on WasmEdge!

    The Mediapipe story is esp interesting since WasmEdge is now adapted as a runtime for stream data process in products like Fluvio, Redpanda, YoMo, RisingWave and others. Mediapipe support allows developers to add AI into the stream data pipeline.

    WasmEdge’s eBPF plugin enables developers to create secure and containerized eBPF tools and controllers in Kubernetes environments.

  • WorkOS

    The modern identity platform for B2B SaaS. The APIs are flexible and easy-to-use, supporting authentication, user identity, and complex enterprise features like SSO and SCIM provisioning.

NOTE: The open source projects on this list are ordered by number of github stars. The number of mentions indicates repo mentiontions in the last 12 Months or since we started tracking (Dec 2020). The latest post mention was on 2024-03-17.

Rust Inference related posts


What are some of the best open-source Inference projects in Rust? This list will help you:

Project Stars
1 aici 1,648
2 ort 527
3 blindai 489
4 snips-nlu-rs 337
5 onnxruntime-rs 261
6 meteorite 8
7 mediapipe-rs 0
SaaSHub - Software Alternatives and Reviews
SaaSHub helps you find the best software and product alternatives