Rust llmops

Open-source Rust projects categorized as llmops
Edit details
Rust LLM AI llms openai
SaaSHub - Software Alternatives and Reviews
SaaSHub helps you find the best software and product alternatives
www.saashub.com
sponsored

Top 15 Rust llmops Projects

Rust logo
llmops

  1. rig

    ⚙️🦀 Build modular and scalable LLM Applications in Rust

    Project mention: Rig – Agentic Workflows in Rust | news.ycombinator.com | 2026-08-29

  2. Kargo

    Stop Scripting Promotions. Start Shipping with Kargo. Kargo automates promotion across dev, staging, and prod with approval gates and verification. Open source, built by the team behind Argo CD. Download now.

    Kargo logo

  3. plano

    Plano is an AI-native proxy server and data plane for agentic apps. Smart LLM routing, observability, agent orchestration, and guardrails so you stay focused on your agents core logic.

    Project mention: Show HN: Signals – finding the most informative agent traces without LLM judges | news.ycombinator.com | 2026-04-04

    Project where Signals are already implemented: https://github.com/katanemo/plano

    Happy to answer questions on the taxonomy, implementation details, or where this breaks down.

  4. bionic-gpt

    Bionic is sovereign AI for the enterprise — ChatGPT-like AI that runs on-premise and can securely work with your sensitive data and systems.

  5. aici

    AICI: Prompts as (Wasm) Programs

  6. VectorChord

    Scalable, fast, and disk-friendly vector search in Postgres, the successor of pgvecto.rs.

    Project mention: The Case Against PGVector | news.ycombinator.com | 2025-11-03

    - We support hybrid search with BM25 through https://github.com/tensorchord/VectorChord-bm25

    The author simplifies the complexity of synchronizing between an existing database and a specialized vector database, as well as how to perform joint queries on them. This is also why we see most users choosing vector solution on PostgreSQL.

  7. paddler

    Open-source LLM/VLM load balancer and serving platform for self-hosting LLMs (and VLMs) at scale 🏓🦙 Alternative to projects like llm-d, Docker Model Runner, etc but with less moving parts and simple deployments built around ggml ecosystem. Runs on CPU and GPU.

  8. motorhead

    🧠 Motorhead is a memory and information retrieval server for LLMs.

  9. AppSignal

    AppSignal knows why the f*#k it crashed. Stop vibe-debugging. Every exception, every backtrace, grouped so you see patterns, not noise.

    AppSignal logo

  10. swiftide

    Fast, streaming indexing, query, and agentic LLM applications in Rust

    Project mention: Announcing Swiftide 0.31 | dev.to | 2025-09-16

    Full write up on all the things in this release at our blog and on github.

  11. nexus

    Govern & Secure your AI (by Nexus-Router)

    Project mention: Claude Code 2.0 | news.ycombinator.com | 2025-09-29

    You could try to use a router. I'm currently building this:

    https://github.com/grafbase/nexus/

    If croq talks openai API, you enable the anthropic protocol, and openai provider with a base url to croq. Set ANTHROPIC_BASE_URL to the open endpoint and start claude.

    I haven't tested croq yet, but this could be an interesting use case...

  12. busbar

    Point your existing SDK at one URL and reach every LLM vendor — with real failover, not a try/except. One static Rust binary.

    Project mention: Show HN: Busbar – every LLM behind one URL, in a single Rust binary | news.ycombinator.com | 2026-06-05

  13. trigix

    AI-native workflow automation platform — Rust execution engine, 180 node types, visual canvas

    Project mention: Building a self-hosted, AI-native workflow engine in Rust (180 node types, no SDK bloat) | dev.to | 2026-06-16

    Repo: https://github.com/bj-qizhi/trigix — feedback on the engine design and the node model especially welcome.

  14. Cachet

    A transparent, 100%-local semantic cache for LLM APIs — drop-in proxy, one line to integrate, written in Rust (by abhix2112)

    Project mention: Show HN: Cachet – A drop-in semantic cache for LLM APIs, 100% local, in Rust | news.ycombinator.com | 2026-06-23

  15. suture-stream-repair

    Ultra-low-latency reverse proxy that repairs truncated & malformed JSON in LLM streaming responses (OpenAI, Anthropic, Vertex AI, Bedrock) — fixes JSONDecodeError / serde_json EOF on truncated tool calls.

    Project mention: Show HN: Suture – a reverse proxy that repairs truncated JSON in LLM streams | news.ycombinator.com | 2026-06-03

  16. peon-lib

    A lightweight, secure executor for Claude Agent Skills. Inspired by OpenClaw but strictly designed with zero-trust whitelisting, written in Rust.

    Project mention: Peon – A Zero-Trust AI Agent Runtime in Rust (Using Casbin) | news.ycombinator.com | 2026-04-17

    3. The Intercept: When the LLM decides to call a script, the runtime intercepts the tool call, grabs the user's ID, and asks Casbin. If it's a deny, the OS is never touched, and the LLM gets a permission error.

    To solve multi-tenant concurrency issues without locking up states, the runtime spawns stateless, ephemeral agents per request, hardcoding the user's ID into the tool struct at build time.

    It's currently integrated with a CLI and a multi-user Telegram bot.

    I built this out of frustration with existing frameworks that just give LLMs a shell and pray. I would love to hear your thoughts on the architecture, and if using Casbin for this level of isolation makes sense to you.

    Repo: https://github.com/stephen94125/peon-lib

  17. recondo

    AI governance gateway. Wire-level LLM observability — every prompt, every tool call, every response.

    Project mention: Recondo – Local MitM Proxy for Coding Agents (Claude Code, Codex, Gemini) | news.ycombinator.com | 2026-05-06

  18. SaaSHub

    SaaSHub - Software Alternatives and Reviews. SaaSHub helps you find the best software and product alternatives

    SaaSHub logo
NOTE: The open source projects on this list are ordered by number of github stars. The number of mentions indicates repo mentiontions in the last 12 Months or since we started tracking (Dec 2020).

Rust llmops discussion

Log in or Post with

Rust llmops related posts

  • AI OSS tool repo goes archived over night after raising $7.3M Seed

    5 projects | news.ycombinator.com | 13 Jun 2026

  • Show HN: Paddler – open-source LLMOps platform for hosting AI in your own infra

    1 project | news.ycombinator.com | 15 Aug 2025

  • BionicGPT: Transforming AI Collaboration and Deployment

    1 project | dev.to | 9 Jan 2025

  • Ask HN: RAG as a Service?

    1 project | news.ycombinator.com | 22 May 2024

  • Ask HN: How to structure Rust, Axum, and SQLx for clean architecture?

    2 projects | news.ycombinator.com | 7 May 2024

  • Ask HN: What's wrong/right with Postgres migrations?

    1 project | news.ycombinator.com | 11 Mar 2024

  • When will "local" LLMs hit?

    1 project | news.ycombinator.com | 5 Mar 2024
  • A note from our sponsor - Kargo
    akuity.io | 8 Sep 2026
    Kargo automates promotion across dev, staging, and prod with approval gates and verification. Open source, built by the team behind Argo CD. Download now. Learn more →

Index

What are some of the best open-source llmops projects in Rust? This list will help you:

# Project Stars
1 rig 8,508
2 plano 7,037
3 bionic-gpt 2,363
4 aici 2,077
5 VectorChord 1,789
6 paddler 1,665
7 motorhead 917
8 swiftide 777
9 nexus 435
10 busbar 120
11 trigix 4
12 Cachet 3
13 suture-stream-repair 1
14 peon-lib 0
15 recondo 0

Sponsored
Stop Scripting Promotions. Start Shipping with Kargo
Kargo automates promotion across dev, staging, and prod with approval gates and verification. Open source, built by the team behind Argo CD. Download now.
akuity.io
Loading...