Stop vibe-debugging. Every exception, every backtrace, grouped so you see patterns, not noise. Learn more →
Top 23 Inference Open-Source Projects
-
Project mention: Can Qwen 3.8 running on your laptop really replace Claude Opus for Agentic coding? | dev.to | 2026-09-11
I use my tool LlamaStash to orchestrate the model and manage the sessions. It is a fast TUI, CLI, daemon, and OpenAI-compatible proxy for running local LLMs via backends like llama.cpp and vLLM. It has a lot of features that make it easy to run local models and manage them, like multiple backends, preset files, named launches, auto start etc.
-
AppSignal
AppSignal knows why the f*#k it crashed. Stop vibe-debugging. Every exception, every backtrace, grouped so you see patterns, not noise.
-
-
DeepSpeed
DeepSpeed is a deep learning optimization library that makes distributed training and inference easy, efficient, and effective.
Project mention: AWS SageMaker HyperPod: Distributed Training for Foundation Models at Scale | dev.to | 2026-01-16DeepSpeed Optimization Library - An open-source library compatible with HyperPod that offers advanced pipeline and system optimizations for LLM training.
-
-
Project mention: Building a Jedi-Style Hand Gesture Interface with TensorFlow.js: Control Your Browser Without Touching Anything | dev.to | 2026-02-09
In this tutorial, I'll show you how to build a production-ready hand gesture control system using TensorFlow.js and MediaPipe Hands that transforms any webcam into a precision input device.
-
sglang
SGLang is a high-performance serving framework for large language models and multimodal models.
I feel like bubble is what this is commonly called in GPU programming circles (e.g. https://github.com/sgl-project/sglang/issues/5593 or any number of other issues). Didn't occur to me that it would be confusing to be honest. But yes stall is maybe a better word.
-
Project mention: Construindo um assistente pessoal 100% local com Ollama, LangChain e voz — e as armadilhas que ninguém conta | dev.to | 2026-07-17
-
Kargo
Stop Scripting Promotions. Start Shipping with Kargo. Kargo automates promotion across dev, staging, and prod with approval gates and verification. Open source, built by the team behind Argo CD. Download now.
-
ncnn
ncnn is a high-performance neural network inference framework optimized for the mobile platform
-
For kernel-level performance tuning you can use the occupancy calculator as pointed out by jplusqualt or you can profile your kernel with Nsight compute which will give you a ton of info.
But for model-wide performance, you basically have to come up with your own calculation to estimate the FLOPs required by your model and based on that figure out how well your model is maxing out the GPU capabilities (MFU/HFU).
Here is a more in-depth example on how you might do this: https://github.com/stas00/ml-engineering/tree/master/trainin...
-
nano-vLLM implements an LLM serving engine in roughly 1,200 lines of Python. Its compact codebase makes the core serving concepts clear and easy to follow, making it a useful starting point for learning how an inference engine works.
-
Project mention: Why I built ts-match: TypeScript branching in the era of coding agents | dev.to | 2026-05-12
I want to be very clear about this: ts-pattern deserves a lot of credit.
-
TensorRT
NVIDIA® TensorRT™ is an SDK for high-performance deep learning inference on NVIDIA GPUs. This repository contains the open source components of TensorRT.
-
Project mention: LLM KV Cache Optimization, Open Model Evaluation, & Agent Engineering Skills for Local Deployment | dev.to | 2026-06-13
Source: https://github.com/LMCache/LMCache
-
amazon-sagemaker-examples
Example 📓 Jupyter notebooks that demonstrate how to build, train, and deploy machine learning models using 🧠 Amazon SageMaker.
-
server
The Triton Inference Server provides an optimized cloud and edge inferencing solution. (by triton-inference-server)
-
Project mention: OpenVINO – open-source toolkit for optimizing and deploying AI inference | news.ycombinator.com | 2025-12-28
-
Project mention: https://github.com/RunanywhereAI/runanywhere-sdks/pull/296 | news.ycombinator.com | 2026-01-25
-
inference
Swap GPT for any LLM by changing a single line of code. Xinference lets you run open-source, speech, and multimodal models on cloud, on-prem, or your laptop — all through one unified, production-ready inference API.
-
Project mention: Open source x 3: GRPO training with OpenEnv, vLLM, and Oumi | news.ycombinator.com | 2025-11-12
-
jetson-inference
Hello AI World guide to deploying deep-learning inference networks and deep vision primitives with TensorRT and NVIDIA Jetson.
-
Project mention: 23 TypeScript Tools for Making Software Explicit in the AI Era | dev.to | 2026-08-21
io-ts makes the boundary between unknown runtime data and typed data explicit.
-
Project mention: Qwen3-Omni-Flash-2025-12-01：a next-generation native multimodal large model | news.ycombinator.com | 2025-12-10
At the moment, no unfortunately. However, to my recent knowledge of open source alternatives, the vLLM team published a separate repository for omni models now:
https://github.com/vllm-project/vllm-omni
I have not yet tested out if this does full speech to speech, but this seems like a promising workspace for omni-modal models.
-
whichllm
Find the local LLM that actually runs and performs best on your hardware. Ranked by real, recency-aware benchmarks, not parameter count. One command, run it instantly.
Source: https://github.com/Andyyyy64/whichllm
-
SaaSHub
SaaSHub - Software Alternatives and Reviews. SaaSHub helps you find the best software and product alternatives
NOTE:
The open source projects on this list are ordered by number of github stars.
The number of mentions indicates repo mentiontions in the last 12 Months or
since we started tracking (Dec 2020).
Inference discussion
Inference related posts
-
SwarmLLM – Every device brings a slice. Together they run the whole model
-
Glq a port trellis quantization of large language models as a vLLM plugin
-
How much faster is macOS CI on dedicated runners? Five open-source projects, measured
-
Inside nano-vLLM: What an RTX 3090 Reveals About LLM Serving
-
Show HN: TensorSharp – Local LLM inference and agent workflows in .NET
-
Show HN: Simurg open-source web search for AI agents that aborts hallucinations
-
We open-sourced SIMURG. Bye-bye hallucinations in local quantized models
-
A note from our sponsor - AppSignal
www.appsignal.com | 13 Sep 2026
Index
What are some of the best open-source Inference projects? This list will help you:
| # | Project | Stars |
|---|---|---|
| 1 | vllm | 91,420 |
| 2 | whisper.cpp | 53,556 |
| 3 | DeepSpeed | 43,091 |
| 4 | ColossalAI | 41,440 |
| 5 | mediapipe | 36,893 |
| 6 | sglang | 35,803 |
| 7 | faster-whisper | 25,343 |
| 8 | ncnn | 23,796 |
| 9 | ml-engineering | 18,950 |
| 10 | nano-vllm | 15,396 |
| 11 | ts-pattern | 15,148 |
| 12 | TensorRT | 13,330 |
| 13 | LMCache | 11,769 |
| 14 | amazon-sagemaker-examples | 10,990 |
| 15 | server | 10,978 |
| 16 | openvino | 10,840 |
| 17 | runanywhere-sdks | 10,283 |
| 18 | inference | 9,568 |
| 19 | oumi | 9,382 |
| 20 | jetson-inference | 8,991 |
| 21 | io-ts | 6,811 |
| 22 | vllm-omni | 6,767 |
| 23 | whichllm | 6,609 |
Sponsored
AppSignal knows why the f*#k it crashed.
Stop vibe-debugging. Every exception, every backtrace, grouped so you see patterns, not noise.
www.appsignal.com