APM, error tracking, and dashboards for modern web apps. Ten-minute setup, transparent flat pricing, and support from engineers who actually use the product. Learn more →
Top 23 Python llm-inference Projects
-
Ray
Ray is an AI compute engine. Ray consists of a core distributed runtime and a set of AI Libraries for accelerating ML workloads.
Project mention: GSoC 2026 Predictions: 30 NEW AI/ML/Security Organizations You Should Start Contributing to NOW! | dev.to | 2026-02-06Main: https://github.com/ray-project/ray ⭐ 34k+
-
AppSignal
Monitoring that respects your time & budget. APM, error tracking, and dashboards for modern web apps. Ten-minute setup, transparent flat pricing, and support from engineers who actually use the product.
-
-
OpenLLM
Run any open-source LLMs, such as DeepSeek and Llama, as OpenAI compatible API endpoint in the cloud.
-
BentoML
The easiest way to serve AI apps and models - Build Model Inference APIs, Job queues, LLM apps, Multi-model pipelines, and more!
-
-
Project mention: Qwen3.7-Max Ran for 35 Hours on Unknown Hardware and Achieved a 10× Speedup | news.ycombinator.com | 2026-05-27
so basically just brute force the kernel.
there are more elegant ways to leverage an LLM, see AlphaEvolve: https://github.com/algorithmicsuperintelligence/openevolve
it's difficult to frame most coding tasks in such a way where you can trivially verify correctness.
-
gpustack
A GPU cluster manager for high-performance AI model serving (vLLM, SGLang) and on-demand SSH-accessible GPU instances.
-
SaaSHub
SaaSHub - Software Alternatives and Reviews. SaaSHub helps you find the best software and product alternatives
-
-
Project mention: OptiLLM: Accuracy improvements on reasoning tasks with zero training | news.ycombinator.com | 2025-11-23
-
-
Project mention: Trying "DFlash," a Diffusion-Model Approach to Parallel Draft-Token Generation, on Gemma | dev.to | 2026-09-14
EAGLE3 ( https://arxiv.org/pdf/2503.01840 )
-
prompt-poet
Streamlines and simplifies prompt design for both developers and non-technical users with a low code approach.
-
OpenAlpha_Evolve
OpenAlpha_Evolve is an open-source Python framework inspired by the groundbreaking research on autonomous coding agents like DeepMind's AlphaEvolve.
-
Project mention: A weekend with TensorFold on a MacBook: the engine mattered, the quant did not | dev.to | 2026-09-28
Almost none of the fast parts are mine. TensorFold is Ash's work (MIT), and it had shipped five tagged releases in the day before I started. The DFlash2 draft model is z-lab's. The MLX 4-bit weights I tested came from Vontra. What I added is the part LLMKube exists for: getting a Mac-only engine behind a Kubernetes Service, so a Mac is one more inference node in a cluster that also has NVIDIA and AMD boxes in it.
-
backburner
Your iPhone helps your Mac run a 27B model: faster prompt reading and more context over a USB-C cable (by StayLameBro)
-
-
-
edsl
Design, conduct and analyze results of AI-powered surveys and experiments. Simulate social science and market research with large numbers of AI agents and LLMs.
-
-
-
JetStream
JetStream is a throughput and memory optimized engine for LLM inference on XLA devices, starting with TPUs (and GPUs in future -- PRs welcome). (by AI-Hypercomputer)
-
local-llm-function-calling
A tool for generating function arguments and choosing what function to call with local LLMs
-
proofofthought
Proof of thought : LLM-based reasoning using Z3 theorem proving with multiple backend support (SMT2 and JSON DSL)
Project mention: Toward Guarantees for Clinical Reasoning in Vision Language Models | news.ycombinator.com | 2026-03-01 -
SaaSHub
SaaSHub - Software Alternatives and Reviews. SaaSHub helps you find the best software and product alternatives
NOTE:
The open source projects on this list are ordered by number of github stars.
The number of mentions indicates repo mentiontions in the last 12 Months or
since we started tracking (Dec 2020).
Python llm-inference discussion
Python llm-inference related posts
-
Gemini 4 Argon
-
What does one token actually cost you?
-
I changed nothing and my LLM server got 27% more expensive
-
Entail – checks that what an LLM's model files declare is what the engine runs
-
Stop Prompt Injection in Your TypeScript LLM App with resk-llm-ts
-
Notes on CUDA Tensor Core GEMM (WMMA)
-
LLMCap – A proxy that hard-stops LLM API calls when you hit a dollar cap
-
A note from our sponsor - AppSignal
www.appsignal.com | 10 Oct 2026
Index
What are some of the best open-source llm-inference projects in Python? This list will help you:
| # | Project | Stars |
|---|---|---|
| 1 | Ray | 43,970 |
| 2 | litgpt | 13,693 |
| 3 | OpenLLM | 12,553 |
| 4 | BentoML | 8,875 |
| 5 | lmdeploy | 8,103 |
| 6 | openevolve | 7,510 |
| 7 | gpustack | 5,782 |
| 8 | superduper | 5,317 |
| 9 | optillm | 4,306 |
| 10 | lorax | 3,835 |
| 11 | EAGLE | 2,548 |
| 12 | prompt-poet | 1,157 |
| 13 | OpenAlpha_Evolve | 1,050 |
| 14 | TensorFold | 1,014 |
| 15 | backburner | 920 |
| 16 | llmflows | 707 |
| 17 | sarathi-serve | 534 |
| 18 | edsl | 502 |
| 19 | LLM-VM | 489 |
| 20 | chipper | 485 |
| 21 | JetStream | 464 |
| 22 | local-llm-function-calling | 435 |
| 23 | proofofthought | 376 |
Sponsored
Monitoring that respects your time & budget
APM, error tracking, and dashboards for modern web apps. Ten-minute setup, transparent flat pricing, and support from engineers who actually use the product.
www.appsignal.com