APM, error tracking, and dashboards for modern web apps. Ten-minute setup, transparent flat pricing, and support from engineers who actually use the product. Learn more →
Ninfer Alternatives
Similar projects and alternatives to ninfer
-
-
AppSignal
Monitoring that respects your time & budget. APM, error tracking, and dashboards for modern web apps. Ten-minute setup, transparent flat pricing, and support from engineers who actually use the product.
-
caveman
🪨 why use many token when few token do trick — Claude Code skill that cuts 65% of tokens by talking like caveman
-
-
warp
Run the full 2.78-trillion-parameter Kimi K3 model or GLM-5.3-Flash beyond available RAM by streaming activated weights directly from NVMe. A dependency-free, embeddable C inference engine. (by sqliteai)
-
-
-
Kargo
Stop Scripting Promotions. Start Shipping with Kargo. Kargo automates promotion across dev, staging, and prod with approval gates and verification. Open source, built by the team behind Argo CD. Download now.
-
MTPLX
Native MTP Speculative Decoding On Apple Silicon | 2x - 2.5x decode TPS increase at temp 0.6 | MLX-native, OpenAI API/Anthropic-compatible serving, no external drafter.
-
-
FreeToken
FreeToken brings datacenter-scale model serving to your desktop. Run massive models locally, fast and efficiently.
-
Swiftlet
Swiftlet is a Swift and Metal runtime that runs large Qwen Mixture-of-Experts models locally on Apple devices by streaming expert weights from storage, enabling 35B and 80B models to run with low RAM, including on iPhone.
-
memra
Rust + CUDA LLM inference engine for Blackwell (Tuned specifically on RTX PRO 6000, RTX 5090, B200): OpenAI-compatible (+converse and ant) serving, per-model X hardware exactness gates. NVFP4/mixed (fp8 hybrid, 4o6, etc - correctness, performance, hardware specific adapted) main quant support.
-
ninfer-3090
Hyper optimised Qwen3.8-27B inference on one RTX 3090: ReplaySSM, MTP3, reasoning effort, C1-C8 batching, and native Windows and Linux builds.
-
-
-
SaaSHub
SaaSHub - Software Alternatives and Reviews. SaaSHub helps you find the best software and product alternatives
NOTE:
The number of mentions on this list indicates mentions on common posts plus user suggested alternatives.
Hence, a higher number means a better ninfer alternative or higher similarity.
ninfer discussion
ninfer reviews and mentions
Posts with mentions or reviews of ninfer.
We have used some of these posts to build our list of alternatives
and similar projects. The last one was on 2026-09-03.
-
Qwen 3.8 27B available on Cerebras at 1500 tok/SEC
Just a couple days ago I learned about ninfer (https://github.com/Neroued/ninfer) and on RTX 5090 I can now get ~200 tok/s and over 400 tok/s on concurrent requests which is plenty fast for a local model of this strength.
-
Apple Caught Off Guard by AI Demand for Mac Mini and Mac Studio
Gives really only one concurrent session that flies because kv caching is not perfect for ninfer https://github.com/Neroued/ninfer
both are very serviceable, I prefer FP8 on 2xR9700
But, yes it doesn't scale that well but in 5 years the same hardware should still be very capable of running some great MoE models, for example Qwen 3.6 35BA3B on 5090 can fly at 600 tg
- Qwen 3.8-Flash-Next releasing tomorrow (125B a6B)
-
Building a mostly self-hosted, disposable agentic software factory
Using a RTX 5090 ($3~4k), I can run Qwen 3.8 27B at ~180 TPS with ninfer [0]. With its thinking maxed out, I can confirm that the quality of output is roughly on par with Opus 4.5~4.6 - that is, this 20GB file really can write software by itself, but the amount of thinking required makes it strictly slower than larger models, even at 180 TPS.
Still, it's largely replaced the cheap tier of the frontiers that I would otherwise be using. It can be run with older GPUs (a 3090 is ~1k), but the time spent thinking will become a fairly noticeable impediment for staying in the flow.
The next step up would be to run DeepSeek V4 Flash 0731 on two DGX Sparks (~$10k), which serve at 60 TPS and sit somewhere around Opus 4.7 level without 3.8-tier thinking.
However, it is worth noting that, if you are buying this hardware just to serve LLMs, it is not cost-effective. It would take over a decade of continuous use to make back the cost of the DGX Spark setup in 0731 tokens. I'm running this setup because I happen to have a 5090, and the two 3090s in my serve$ were cheap enough when amortised over several years.
[0]: https://github.com/Neroued/ninfer
-
Qwen 3.8 27B Topped Hacker News in a Day. Here's How to Run It Locally From Spring Boot
Speed depends heavily on the engine. One RTX 5090 owner reports about 138 tokens per second using the ninfer inference engine, roughly double their naive llama.cpp setup (comment). A 20GB VRAM card user reports about 30 tokens per second with a 30K context, and notes Muse Glimmer gives them 65-80 tokens per second at 128K on the same card (comment).
- High-performance single-GPU inference for selected model checkpoints and GPUs
-
Qwen 3.8 27B is out: open weights, best local dense model yet
If anyone else is running this on an RTX 5090, https://github.com/Neroued/ninfer as inference engine gets me ~138 tokens/second, roughly double what I get with a naive llama.cpp setup.
- Ninfer: High-performance single-GPU inference f
- NInfer: Tok/s (543 with speculative) for Qwen3.6-35B-A3B on one RTX 5090
-
A note from our sponsor - AppSignal
www.appsignal.com | 11 Sep 2026
Stats
Basic ninfer repo stats
9
1,562
9.9
1 day ago
Neroued/ninfer is an open source project licensed under Apache License 2.0 which is an OSI approved license.
The primary programming language of ninfer is C++.
Sponsored
Monitoring that respects your time & budget
APM, error tracking, and dashboards for modern web apps. Ten-minute setup, transparent flat pricing, and support from engineers who actually use the product.
www.appsignal.com