Nitro: A fast, lightweight 3MB inference server with OpenAI-Compatible API

1. llama.cpp

   1 1,033 115,929 10.0 C++

   LLM inference in C/C++

   

   Although Rust is an amazing language with rapid adoption, it still has a relatively smaller user base in low-level programming, where llama.cpp operates. As a result, the pool of talent that can contribute to such projects is more limited. Almost all low-level programmers can write in C++ if needed, as for C programmers, it's essentially like writing C with classes. Rust programmers, especially those who care about low-level details including throughput and latency, almost always have a background in C or C++. If llama.cpp were written in Rust, there would likely be far fewer contributors. Considering that one needs to be at least interested in deep learning to contribute, the fact that it currently has 476 contributors is impressive. [1] I think this is one the most important reasons the project can move so fast and be such an essential project in the LLM scene.

   [1]: https://github.com/ggerganov/llama.cpp/graphs/contributors

2. SaaSHub

   www.saashub.com featured

   SaaSHub - Software Alternatives and Reviews. SaaSHub helps you find the best software and product alternatives

   

3. cortex.cpp

   2 10 2,764 9.9 C++

   Discontinued Local AI API Platform

   

4. ollama

   3 754 173,924 9.9 Go

   Get up and running with Kimi-K2.6, GLM-5.1, MiniMax, DeepSeek, gpt-oss, Qwen, Gemma and other models.

   

   I recommend using https://ollama.ai/ if you dont want openai compatibility.

5. nitro

   4 33 10,915 9.9 TypeScript

   Next Generation Server Toolkit. Create web servers with everything you need and deploy them wherever you prefer.

   

   Not to be confused with https://nitro.unjs.io the server tech behind Nuxt and SolidStart

6. omnitool

   5 2 442 3.9 TypeScript

   Official Omnitool repository

   

   Our you could use something like omnitool (https://github.com/omnitool-ai/omnitool) and interface with both cloud and local AI, not limited to llms.

7. metal-cpp

   6 16 383 3.3 C++

   Metal-cpp is a low-overhead C++ interface for Metal that helps developers add Metal functionality to graphics apps, games, and game engines that are written in C++.

   

   My understanding is the proliferation of “XYZ-cpp” AI frameworks is due to the c++ support in Apple’s gpu library ‘Metal’, and the popularity of apple silicon for inference (and there are a few technical reasons for this): https://developer.apple.com/metal/cpp/

Suggest a related project