Top 4 C++ llm Projects
-
infinity
The AI-native database built for LLM applications, providing incredibly fast full-text and vector search (by infiniflow)
-
cortex
Drop-in, local AI alternative to the OpenAI stack. Multi-engine (llama.cpp, TensorRT-LLM). Powers 👋 Jan (by janhq)
-
InfluxDB
Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.
-
vectordb
Epsilla is a high performance Vector Database Management System. Try out hosted Epsilla at https://cloud.epsilla.com/
Using C++20 modules, take a look at this project: https://github.com/infiniflow/infinity
Jan incorporates a lightweight, built-in inference server called Nitro. Nitro supports both llama.cpp and NVIDIA's TensorRT-LLM engines. This means many open LLMs in the GGUF format are supported. Jan's Model Hub is designed for easy installation of pre-configured models but it also allows you to install virtually any model from Hugging Face or even your own.
I helped create this using https://www.epsilla.com/ which is an incredibly good vector search tool. Our long covid guidance got 10x better when we switched to it!
Got a web version here, check it out and let us know if it’s helpful to you:
For me, the easiest way was to start was to use https://github.com/abdeladim-s/pyllamacpp. If you have Python installed, do `pip install numpy pyllamacpp`, download compatible model file (links are in README.md) and simply run with `pyllamacpp model.file`.
C++ llms related posts
Index
What are some of the best open-source llm projects in C++? This list will help you:
| Project | Stars | |
|---|---|---|
| 1 | infinity | 1,726 |
| 2 | cortex | 1,613 |
| 3 | vectordb | 872 |
| 4 | pyllamacpp | 59 |
Sponsored