C++ inference-engine

Open-source C++ projects categorized as inference-engine
Edit details
Topics: Cuda Deep Learning Deployment CPP AI
Power Real-Time Data Analytics at Scale
Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.
www.influxdata.com
featured

Top 4 C++ inference-engine Projects

  • cortex

    Drop-in, local AI alternative to the OpenAI stack. Multi-engine (llama.cpp, TensorRT-LLM). Powers 👋 Jan (by janhq)

    • Project mention: Introducing Jan | dev.to | 2024-05-05

    Jan incorporates a lightweight, built-in inference server called Nitro. Nitro supports both llama.cpp and NVIDIA's TensorRT-LLM engines. This means many open LLMs in the GGUF format are supported. Jan's Model Hub is designed for easy installation of pre-configured models but it also allows you to install virtually any model from Hugging Face or even your own.

  • Daisykit

    Daisykit is an easy AI toolkit with face mask detection, pose detection, background matting, barcode detection, and more. With Daisykit, you don't need AI knowledge to build AI software.

  • InfluxDB

    Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.

    InfluxDB logo

  • EasyOCR-cpp

    Custom C++ implementation of deep learning based OCR

    • Project mention: [P] EasyOCR in C++! | /r/MachineLearning | 2023-12-02

  • nnl

    a low-latency and high-performance inference engine for large models on low-memory GPU platform.

    • Project mention: Run 70B LLM Inference on a Single 4GB GPU with This New Technique | news.ycombinator.com | 2023-12-03

    I did roughly the same thing in one of my hobby project https://github.com/fengwang/nnl. But in stead of using SSD, I load all the weights to the host memory, and while inferencing the model layer by layer, I asynchronously copy memory from global to shared memory in the hope of better performance. However, my approach is bounded by the PCI-E bandwidth.

NOTE: The open source projects on this list are ordered by number of github stars. The number of mentions indicates repo mentiontions in the last 12 Months or since we started tracking (Dec 2020).

C++ inference-engine related posts

  • QUIK is a method for quantizing LLM post-training weights to 4 bit precision

    2 projects | news.ycombinator.com | 6 Nov 2023

  • Intel OpenVINO 2023.1.0 released

    1 project | /r/intel | 20 Sep 2023

  • Intel OpenVINO 2023.1.0 released, open-source toolkit for optimizing and deploying AI inference

    1 project | /r/opensource | 20 Sep 2023

  • OpenVINO 2023.1.0 released

    1 project | /r/IntelArc | 20 Sep 2023

  • [N] Intel OpenVINO 2023.1.0 released, open-source toolkit for optimizing and deploying AI inference

    1 project | /r/MachineLearning | 19 Sep 2023

  • Intel OpenVINO 2023.1.0 released, open-source toolkit for optimizing and deploying AI inference

    1 project | /r/computervision | 19 Sep 2023

  • Intel OpenVINO 2023.1.0 released, open-source toolkit for optimizing and deploying AI inference

    1 project | /r/pytorch | 19 Sep 2023
  • A note from our sponsor - InfluxDB
    www.influxdata.com | 7 May 2024
    Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality. Learn more →

Index

What are some of the best open-source inference-engine projects in C++? This list will help you:

Project Stars
1 cortex 1,600
2 Daisykit 95
3 EasyOCR-cpp 27
4 nnl 4

Sponsored
SaaSHub - Software Alternatives and Reviews
SaaSHub helps you find the best software and product alternatives
www.saashub.com