Top 5 C++ Inference Projects
Cross-platform, customizable ML solutions for live and streaming media.Project mention: First try to replicate mediapipe codepen hand detection. | reddit.com/r/cpp | 2021-12-28
I have used mediapipie library to make this. https://github.com/google/mediapipe
ncnn is a high-performance neural network inference framework optimized for the mobile platformProject mention: ncnn convert | dev.to | 2021-09-29
ncnn install ncnn
OPS - Build and Run Open Source Unikernels. Quickly and easily build and deploy open source unikernels in tens of seconds. Deploy in any language to any cloud.
Hello AI World guide to deploying deep-learning inference networks and deep vision primitives with TensorRT and NVIDIA Jetson.Project mention: Jetson Nano 2GB Issues During Training (Out Of Memory / Process Killed) & Other Questions! | reddit.com/r/JetsonNano | 2021-11-05
I’m trying to do the tutorial, where they retrain the neural network to detect fruits (jetson-inference/pytorch-ssd.md at master · dusty-nv/jetson-inference · GitHub 1)
The Triton Inference Server provides an optimized cloud and edge inferencing solution. (by triton-inference-server)Project mention: [D] patterns for scaling video inference service? | reddit.com/r/MachineLearning | 2021-12-22
If you are working at an infrastructure level I would use ECS and utilize the NVIDIA Triton Inference Server. It can handle the multimodel paradigm through their ensemble method (bit of a misnomer since its really just a DAG of data flow through your models though you can add an ensembling method at the end of desired). Also provides a nice HTTP or GRPC interface. With ECS you can also use Application Load Balancer to further scale but how you set that up will also heavily depend on if you are using stateful models or not.
Run TensorFlow models in C++ without installation and without BazelProject mention: [Python] Importing a TensorFlow AI? | reddit.com/r/godot | 2021-10-12
I toyed around with this idea a while back but I never got around to finishing the implementation. If all you need is inference with no training and you are relatively familiar with c++ you could look into creating a module for Godot that interfaces with the Tensorflow C API. Something like cppflow would provide an even easier API to work with. Looking into that project could also explain how they interface with the Tensorflow C API if you'd rather cut out the middle man. A module like this would let you train your model in Python and then load it and perform inference in Godot natively.
C++ Inference related posts
[D] patterns for scaling video inference service?
1 project | reddit.com/r/MachineLearning | 22 Dec 2021
Deploying models in the real world
1 project | reddit.com/r/deeplearning | 18 Dec 2021
[P] Python library to optimize Hugging Face transformer for inference: < 0.5 ms latency / 2850 infer/sec
4 projects | reddit.com/r/MachineLearning | 23 Nov 2021
Nvidia Fiscal Q3 2022 Financial Result
4 projects | reddit.com/r/nvidia | 17 Nov 2021
Jetson Nano 2GB Issues During Training (Out Of Memory / Process Killed) & Other Questions!
1 project | reddit.com/r/JetsonNano | 5 Nov 2021
[Python] Importing a TensorFlow AI?
1 project | reddit.com/r/godot | 12 Oct 2021
Simplest way to deploy Keras NN model into C++?
2 projects | reddit.com/r/learnmachinelearning | 29 Aug 2021
What are some of the best open-source Inference projects in C++? This list will help you:
Are you hiring? Post a new remote job listing for free.