torch2trt
onnx-simplifier
| torch2trt | onnx-simplifier | |
|---|---|---|
| 5 | 3 | |
| 4,395 | 3,550 | |
| 1.0% | - | |
| 3.1 | 6.5 | |
| 5 days ago | 22 days ago | |
| Python | C++ | |
| MIT License | Apache License 2.0 |
Stars - the number of stars that a project has on GitHub. Growth - month over month growth in stars.
Activity is a relative number indicating how actively a project is being developed. Recent commits have higher weight than older ones.
For example, an activity of 9.0 indicates that a project is amongst the top 10% of the most actively developed projects that we are tracking.
torch2trt
- [D] How you deploy your ML model?
-
PyTorch 1.10
Main thing you want for server inference is auto batching. It's a feature that's included in onnxruntime, torchserve, nvidia triton inference server and ray serve.
If you have a lot of preprocessing and post logic in your model it can be hard to export it for onnxruntime or triton so I usually recommend starting with Ray Serve (https://docs.ray.io/en/latest/serve/index.html) and using an actor that runs inference with a quantized model or optimized with tensorrt (https://github.com/NVIDIA-AI-IOT/torch2trt)
-
Jetson Nano: TensorFlow model. Possibly I should use PyTorch instead?
https://github.com/NVIDIA-AI-IOT/torch2trt <- pretty straightforward https://github.com/jkjung-avt/tensorrt_demos <- this helped me a lot
-
How to get TensorFlow model to run on Jetson Nano?
I find Pytorch easier to work with generally. Nvidia has a Pytorch --> TensorRT converter which yields some significant speedups and has a simple Python API. Convert the Pytorch model on the Nano.
onnx-simplifier
-
Show: Cross-platform Image segmentation on video using eGUI, onnxruntime and ffmpeg
onnx-simplifier can shed some of incompatibilities in widespread use, but is itself bug ridden and lagging behind the standard. For any serious model, or when you don't get lucky simplifying the model upstream, you'd generally want good support of opset 11.
-
[Technical Article] OCR Upgrade
ONNX Simplifier：https://github.com/daquexian/onnx-simplifier
-
PyTorch 1.10
As far as I know, the ONNX format won't give you a performance boost on its own. However, there are ONNX optimizers for the ONNX runtime which will speed up your inference.
But if you are using Nvidia Hardware, then TensorRT should give you the best performance possible, especially if you change the precision level. Don't forget to simplify your ONNX model before you converting it to TensorRT though: https://github.com/daquexian/onnx-simplifier
What are some alternatives?
TensorRT - PyTorch/TorchScript/FX compiler for NVIDIA GPUs using TensorRT
onnxruntime - ONNX Runtime: cross-platform, high performance ML inferencing and training accelerator
Pytorch - Tensors and Dynamic neural networks in Python with strong GPU acceleration
PaddleOCR - Awesome multilingual OCR toolkits based on PaddlePaddle (practical ultra lightweight OCR system, support 80+ languages recognition, provide data annotation and synthesis tools, support training and deployment among server, mobile, embedded and IoT devices)
transformer-deploy - Efficient, scalable and enterprise-grade CPU/GPU inference server for 🤗 Hugging Face transformer models 🚀
functorch - functorch is JAX-like composable function transforms for PyTorch.
nn - 🧑🏫 60 Implementations/tutorials of deep learning papers with side-by-side notes 📝; including transformers (original, xl, switch, feedback, vit, ...), optimizers (adam, adabelief, sophia, ...), gans(cyclegan, stylegan2, ...), 🎮 reinforcement learning (ppo, dqn), capsnet, distillation, ... 🧠
tensorrt_demos - TensorRT MODNet, YOLOv4, YOLOv3, SSD, MTCNN, and GoogLeNet
trt_pose - Real-time pose estimation accelerated with NVIDIA TensorRT
ncnn - ncnn is a high-performance neural network inference framework optimized for the mobile platform