Python Inference

Open-source Python projects categorized as Inference | Edit details

Top 14 Python Inference Projects

  • server

    The Triton Inference Server provides an optimized cloud and edge inferencing solution. (by triton-inference-server)

    Project mention: Popular Machine Learning Deployment Tools | | 2022-04-16


  • torch2trt

    An easy to use PyTorch to TensorRT converter

    Project mention: [D] How you deploy your ML model? | | 2021-10-27
  • Scout APM

    Less time debugging, more time building. Scout APM allows you to find and fix performance issues with no hassle. Now with error monitoring and external services monitoring, Scout is a developer's best friend when it comes to application development.

  • adversarial-robustness-toolbox

    Adversarial Robustness Toolbox (ART) - Python Library for Machine Learning Security - Evasion, Poisoning, Extraction, Inference - Red and Blue Teams

    Project mention: adversarial-robustness-toolbox: Adversarial Robustness Toolbox (ART) - Python Library for Machine Learning Security - Evasion, Poisoning, Extraction, Inference - Red and Blue Teams | | 2021-12-26
  • budgetml

    Deploy a ML inference service on a budget in less than 10 lines of code.



    Project mention: Training NER models for detecting custom entities | | 2021-10-08
  • transformer-deploy

    Efficient, scalable and enterprise-grade CPU/GPU inference server for 🤗 Hugging Face transformer models 🚀

    Project mention: [P] 4.5 times faster Hugging Face transformer inference by modifying some Python AST | | 2021-12-28

    Regarding CPU inference, quantization is very easy, and supported by Transformer-deploy , however performance on transformer are very low outside corner cases (like no batch, very short sequence and distilled model), and last Intel generation CPU based instance like C6 or M6 on AWS are quite expensive compared to a cheap GPU like Nvidia T4, to say it otherwise, on transformer, until you are ok with slow inference and takes a small instance (for a PoC for instance), CPU inference is probably not a good idea.

  • deepsparse

    Neural network inference engine that delivers GPU-class performance for sparsified models on CPUs

    Project mention: [P] SparseServer.UI : A UI to test performance of Sparse Transformers | | 2022-04-19

    Hi _Arsenie, this runs the deepsparse.server command for multiple models. and btw, we recently updated the READMEs for the Deepsparse Engine

  • SonarQube

    Static code analysis for 29 languages.. Your projects are multi-language. So is SonarQube analysis. Find Bugs, Vulnerabilities, Security Hotspots, and Code Smells so you can release quality code every time. Get started analyzing your projects today for free.


    Small, dependency-free, fast Python package to infer binary file types checking the magic numbers signature

  • fastT5

    ⚡ boost inference speed of T5 models by 5x & reduce the model size by 3x.

    Project mention: Conceptually, what are the "Past key values" in the T5 Decoder? | | 2021-11-14

    Here is the fastT5 model code for reference code:

  • sparktorch

    Train and run Pytorch models on Apache Spark.

    Project mention: Spark2 + pytorch on GPU | | 2021-09-17

    Was reading the documentation of sparktorch ( which says you need spark >= 2.4.4. But to the best of my knowledge spark2 doesn't have gpu compute capabilities. Does that mean it can only use cpu compute? Am I missing something?

  • emlearn

    Machine Learning inference engine for Microcontrollers and Embedded devices

    Project mention: Simple and embedded friendly C code for Machine Learning inference algorithms | | 2022-01-01

    Examples: Gaussian Mixture Models (GMM) for anomaly detection or clustering Mahalanobis distance (EllipticEnvelope) for anomaly detection Decision trees and tree ensembles (Random Forest, ExtraTrees) Feed-forward Neural Networks (Multilayer Perceptron, MLP) for classification Gaussian Naive Bayes for classification

  • pinferencia

    Python + Inference - Model Deployment library in Python. Simplest model inference server ever.

    Project mention: Stop Writing Flask to Serve/Deploy Your Model: Pinferencia is Here | | 2022-04-27

    Go visit: Pinferencia ( for detailed examples.

  • nostril

    Nostril: Nonsense String Evaluator

  • experta

    Expert Systems for Python

NOTE: The open source projects on this list are ordered by number of github stars. The number of mentions indicates repo mentiontions in the last 12 Months or since we started tracking (Dec 2020). The latest post mention was on 2022-04-27.

Python Inference related posts


What are some of the best open-source Inference projects in Python? This list will help you:

Project Stars
1 server 3,638
2 torch2trt 3,341
3 adversarial-robustness-toolbox 2,984
4 budgetml 1,267
5 BERT-NER 1,024
6 transformer-deploy 880
7 deepsparse 579
8 374
9 fastT5 292
10 sparktorch 255
11 emlearn 215
12 pinferencia 188
13 nostril 126
14 experta 91
Find remote jobs at our new job board There are 7 new remote jobs listed recently.
Are you hiring? Post a new remote job listing for free.
SaaSHub - Software Alternatives and Reviews
SaaSHub helps you find the best software and product alternatives