Batch-inference Alternatives
Similar projects and alternatives to batch-inference
-
adversarial-robustness-toolbox
Adversarial Robustness Toolbox (ART) - Python Library for Machine Learning Security - Evasion, Poisoning, Extraction, Inference - Red and Blue Teams
-
-
InfluxDB
Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.
-
server
The Triton Inference Server provides an optimized cloud and edge inferencing solution. (by triton-inference-server)
-
pinferencia
Python + Inference - Model Deployment library in Python. Simplest model inference server ever.
-
DeepSpeed
DeepSpeed is a deep learning optimization library that makes distributed training and inference easy, efficient, and effective.
-
-
SaaSHub
SaaSHub - Software Alternatives and Reviews. SaaSHub helps you find the best software and product alternatives
batch-inference reviews and mentions
-
Tutorial to improve GPT throughput 16 times with dynamic batching
I wrote a tutorial to improve GPT completion throughput with dynamic batching https://microsoft.github.io/batch-inference/examples/gpt_completion.html. And I can achieve 16 times throughput on V100 comparing to baseline. We built a python dynamic batching library so you can apply it on your own models easily https://github.com/microsoft/batch-inference.
Stats
microsoft/batch-inference is an open source project licensed under MIT License which is an OSI approved license.
The primary programming language of batch-inference is Python.
Popular Comparisons
Sponsored