Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality. Learn more →
Top 4 Python inference-server Projects
-
inference
A fast, easy-to-use, production-ready inference server for computer vision supporting deployment of many popular model architectures and fine-tuned models. (by roboflow)
-
InfluxDB
Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.
-
pinferencia
Python + Inference - Model Deployment library in Python. Simplest model inference server ever.
-
inference-benchmark
Benchmark for machine learning model online serving (LLM, embedding, Stable-Diffusion, Whisper)
Yeah, inference[1] is our open source package for running locally (either directly in Python or via a Docker container). It works with all the models on Universe, models you train yourself (assuming we support the architecture; we have a bunch of notebooks available[2]), or train in our platform, plus several more general foundation models[3] (for things like embeddings, zero-shot detection, question answering, OCR, etc).
We also have a hosted API[4] you can hit for most models we support (except some of the large vision models that are really GPU-heavy) if you prefer.
[1] https://github.com/roboflow/inference
[2] https://github.com/roboflow/notebooks
[3] https://inference.roboflow.com/foundation/about/
[4] https://docs.roboflow.com/deploy/hosted-api
I have done some benchmarks before: https://github.com/tensorchord/inference-benchmark
Python inference-server related posts
- Show HN: Pinferencia, Deploy Your AI Models with Pretty UI and REST API
- Stop Writing Flask to Serve/Deploy Your Model: Pinferencia is Here
- Looking for a reference design pattern for an image to image microservice
- Google T5 Translation as a Service with Just 7 lines of Codes
- Pre-trained Model with Fine Tuning/Transfer Learning or Design and Train from Scratch?
- [D] Pre-trained Model with Fine Tuning/Transfer Learning or Design and Train from Scratch?
- GPT2 — Text Generation Transformer: How to Use & How to Serve
-
A note from our sponsor - InfluxDB
www.influxdata.com | 23 Apr 2024
Index
What are some of the best open-source inference-server projects in Python? This list will help you:
Project | Stars | |
---|---|---|
1 | inference | 1,015 |
2 | truss | 830 |
3 | pinferencia | 556 |
4 | inference-benchmark | 25 |
Sponsored