[D] Handling Concurrent Request for ML Model API

• mosec

  11 707 8.5 Python

  A high-performance ML model serving framework, offers dynamic batching and CPU/GPU pipelines to fully exploit your compute machine

  

- Yes C++ would be better, but you can try mosec. It has a Python interface and helps you handle all the difficult things about Python multiprocessing. The web service part is implemented in Rust thus it's fast enough for machine learning services.

• inference-benchmark

  1 26 6.4 Python

  Benchmark for machine learning model online serving (LLM, embedding, Stable-Diffusion, Whisper)

  

I have done some benchmarks before: https://github.com/tensorchord/inference-benchmark

• InfluxDB

  www.influxdata.com featured

  Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.

  

• text-generation-inference

  29 7,881 9.6 Python

  Large Language Model Text Generation Inference

  

Look at something like https://github.com/huggingface/text-generation-inference to get an idea of how to do this (they use a lot of optimizations)... or just use/adapt the project if you want to instead

Suggest a related project