The APIs are flexible and easy-to-use, supporting authentication, user identity, and complex enterprise features like SSO and SCIM provisioning. Learn more →
Top 5 Python llm-serving Projects
-
Ray
Ray is a unified framework for scaling AI and Python applications. Ray consists of a core distributed runtime and a set of AI Libraries for accelerating ML workloads.
-
WorkOS
The modern identity platform for B2B SaaS. The APIs are flexible and easy-to-use, supporting authentication, user identity, and complex enterprise features like SSO and SCIM provisioning.
-
OpenLLM
Run any open-source LLMs, such as Llama 2, Mistral, as OpenAI compatible API endpoint, locally and in the cloud.
-
skypilot
SkyPilot: Run LLMs, AI, and Batch jobs on any cloud. Get maximum savings, highest GPU availability, and managed execution—all with a simple interface.
-
mosec
A high-performance ML model serving framework, offers dynamic batching and CPU/GPU pipelines to fully exploit your compute machine
-
InfluxDB
Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.
22. Ray | Github | tutorial
The easiest is to use vllm (https://github.com/vllm-project/vllm) to run it on a Couple of A100's, and you can benchmark this using this library (https://github.com/EleutherAI/lm-evaluation-harness)
13. OpenLLM by BentoML | Github | tutorial
Project mention: Ask HN: Most efficient way to fine-tune an LLM in 2024? | news.ycombinator.com | 2024-04-04
Project mention: 20x Faster as the Beginning: Introducing pgvecto.rs extension written in Rust | dev.to | 2023-08-06Mosec - A high-performance serving framework for ML models, offers dynamic batching and CPU/GPU pipelines to fully exploit your compute machine. Simple and faster alternative to NVIDIA Triton.
Python llm-serving related posts
- The Lost Arts of CLJS Frontend
- OpenLLM: An open platform for operating large language models (LLMs) in production.
- OpenLLM: An open platform for operating large language models (LLMs) in production.
- OpenLLM: An open platform for operating large language models (LLMs) in production.
- GPT Weekly - 26the June Edition - 🎙️ Meta's Voicebox is Paused, 🖼️SDXL 0.9, 📜AI Compliance & EU Act and more
- OpenLLM: OSS to easily serve Open Source LLMs
- OpenLLM: OSS to easily serve Open Source LLMs
-
A note from our sponsor - WorkOS
workos.com | 27 Apr 2024
Index
What are some of the best open-source llm-serving projects in Python? This list will help you:
Project | Stars | |
---|---|---|
1 | Ray | 31,101 |
2 | vllm | 18,041 |
3 | OpenLLM | 8,733 |
4 | skypilot | 5,636 |
5 | mosec | 703 |
Sponsored