[P] What are the latest "out of the box solutions" for deploying the very large LLMs as API endpoints?

• transformers-bloom-inference

  2 548 2.4 Python

  Fast Inference Solutions for BLOOM

  

This repo that has some "demos": https://github.com/huggingface/transformers-bloom-inference

• FlexGen

  39 9,017 3.5 Python

  Running large language models on a single GPU for throughput-oriented scenarios.

  

FlexGen: https://github.com/FMInference/FlexGen but that only works for opt and is not a model hosting solution but more of an academic PoC

• InfluxDB

  www.influxdata.com featured

  Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.

  

• text-generation-inference

  29 7,938 9.6 Python

  Large Language Model Text Generation Inference

  

Suggest a related project