Deploying Llama2 with vLLM vs TGI. Need advice

• vllm

  31 18,571 9.9 Python

  A high-throughput and memory-efficient inference and serving engine for LLMs

  

I've been experimenting with deploying a model using two platforms: vLLM and TGI. While using the standard fp16 version, both platforms perform fairly comparably. However, I observed a significant performance gap when deploying the GPTQ 4bits version on TGI as opposed to vLLM.

• text-generation-inference

  29 7,881 9.6 Python

  Large Language Model Text Generation Inference

  

I've been experimenting with deploying a model using two platforms: vLLM and TGI. While using the standard fp16 version, both platforms perform fairly comparably. However, I observed a significant performance gap when deploying the GPTQ 4bits version on TGI as opposed to vLLM.

• InfluxDB

  www.influxdata.com featured

  Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.

  

• vllm-gptq

  1 118 9.9 Python

  A high-throughput and memory-efficient inference and serving engine for LLMs

The models are TheBloke/Llama2-7B-fp16 and TheBloke/Llama2-7B-GPTQ. I'm using 1000 prompts with a request rate (number of requests per second) of 10. By default, vLLM does not support for GPTQ, so I'm using this version: vLLM-GPTQ.

Suggest a related project