Have to abandon my (almost) finished LLaMA-API-Inference server. If anybody finds it useful and wants to continue, the repo is yours. :)

• TALIS

  2 23 8.7 Python

  Simple and fast server for GPTQ-quantized LLaMA inference

The repo can be found here, the readme is not up-to-date. The code is a bit messy.

• accelerate

  18 6,996 9.7 Python

  🚀 A simple way to launch, train, and use PyTorch models on almost any device and distributed configuration, automatic mixed precision (including fp8), and easy-to-configure FSDP and DeepSpeed support

  

As /u/RabbitHole32 already mentioned, the speed increase stems from a patch which modifies, how a certain, large tensor is distributed between the GPU's. The patch was created by /u/emvw7yf. Here you can find the respective GitHub issue: https://github.com/huggingface/accelerate/issues/1394

• InfluxDB

  www.influxdata.com featured

  Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.

  

• llamazoo

  2 57 10.0 C

  Discontinued Large Model Collider - The Platform for serving LLM models [Moved to: https://github.com/gotzmann/collider]

Looks like exactly same idea I'm doing right now with LLaMAZoo: https://github.com/gotzmann/llamazoo

Suggest a related project