AI Code assistant for about 50-70 users

• vllm

  31 18,571 9.9 Python

  A high-throughput and memory-efficient inference and serving engine for LLMs

  

https://github.com/vllm-project/vllm is probably more optimized for that use case.

• refact

  34 1,422 9.8 JavaScript

  WebUI for Fine-Tuning and Self-hosting of Open-Source Large Language Models for Coding

  

Refact was made for this: https://github.com/smallcloudai/refact

• InfluxDB

  www.influxdata.com featured

  Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.

  

• text-generation-inference

  29 7,881 9.6 Python

  Large Language Model Text Generation Inference

  

Setting up a server for multiple users is very different from setting up LLM for yourself. A safe bet would be to just use TGI, which supports continuous batching and is very easy to run via Docker on your server. https://github.com/huggingface/text-generation-inference

• deploy-os-code-llm

  2 47 5.5

  🌉 How to deploy an open-source code LLM for your dev team

  

I looked into how to deploy an open-source code LLM for a dev team a couple months ago and identified five questions to figure out:

Suggest a related project