Top 4 Python vllm Projects
-
OpenRLHF
An Easy-to-use, Scalable and High-performance RLHF Framework (Support 70B+ full tuning & LoRA & Mixtral & KTO)
-
-
Scout Monitoring
Free Django app performance insights with Scout Monitoring. Get Scout setup in minutes, and let us sweat the small stuff. A couple lines in settings.py is all you need to start monitoring your apps. Sign up for our free tier today.
-
llm-vscode-inference-server
An endpoint server for efficiently serving quantized open-source LLMs for code.
Project mention: GPUsGoBurr: Get up to 2x higher performance by Tuning LLM Inference Deployment | news.ycombinator.com | 2024-05-15Do check out the GitHub repo https://github.com/microsoft/vidur . You can run it without any GPUs.
Project mention: Replit's new AI Model now available on Hugging Face | news.ycombinator.com | 2023-10-11Requests for code generation are made via an HTTP request.
You can use the Hugging Face Inference API or your own HTTP endpoint, provided it adheres to the API specified here[1] or here[2]."
It's fairly easy to use your own model locally with the plugin. You can just use the one of the community developed inference servers, which are listed at the bottom of the page, but here's the links[3] to both[4].
[1]: https://huggingface.co/docs/api-inference/detailed_parameter...
[2]: https://huggingface.github.io/text-generation-inference/#/Te...
[3]: https://github.com/wangcx18/llm-vscode-inference-server
[4]: https://github.com/wangcx18/llm-vscode-inference-server
Index
What are some of the best open-source vllm projects in Python? This list will help you:
| Project | Stars | |
|---|---|---|
| 1 | OpenRLHF | 1,477 |
| 2 | vidur | 94 |
| 3 | llm-vscode-inference-server | 44 |
| 4 | llama3v | 2 |
Sponsored