[D] Would a Tesla M40 provide cheap inference acceleration for self-hosted LLMs?

1. llama.cpp

   1 876 76,599 10.0 C++

   LLM inference in C/C++

   

   Not what you asked, but there are projects out there to run llama and its descendants on CPU only using system RAM. See https://github.com/ggerganov/llama.cpp/discussions/643 for a discussion about running Vicuna-13b in particular. The 4 bit quantized model should run comfortably within a 16GB system.

2. CodeRabbit

   coderabbit.ai featured

   CodeRabbit: AI Code Reviews for Developers. Revolutionize your code reviews with AI. CodeRabbit offers PR summaries, code walkthroughs, 1-click suggestions, and AST-based analysis. Boost productivity and code quality across all major languages with each PR.

   

3. nvidia-docker

   2 53 16,998 0.0 Makefile

   Discontinued Build and run Docker containers leveraging NVIDIA GPUs

   

4. nvidia-patch

   3 313 3,848 8.0 Python

   This patch removes restriction on maximum number of simultaneous NVENC video encoding sessions imposed by Nvidia to consumer-grade GPUs.

   

5. vGPU_LicenseBypass

   4 3 240 0.0 PowerShell

   A simple script that works around Nvidia vGPU licensing with a scheduled task.

   

6. text-generation-webui

   5 884 42,886 9.7 Python

   A Gradio web UI for Large Language Models with support for multiple inference backends.

   

   I’ve been using oobabooga text webui (the one-click installer) and it works mostly fine. Just need to make sure you have a model that works with it, like one of the 4bit 128 group quantized models, and set the right arguments in start-webui.bat for the model

7. turbopilot

   6 15 3,839 10.0 C++

   Discontinued Turbopilot is an open source large-language-model based code completion engine that runs locally on CPU

   I don't know if this applies to your use case but this would probably work if you are looking for an llm to help with programming. Haven't really played around with it but this may work for general llm tasks, it doesn't have a web UI though.

8. simpleAI

   7 11 329 2.4 Python

   An easy way to host your own AI API and expose alternative models, while being compatible with "open" AI clients.

   

   I don't know if this applies to your use case but this would probably work if you are looking for an llm to help with programming. Haven't really played around with it but this may work for general llm tasks, it doesn't have a web UI though.

9. SaaSHub

   www.saashub.com featured

   SaaSHub - Software Alternatives and Reviews. SaaSHub helps you find the best software and product alternatives

   

Suggest a related project