Run 70B LLM Inference on a Single 4GB GPU with This New Technique

1. FlexLLMGen

   1 39 9,224 5.0 Python

   Discontinued Running large language models on a single GPU for throughput-oriented scenarios.

   

2. SaaSHub

   www.saashub.com featured

   SaaSHub - Software Alternatives and Reviews. SaaSHub helps you find the best software and product alternatives

   

3. petals

   2 101 9,347 7.2 Python

   🌸 Run LLMs at home, BitTorrent-style. Fine-tuning and inference up to 10x faster than offloading

   

   There is already an implementation along the same line using the torrent architecture.

   https://petals.dev/

4. nnl

   3 1 5 5.4 C++

   a low-latency and high-performance inference engine for large models on low-memory GPU platform.

   I did roughly the same thing in one of my hobby project https://github.com/fengwang/nnl. But in stead of using SSD, I load all the weights to the host memory, and while inferencing the model layer by layer, I asynchronously copy memory from global to shared memory in the hope of better performance. However, my approach is bounded by the PCI-E bandwidth.

Suggest a related project