Run 70B LLM Inference on a Single 4GB GPU with This New Technique

• FlexGen

  39 8,999 3.0 Python

  Running large language models on a single GPU for throughput-oriented scenarios.

  

• petals

  98 8,661 8.5 Python

  🌸 Run LLMs at home, BitTorrent-style. Fine-tuning and inference up to 10x faster than offloading

  

There is already an implementation along the same line using the torrent architecture.

https://petals.dev/

• WorkOS

  workos.com sponsored

  The modern identity platform for B2B SaaS. The APIs are flexible and easy-to-use, supporting authentication, user identity, and complex enterprise features like SSO and SCIM provisioning.

  

• nnl

  1 4 5.4 C++

  a low-latency and high-performance inference engine for large models on low-memory GPU platform.

I did roughly the same thing in one of my hobby project https://github.com/fengwang/nnl. But in stead of using SSD, I load all the weights to the host memory, and while inferencing the model layer by layer, I asynchronously copy memory from global to shared memory in the hope of better performance. However, my approach is bounded by the PCI-E bandwidth.

Suggest a related project