[D] Large Language Models feasible to run on 32GB RAM / 8 GB VRAM / 24GB VRAM

• bitsandbytes

  61 5,389 9.4 Python

  Accessible large language models via k-bit quantization for PyTorch.

• rwkvstic

  2 144 6.7 Python

  Framework agnostic python runtime for RWKV models

RWVK can run on very little VRAM with Rwvkstic streaming and 8-bit. I've not tested streaming, but I expect it's a lot slower. 7B model sadly takes 8 GB with just 8-bit quantization.

• InfluxDB

  www.influxdata.com sponsored

  Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.

  

• FlexGen

  19 5,350 10.0 Python

  Discontinued Running large language models like OPT-175B/GPT-3 on a single GPU. Focusing on high-throughput generation. [Moved to: https://github.com/FMInference/FlexGen] (by Ying1123)

  

Could try something like this: https://github.com/Ying1123/FlexGen

• accelerate

  18 6,948 9.7 Python

  🚀 A simple way to launch, train, and use PyTorch models on almost any device and distributed configuration, automatic mixed precision (including fp8), and easy-to-configure FSDP and DeepSpeed support

  

Try to use both GPUs with this one: https://github.com/huggingface/accelerate https://huggingface.co/docs/accelerate/usage_guides/big_modeling https://huggingface.co/blog/accelerate-large-models Maybe it will help (the last link is clearer IMHO).

Suggest a related project