What's the most basic NVIDIA graphics card that will work with mainstream 7B GPU models?

• alpaca-electron

  8 1,261 5.9 JavaScript

  The simplest way to run Alpaca (and other LLaMA-based local LLMs) on your own computer

• mlc-llm

  89 17,215 9.9 Python

  Enable everyone to develop, optimize and deploy AI models natively on everyone's devices.

  

4GB of VRAM is enough to run a 7B model. Try this, https://github.com/mlc-ai/mlc-llm. It uses Vulkan instead of CUDA. Their converted 7B model runs on a GTX 4GB card for me. It's pretty speedy too.

• InfluxDB

  www.influxdata.com featured

  Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.

  

• llama.cpp

  780 58,425 10.0 C++

  LLM inference in C/C++

They work great for me. No need to get fancy, the easiest thing is to use the distribution from the person who wrote the code that everyone else is using, it works great. It's at https://github.com/ggerganov/llama.cpp. I use it with everything from 7B to 65B models.

Suggest a related project