Damn, I was so satisfied with my 3080 with 10GB of VRAM until I found this subreddit.

• exllama

  64 2,624 9.0 Python

  A more memory-efficient rewrite of the HF transformers implementation of Llama for use with quantized weights.

For exllama (https://github.com/turboderp/exllama) the instructions are on the post itself.

• KoboldAI

  58 150 8.6 Python

• InfluxDB

  www.influxdata.com featured

  Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.

  

• exllama

  6 7 9.0 Python

  A more memory-efficient rewrite of the HF transformers implementation of Llama for use with quantized weights. (by 0cc4m)

• koboldcpp

  180 3,951 10.0 C++

  A simple one-file way to run various GGML and GGUF models with KoboldAI's UI

Just about any llama-based model can be run purely on your CPU, or split between your CPU and GPU. Download KoboldCPP, assign as many layers to your GPU as it can handle, and let the CPU and system RAM handle the rest.

• gpt4all

  139 65,076 9.8 C++

  gpt4all: run open-source LLMs anywhere

  

Specifically with this project: https://github.com/nomic-ai/gpt4all

• SaaSHub

  www.saashub.com featured

  SaaSHub - Software Alternatives and Reviews. SaaSHub helps you find the best software and product alternatives

  

Suggest a related project