[P] Discord Chatbot for LLaMA 4-bit quantized that runs 13b in <9 GiB VRAM

• llama.cpp

  773 56,891 10.0 C++

  LLM inference in C/C++

I'm running it using https://github.com/ggerganov/llama.cpp. The 4-bit version of 13b runs ok without GPU acceleration.

• yal-discord-bot

  5 72 6.5 Python

  Yet Another LLaMA/ALPACA Discord Bot

• InfluxDB

  www.influxdata.com featured

  Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.

  

• text-generation-webui

  876 36,293 9.9 Python

  A Gradio web UI for Large Language Models. Supports transformers, GPTQ, AWQ, EXL2, llama.cpp (GGUF), Llama models.

My question seemed to have been answered here, and it is a VRAM limitation. Also, that last link seems to support 4-bit models as well. Doesn't seem too bad to set up.... Though I installed A1111 when it first came out, so I learned through the garbage of that. Lol.

• transformers

  176 125,369 10.0 Python

  🤗 Transformers: State-of-the-art Machine Learning for Pytorch, TensorFlow, and JAX.

  

• GPTQ-for-LLaMa

  1 - -

• GPTQ-for-LLaMa

  75 2,916 8.6 Python

  4 bits quantization of LLaMA using GPTQ

• pifs

  65 6,555 0.0 C

  πfs - the data-free filesystem!

• SaaSHub

  www.saashub.com featured

  SaaSHub - Software Alternatives and Reviews. SaaSHub helps you find the best software and product alternatives

  

Suggest a related project