Gibberish with LLaMa 7B 4bit

• text-generation-webui

  876 36,552 9.9 Python

  A Gradio web UI for Large Language Models. Supports transformers, GPTQ, AWQ, EXL2, llama.cpp (GGUF), Llama models.

For some background, running a GTX 1080 with 8GB of vram on Windows. Installed using a combination of the one-click installer, the How to guide by /u/Technical_Leather949, and using the pre-compiled wheel by Brawlence (to avoid having to install visual studio). I've downloaded the latest 4bit LLaMa 7b 4bit model, and the tokenizer/config files.

• text-generation-webui

  3 5 9.0 Python

  A gradio web UI for running Large Language Models like GPT-J 6B, OPT, GALACTICA, GPT-Neo, and Pygmalion. (by TheTerrasque)

git clone https://github.com/TheTerrasque/text-generation-webui.git

• InfluxDB

  www.influxdata.com featured

  Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.

  

• docker

  152 516 0.0 Go

  Docker - the open-source application container engine (by microsoft)

  

One alternative you could try.. I've set up a docker environment to build things and set it up. It would require you to install some tools if you don't have: Git and Docker Desktop for Windows.

Suggest a related project