[D] Tutorial: Run LLaMA on 8gb vram on windows (thanks to bitsandbytes 8bit quantization)

• text-generation-webui

  876 36,552 9.9 Python

  A Gradio web UI for Large Language Models. Supports transformers, GPTQ, AWQ, EXL2, llama.cpp (GGUF), Llama models.

conda create -n textgen conda activate textgen conda install torchvision torchaudio pytorch-cuda=11.7 git -c pytorch -c nvidia git clone https://github.com/oobabooga/text-generation-webui cd text-generation-webui pip install -r requirements.txt

• bitsandbytes-win-prebuilt

  4 75 10.0

put libbitsandbytes_cuda116.dll in C:\Users\xxx\miniconda3\envs\textgen\lib\site-packages\bitsandbytes\

• InfluxDB

  www.influxdata.com featured

  Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.

  

• llama-cpu

  9 775 3.1 Python

  Fork of Facebooks LLaMa model to run on CPU

I tried to port the llama-cpu version to a gpu-accelerated mps version for macs, it runs, but the outputs are not as good as expected and it often gives "-1" tokens. Any help and contributions on fixing it are welcome!

• llama-mps

  4 83 3.8 Python

  Experimental fork of Facebooks LLaMa model which runs it with GPU acceleration on Apple Silicon M1/M2

I tried to port the llama-cpu version to a gpu-accelerated mps version for macs, it runs, but the outputs are not as good as expected and it often gives "-1" tokens. Any help and contributions on fixing it are welcome!

• awesome-ml

  27 1,422 8.8

  Curated list of useful LLM / Analytics / Datascience resources

use the prebuilt windows wheels or my WSL2 solution

• one-click-installers

  18 470 8.9 Python

  Discontinued Simplified installers for oobabooga/text-generation-webui.

• transformers

  176 125,369 10.0 Python

  🤗 Transformers: State-of-the-art Machine Learning for Pytorch, TensorFlow, and JAX.

  

git clone https://github.com/huggingface/transformers.git cd transformers pip install -e .

• SaaSHub

  www.saashub.com featured

  SaaSHub - Software Alternatives and Reviews. SaaSHub helps you find the best software and product alternatives

  

Suggest a related project