LLM-As-Chatbot
| minimal-llama | LLM-As-Chatbot | |
|---|---|---|
| 4 | 3 | |
| 456 | 3,237 | |
| - | - | |
| 8.5 | 9.0 | |
| 7 months ago | 6 months ago | |
| Python | Python | |
| - | Apache License 2.0 |
Stars - the number of stars that a project has on GitHub. Growth - month over month growth in stars.
Activity is a relative number indicating how actively a project is being developed. Recent commits have higher weight than older ones.
For example, an activity of 9.0 indicates that a project is amongst the top 10% of the most actively developed projects that we are tracking.
minimal-llama
- Show HN: Finetune LLaMA-7B on commodity GPUs using your own text
-
Visual ChatGPT
I can't edit my comment now, but it's 30B that needs 18GB of VRAM.
LLaMA-13B, GPT-3 175B level, only needs 10GB of VRAM with the GPTQ 4bit quantization.
>do you think there's anything left to trim? like weight pruning, or LoRA, or I dunno, some kind of Huffman coding scheme that lets you mix 4-bit, 2-bit and 1-bit quantizations?
Absolutely. The GPTQ paper claims negligible output quality loss with 3-bit quantization. The GPTQ-for-LLaMA repo supports 3-bit quantization and inference. So this extra 25% savings is already possible.
As of right GPTQ-for-LLaMA is using a VRAM hungry attention method. Flash attention will reduce the requirements for 7B to 4GB and possibly fit 30B with a 2048 context window into 16GB, all before stacking 3-bit.
Pruning is a possibility but I'm not aware of anyone working on it yet.
LoRa has already been implemented. See https://github.com/zphang/minimal-llama#peft-fine-tuning-wit...
LLM-As-Chatbot
-
OpenAI's GPT-4 Red Teamer Nathan Labenz: the GPT-4 base model recommends assassinating humans, naming specific targets
The first one is from https://github.com/deep-diver/Alpaca-LoRA-Serve
-
Show HN: ChatLLaMA – A ChatGPT style chatbot for Facebook's LLaMA
this is useless because it doesn't handle context:
Q: Name five genres of music.
A: Jazz, country, hip-hop, blues, classical.
Q: Name a famous artist from the third genre.
A: Salvador Dalí.
Whereas this one actually supports context: https://github.com/deep-diver/Alpaca-LoRA-Serve
- Show HN: Finetune LLaMA-7B on commodity GPUs using your own text
What are some alternatives?
FlexGen - Running large language models on a single GPU for throughput-oriented scenarios.
alpaca-lora - Instruct-tune LLaMA on consumer hardware
visual-chatgpt - Official repo for the paper: Visual ChatGPT: Talking, Drawing and Editing with Visual Foundation Models [Moved to: https://github.com/microsoft/TaskMatrix]
simple-llm-finetuner - Simple UI for LLM Model Finetuning
whisper.cpp - Port of OpenAI's Whisper model in C/C++
peft - 🤗 PEFT: State-of-the-art Parameter-Efficient Fine-Tuning.
hh-rlhf - Human preference data for "Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback"
alpaca-7b-truss
GPTQ-for-LLaMa - 4 bits quantization of LLaMA using GPTQ
stanford_alpaca - Code and documentation to train Stanford's Alpaca models, and generate the data.