AutoGPTQ
catai
| AutoGPTQ | catai | |
|---|---|---|
| 19 | 7 | |
| 3,806 | 408 | |
| 5.0% | 1.7% | |
| 9.3 | 8.6 | |
| 5 days ago | 3 months ago | |
| Python | TypeScript | |
| MIT License | MIT License |
Stars - the number of stars that a project has on GitHub. Growth - month over month growth in stars.
Activity is a relative number indicating how actively a project is being developed. Recent commits have higher weight than older ones.
For example, an activity of 9.0 indicates that a project is amongst the top 10% of the most actively developed projects that we are tracking.
AutoGPTQ
- Setting up LLAMA2 70B Chat locally
- Experience of setting up LLAMA 2 70B Chat locally
-
GPT-4 Details Leaked
Deploying the 60B version is a challenge though and you might need to apply 4-bit quantization with something like https://github.com/PanQiWei/AutoGPTQ or https://github.com/qwopqwop200/GPTQ-for-LLaMa . Then you can improve the inference speed by using https://github.com/turboderp/exllama .
If you prefer to use an "instruct" model à la ChatGPT (i.e. that does not need few-shot learning to output good results) you can use something like this: https://huggingface.co/TheBloke/Wizard-Vicuna-30B-Uncensored...
-
Loader Types
AutoGPTQ: an attempt at standardizing GPTQ-for-LLaMa and turning it into a library that is easier to install and use, and that supports more models. https://github.com/PanQiWei/AutoGPTQ
- WizardLM-33B-V1.0-Uncensored
-
Any help converting an interesting .bin model to 4 bit 128g GPTQ? Bloke?
Just use the script: https://github.com/PanQiWei/AutoGPTQ/blob/main/examples/quantization/quant_with_alpaca.py
-
LLM.int8(): 8-Bit Matrix Multiplication for Transformers at Scale
In the wild, people tend to use GTPQ quantization for pure GPU inference: https://github.com/PanQiWei/AutoGPTQ
And ggml's quant for CPU inference with some offload, which just got updated to a more GPTQ-like method days ago: https://github.com/ggerganov/llama.cpp/pull/1684
Some other runtimes like Apache TVM also have their own quant implementations: https://github.com/mlc-ai/mlc-llm
For training, 4-bit bitsandbytes is SOTA, as far as I know.
TBH I'm not sure why this November paper is being linked. Few are running 8 bit models when they could fit a better 3-5 bit model in the same memory pool.
-
Introducing Basaran: self-hosted open-source alternative to the OpenAI text completion API
Instead of integrating GPTQ-for-Lllama, use AutoGPTQ instead.
- AutoGPTQ - An easy-to-use LLMs quantization package with user-friendly apis, based on GPTQ algorithm
catai
-
Are you sure you are focusing on the right things? (venting)
The easiest tool I found is CatAI: https://github.com/ido-pluto/catai You just type 3 npm commands and THATS IT! You have your own Chat Web UI on your computer without hundrets of settings
- How to use CatAI to apologize to your boss
-
Wizard-Vicuna-13B-Uncensored
I am a noob. I saw your comment on github and another post here. I am confused about what has changed and what us users have to do. Do we have to update llama.cpp and redownload all the models(I am using something called catai instead of the webui, i think it also uses llama.cpp)? How do we know which versions of the models are compatible with which vesions of llama.cpp?
-
Google removes the waitlist on Bard today and will be available in 180 more countries
https://github.com/ggerganov/llama.cpp https://github.com/oobabooga/text-generation-webui https://github.com/mlc-ai/mlc-llm https://github.com/cocktailpeanut/dalai https://github.com/ido-pluto/catai (this is super easy to install but it doesnt provide an api or have integration with langchain)
-
GPT For All 13B (/GPT4All-13B-snoozy-GPTQ) is Completely Uncensored, a great model
Pretty simple using catai.
- How to run something like chatgpt, locally?
-
How to install Wizard-Vicuna
You can check out the original GitHub project here
What are some alternatives?
exllama - A more memory-efficient rewrite of the HF transformers implementation of Llama for use with quantized weights.
alpaca-electron - The simplest way to run Alpaca (and other LLaMA-based local LLMs) on your own computer
llama.cpp - LLM inference in C/C++
HyunGPT - chatbot thing
text-generation-webui - A Gradio web UI for Large Language Models. Supports transformers, GPTQ, AWQ, EXL2, llama.cpp (GGUF), Llama models.
basaran - Basaran is an open-source alternative to the OpenAI text completion API. It provides a compatible streaming API for your Hugging Face Transformers-based text generation models.
dalai - The simplest way to run LLaMA on your local machine
GPTQ-for-LLaMa - 4 bits quantization of LLaMA using GPTQ
mlc-llm - Enable everyone to develop, optimize and deploy AI models natively on everyone's devices.
self-refine - LLMs can generate feedback on their work, use it to improve the output, and repeat this process iteratively.