CML_AMP_AI_Text_Summarization_with_Amazon_Bedrock
llm-awq
CML_AMP_AI_Text_Summarization_with_Amazon_Bedrock | llm-awq | |
---|---|---|
2 | 7 | |
1 | 1,902 | |
- | 10.9% | |
4.9 | 8.0 | |
8 months ago | 8 days ago | |
Jupyter Notebook | Python | |
- | MIT License |
Stars - the number of stars that a project has on GitHub. Growth - month over month growth in stars.
Activity is a relative number indicating how actively a project is being developed. Recent commits have higher weight than older ones.
For example, an activity of 9.0 indicates that a project is amongst the top 10% of the most actively developed projects that we are tracking.
CML_AMP_AI_Text_Summarization_with_Amazon_Bedrock
llm-awq
-
TinyChat: Large Language Model on the Edge
TinyChat is an efficient, lightweight, Python-native serving framework for 4-bit LLMs by AWQ. It delivers 2.3x generation speed up on RTX4090.
Code: https://github.com/mit-han-lab/llm-awq/tree/main/tinychat
- FLaNK Stack Weekly 23 Oct 2023
-
New base model InternLM 7B weights released, with 8k context window.
I am having trouble finding any 8bit GPTQ models at all, there don't seem to be any on HF it's almost all 4bit with the odd 3bit of the big ones. Suspect I will have to make my own for eval purposes but it's lower priority on my list then finding a 4bit that's GPU friendly but doesn't have such a performance penalty... Looking at AWQ they have 3 and 4bit versions.
-
Llama33B vs Falcon40B vs MPT30B
Using the currently popular gptq the 3bit quantization hurts performance much more than 4bit, but there's also awq (https://github.com/mit-han-lab/llm-awq) and squishllm (https://github.com/SqueezeAILab/SqueezeLLM) which are able to manage 3bit without as much performance drop - I hope to see them used more commonly.
- New hardware-friendly quantization method
-
Activation-Aware Weight Quantization for LLM Compression Outperforms GPTQ
Better quantization would have a direct and meaningful impact for everyone running local LLMs. The technique has already been applied to both Vicuna and the multimodal LLaMA variant LLaVA.
https://github.com/mit-han-lab/llm-awq
-
New quantization method AWQ outperforms GPTQ in 4-bit and 3-bit with 1.45x speedup and works with multimodal LLMs
GitHub: https://github.com/mit-han-lab/llm-awq
What are some alternatives?
JsonGenius - Get structured JSON data from any page.
SqueezeLLM - [ICML 2024] SqueezeLLM: Dense-and-Sparse Quantization
fastkafka - FastKafka is a powerful and easy-to-use Python library for building asynchronous web services that interact with Kafka topics. Built on top of Pydantic, AIOKafka and AsyncAPI, FastKafka simplifies the process of writing producers and consumers for Kafka topics.
GPTQ-for-LLaMa - 4 bits quantization of LLaMA using GPTQ
deep-chat - Fully customizable AI chatbot component for your website
Voyager - An Open-Ended Embodied Agent with Large Language Models
milvus-lite - A lightweight version of Milvus
langchain4j-examples
inference - A fast, easy-to-use, production-ready inference server for computer vision supporting deployment of many popular model architectures and fine-tuned models.
kafka-streams-dashboards - showcases Grafana dashboards for Kafka Stream applications leveraging client JMX metrics.
data-in-motion - This is repository for tutorials of Data In Motion starting with Data Distribution