New quantization method AWQ outperforms GPTQ in 4-bit and 3-bit with 1.45x speedup and works with multimodal LLMs

• GPTQ-for-LLaMa

  75 2,913 8.6 Python

  4 bits quantization of LLaMA using GPTQ

And exactly what Triton version are they comparing against? I just tried the latest version of this, and on my 4090/12900K I get 77 tokens per second for Llama 7B-128g. My own GPTQ CUDA implementation gets 151 tokens/second on the same model, same hardware. That makes it 96% faster, whereas AWQ is only 79% faster. For 30B-128g I'm currently only getting a 110% speedup over Triton compared to their 178%, but it still seems a little disingenuous to compare against their own CUDA implementation only, when they're trying to present the quantization method as being faster for inference.

• llm-awq

  7 1,794 8.0 Python

  AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration

  

GitHub: https://github.com/mit-han-lab/llm-awq

• InfluxDB

  www.influxdata.com featured

  Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.

  

• Voyager

  53 5,152 4.7 JavaScript

  An Open-Ended Embodied Agent with Large Language Models (by MineDojo)

  

Summary of the study by Claude-100k if anyone is interested:

Suggest a related project