AWQ (Activation-aware Weight Quantization)

AWQ (Activation-aware Weight Quantization) is a low-bit, weight-only quantization method for large language models, introduced by Lin et al. and recognized with a best-paper award at MLSys 2024. Its central observation is that not all weights matter equally: protecting roughly the top 1% of "salient" weight channels dramatically reduces the error introduced by quantization. This lets AWQ compress models to INT3 or INT4 while preserving quality, which is useful for anyone running models locally on a desktop or even a mobile GPU.

The activation-aware insight

Rather than deciding which weights are important by looking at the weights themselves, AWQ looks at the activation distribution observed on a small calibration set. Channels that see large activations are treated as salient. AWQ then searches for a per-channel scaling factor that effectively gives those important channels more headroom before quantization, reducing their relative error. Because the method relies on scaling rather than backpropagation or reconstruction, it avoids overfitting to the calibration data and tends to generalize across domains and even modalities.

Practical benefits

AWQ ships with an inference framework tuned for edge devices, and the original work reported more than a 3x speedup over a baseline FP16 implementation on both desktop and mobile GPUs. It is a popular choice for GPU-served INT4 deployments and is widely supported across local-inference tooling.

For comparison with the second-order approach, see GPTQ, and for the bigger picture review our LLM quantization entry.

See what fits at 4-bit in the GPU–LLM fit dataset.