Source
Lin et al. (2023), LIT-585 — ARXIV-2306.00978.
What to do
Before quantizing a linear layer’s weights to 4 bits:
- Run a small calibration set forward and collect activation statistics per input channel.
- Take the channels with the largest activations — about 1% — as salient.
- Scale those weight channels up, and scale the corresponding activations down by the same factor, so the layer computes the same function.
- Round everything to the target width uniformly.
There is no backward pass and nothing is fitted to a reconstruction objective.
The method is LIT-585’s. Lin et al. proposed it as AWQ, reporting that protecting roughly 1% of channels recovers most of the quantization error provided the channels are chosen from activations, and deriving the scaling as an equivalent transformation that cuts the salient channels’ relative rounding error while leaving the layer’s function unchanged.
The two choices that carry it
Salience is read from activations, not from weights. This is the part the name states and the part that is easy to get backwards. A weight channel is worth protecting because the activations flowing through it are large, and weight magnitude is a poor proxy for that. Choosing by weight magnitude is a different and worse method.
The protection is a scaling, not a precision carve-out. Keeping 1% of channels in 16 bits is the obvious implementation and it makes the kernel irregular, which costs more than the quantization saves. Scaling instead leaves every weight at the same width, so the matrix multiply stays dense and uniform.
When this is the wrong tool
This is weight-only quantization: the weights shrink, the activations stay in 16-bit. It buys memory and the bandwidth that follows from it, and it does not buy int8 arithmetic. A deployment that is compute-bound rather than bandwidth-bound gets much less from it than the memory figures suggest.
Against GPTQ
SOTA-185 solves the same problem by compensating each rounding error into
the columns not yet quantized, using second-order information from a
calibration set. Both are one-shot post-training methods at similar bit
widths, and the record carries both because the field runs both.
The distinguishing claim is generalization, not accuracy at a bit width. GPTQ fits a calibration objective; AWQ deliberately does not, and its argument for itself is that this is why it holds on instruction-tuned and multi-modal models rather than only on the calibration distribution. A reader choosing between them should be checking that claim, not the headline perplexity — and the record does not hold an independent test of it.
What it does hold is the authors’ own. Lin et al. ran the comparison with SOTA-185 in LIT-585, against GPTQ with and without its reordering trick. Calibrating on one of PubMed or Enron and evaluating on the other costs AWQ 0.5–0.6 perplexity and GPTQ 2.3–4.9, and AWQ does better with a calibration set ten times smaller. At a fixed width the gap is narrower: at INT3 with group size 128, Llama-2-7B scores 6.24 with AWQ against 6.43 for GPTQ and 6.42 with reordering, and at INT4 the two are within 0.05 of each other on every model in the table.