Lin et al. (2023) — ARXIV-2306.00978
Key takeaways
- Not all weights matter equally, and the weights do not tell you which. Protecting roughly 1% of channels recovers most of the quantization error, and the channels worth protecting are identified from the activation distribution, not from weight magnitude. That is the paper’s central claim and the one its name states.
- The protection is a scaling, not a mixed-precision carve-out. Keeping 1% of channels in 16 bits would be hardware-inefficient — it makes the kernel irregular. Instead the salient channels are scaled up by an equivalent transformation before rounding, which the paper derives as reducing their relative quantization error while leaving the layer’s function unchanged. The scale comes from activation statistics collected offline.
- No backpropagation, no reconstruction. This is the difference the authors press on versus methods that fit a calibration objective: nothing is optimized against the calibration set, so the result transfers across domains and modalities. They report it holding for instruction-tuned models and, they claim first, for multi-modal models.
- TinyChat is the accompanying kernel work, reporting more than 3× over an FP16 Hugging Face baseline on desktop and mobile GPUs through kernel fusion and platform-aware weight packing. Worth separating from the quantization claim: the speedup is the kernels, the accuracy is AWQ.
What this is doing in the record
Filed from #289, which swept what the big libraries implement. AWQ is
carried by both vllm and transformers — one of only three items in 91
that two independent adoption indexes agreed on — while the record held
GPTQ (LIT-081) three times and its direct competitor not at all.
compared_against: LIT-081 because the paper runs that comparison itself,
which is what ADR-011 requires of the field: somebody ran it, rather than
the record noticing a family resemblance.
Reading it against GPTQ
The two solve the same problem and disagree about the mechanism, which is the interesting part and is why both belong in the record. GPTQ treats quantization as an error-compensation problem — round a column, push the error into the columns not yet done, using second-order information from a calibration set. AWQ treats it as a salience problem — find the channels that matter from activations, scale them, round everything once.
The claim that distinguishes them is about generalization: GPTQ fits a calibration objective and AWQ deliberately does not, and AWQ’s argument for itself is that this is why it holds up on instruction-tuned and multi-modal models. That is a claim about robustness rather than about accuracy at a bit width, and it is the one a reader choosing between them should be checking.
The paper runs that check against GPTQ (LIT-081) directly, with and without GPTQ’s reordering trick. Calibrating on one of PubMed or Enron and evaluating on the other costs AWQ 0.5–0.6 perplexity and GPTQ 2.3–4.9 (Figure 8), and AWQ reaches better perplexity with a ten times smaller calibration set. At a fixed bit width the gap is smaller: in Table 4, at INT3 with group size 128, Llama-2-7B scores 6.24 with AWQ against 6.43 for GPTQ and 6.42 with reordering (FP16 5.47). At INT4, AWQ and reordered GPTQ are within 0.05 of each other on every model in the table.