anthology-of-the-sota

LIT-081: GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers

3 min read

Properties7
StatusActive
Version3
Tagsnumerics-and-precision, inference-optimization
Extended byLIT-567 — MARLIN: Mixed-Precision Auto-Regressive Parallel Inference on Large Language Models
Compared againstLIT-585 — AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration
Cited as “introduced by” bySOTA-185 — Quantize weights after training by compensating each rounding error into the columns not yet quantized
Cited as “source” bySOTA-185 — Quantize weights after training by compensating each rounding error into the columns not yet quantized

Graph View

Frantar et al. (2022) — ARXIV-2210.17323

Key takeaways

  • The problem with round-to-nearest is that it treats weights as independent. They are not: the layer’s output error depends on how the weights interact, so a rounding error made in one column can be absorbed by adjusting columns that have not been quantized yet.
  • The method quantizes column by column and compensates, using approximate second-order (Hessian) information estimated from a small calibration set. This is a one-shot procedure — no retraining, no gradient steps.
  • The headline: 3- and 4-bit weight quantization of models up to 175B in roughly four GPU-hours, with accuracy close to the uncompressed baseline. Reaching 4 bits at that scale had previously meant quantization-aware training.
  • It is weight-only. The win is memory footprint and bandwidth, which is what makes decode faster, and it does nothing for a compute-bound prefill.
  • The calibration set is an input, not an implementation detail. A calibration distribution unlike the serving distribution is where this degrades.

Standing in the anthology

The source of SOTA-185, which files the approach rather than the artifact: error-compensating one-shot PTQ with a calibration set is what converged, and the specific search and ordering have been improved on repeatedly since.

AWQ (LIT-585) ran the head-to-head against this method, with and without its reordering trick, and reports lower perplexity than both on LLaMA and Llama-2 at 7B to 70B. The sharper result bears on the calibration bullet above: on OPT-6.7B at INT3-g128, calibrating on one Pile subset (PubMed or Enron) and evaluating on the other cost AWQ 0.5–0.6 perplexity and this method 2.3–4.9, and AWQ reached better perplexity from 16 calibration sequences than this did from 192.

Unfiled for a year, and the diagnosis has since been overtaken twice. The note is tagged inference-optimization, which was not one of the seven topics a practice could carry — but the record already held quantization practices under systems-optimization and training-optimization, so the tag was concealing the paper rather than excluding it.

ADR-021 added inference-optimization to the practice vocabulary, and ADR-026 has since made the two vocabularies identical. No LIT tag can look like a scope boundary any more, because every one of them is a word a practice can carry. That removes the mechanism this paragraph describes; the thing it was a symptom of — a note nobody filed a practice from and nobody recorded a reason for — is the part still worth watching.

Backlinks