Source
Dettmers et al. (2022), LIT-586 — ARXIV-2208.07339.
Two conditions, one on the word in the title and one on step 2
“Emergent” is contested. Ahmadian et al. (LIT-656) ran the controlled version of the question — same architecture, varying weight decay, dropout, gradient clipping and half-precision format, 410M to 52B, every variant from scratch and at comparable pre-quantization quality — and found the sensitivity tracks those choices rather than parameter count. Their 52B model loses nothing to plain INT8 where OPT-66B is reported to lose about 42%. THEORY-100 holds that account; SOTA-399 is what to do about it before training. The title keeps the word because it is the source’s and the technique is known by it.
The detection step has failed an independent replication. Step 2 below asks you to find the outlier dimensions. The same group reports that the published threshold of 6.0 was “too high to classify a feature dimension as an outlier for all the variants we consider”, and that after corresponding with the original authors and trying several adaptations, “we did not observe a clear correlation between these measures and sensitivity to quantization.”
That is the only independent attempt at this detection the record holds, and it did not work. It does not make the practice wrong — the implementations below ship a detector that works well enough in production — but a reader implementing step 2 from the paper should know that one group could not, and that the threshold may be specific to how the model was trained.
What to do
For the feed-forward and attention projection matrices:
- Quantize vector-wise — one normalization constant per inner product, not one per tensor.
- Detect the feature dimensions whose magnitudes are far outside the rest, and route those columns into a separate 16-bit matrix multiplication.
- Sum the two products.
More than 99.9% of values go through the 8-bit path.
This is LIT-586’s recipe, and the evidence for it is the same paper’s. Dettmers et al. proposed vector-wise quantization plus the mixed-precision decomposition as LLM.int8(), and report that a 175B 16/32-bit checkpoint converted this way runs at half the memory with no degradation on their evaluations at any scale they tested up to 175B.
Why the decomposition is the whole method
Naive int8 works on small transformers and breaks on large ones, and for years that read as “quantization gets harder with scale”. The paper’s contribution is the diagnosis: a small number of feature dimensions emerge with very large magnitudes, they recur across layers, and they dominate attention and prediction. A per-tensor scale chosen to cover them destroys the resolution of everything else.
So the fix is not a better global scale. It is to stop asking one scale to cover two very different distributions.
What it buys, and what it does not
Memory, and access. A 175B 16/32-bit checkpoint loads, converts and runs at roughly half the memory — the stated payoff is OPT-175B or BLOOM on a single server of consumer GPUs, not throughput. The extra 16-bit path and the outlier detection cost time, and reports of int8 inference being slower than 16-bit at small batch are consistent with this method rather than evidence against it.
The claim to watch
“Without any performance degradation” means no degradation the authors
measured, on their evaluations, up to 175B. It is a strong result and it is
an absence-of-evidence claim about a specific suite — the shape DP-010
names. The emergent-outlier characterization is the part later work has
built on; the no-degradation headline is the part a harder evaluation could
move, and this record holds no independent test of it.