Source
Bondarenko et al. (2023), LIT-414 — ARXIV-2306.12929, NeurIPS 2023.
The reason it works is THEORY-019: the softmax denominator forces a head to place its mass somewhere, so a head with nothing to attend to can only approach a no-op by making its scores extreme. This gives it a way to reach zero in finite range instead.
The change
clipped_softmax(x; ζ, γ) := clip((ζ − γ)·softmax(x) + γ, 0, 1)
with γ ≤ 0 and ζ ≥ 1. Stretch the output to (γ, ζ), clip back to
(0,1). Exact zeros are now reachable — values below −γ/(ζ−γ) round to
zero — and a clipped value contributes no gradient, so nothing keeps
driving the score outward.
It is a drop-in replacement for the softmax and adds no parameters.
What it bought, on BERT-base
| FP16 ppl ↓ | max ‖x‖∞ | avg kurtosis | W8A8 ppl ↓ | |
|---|---|---|---|---|
| vanilla | 4.49±0.01 | 735±55 | 3076±262 | 1294±1046 |
γ = −0.03 | 4.41±0.01 | 20±1 | 80±6 | 4.55±0.01 |
Two things worth separating. INT8 post-training quantization goes from broken to working — 1294 perplexity is not a degraded model, it is a destroyed one. And the floating-point model gets slightly better, so the outliers were not carrying something the model needed.
Clipped softmax originates in LIT-414, which is also where these numbers come from. Bondarenko et al. traced the outliers that break INT8 to heads trying to learn a no-op, proposed clipped softmax and gated attention as two architectural fixes for it, and measured both; on their summary comparison clipped softmax was the more consistent of the two (infinity-norm 21.5±1.5 against 39.2±26.0 for gating).
γ < 0 is where essentially all of the improvement comes from; stretching
only the top end (ζ > 1) does almost nothing. Reported across BERT-base
(109M), OPT-125M and ViT-S/16 on ImageNet-1K, each trained twice with each
quantization run repeated three times.
What this does not claim
It has to be pre-trained in. This is an architectural change that shapes how outliers form during training, not a post-hoc quantization technique. It does nothing for a checkpoint you already have — which is the whole difference between it and the calibration-side literature.
The scale is the reason for Proposed. 109M encoder, 125M decoder, 22M
vision. Nothing here is at the scale the practice registry mostly advises on,
and the outlier phenomenon is known to intensify with size, so the
extrapolation runs in the direction where it has not been checked.
Its sibling has better standing. Gated attention, from the same paper, was
confirmed at 15B by an independent group and is SOTA-134,
Active. That is evidence for the shared diagnosis, not for this remedy —
the two are different interventions and only one of them has been scaled.
Untested against the third option. softmax₁ — one added to the
denominator (LIT-413) — reaches for the
same property by a simpler route and has never been evaluated against this.
Known implementations
- None known outside the paper.