Source
Rae et al. (2021), LIT-617, §3.2.
When this applies
You are training a dense autoregressive transformer past a few billion parameters, you clip the global gradient norm, and you inherited the clip value from a smaller run or from a framework default.
Do this
Treat the clip value as scale-dependent. The Gopher family used 1.0 up to 1.4B and 0.25 at 7.1B and 280B — a 4× tightening across roughly two orders of magnitude of parameters, introduced because the large runs needed it for stability.
What makes this worth filing rather than noting: three practices in this record tell you to clip and not one of them names a value. SOTA-035 says to clip, SOTA-071 says to use a dynamic threshold. A reader following them has to pick a number, and the number they will pick is a framework default set for models much smaller than the one they are training.
The values come from LIT-617’s training setup (§3.2). Rae et al. trained six models from 44M to 280B on the same dataset for the same 300B tokens, clipped the global gradient norm at 1.0, and reduced it to 0.25 for the 7.1B and 280B models “for improved stability”. The recommendation to treat the clip as scale-dependent is drawn from that configuration and originates there, though the paper reports the value rather than advising it.
Why it is only Proposed
The evidence is a reported configuration, not an ablation. Rae et al. state the value and the reason in one sentence and do not show the 1.0 run diverging, so what is established is that one group found 1.0 insufficient at 7.1B and above — not the shape of the relationship, not where the threshold sits, and not whether 0.25 is near-optimal or merely sufficient.
The direction is corroborated indirectly by everything else in the same
paragraph moving the same way with scale — the maximum learning rate falls
from 6×10⁻⁴ to 4×10⁻⁵ across the same family — but that is a pattern
about hyperparameters generally, not evidence about clipping.
Limitations
- The clip and the learning rate moved together, and the paper does not separate them. A reader who tightens the clip while holding the learning rate at a small-model value is not reproducing this configuration.
- Dense decoder-only transformers in bfloat16, trained with Adam. The same family also used stochastic rounding and later reported it cost performance, so the numerical setting these values were tuned inside is not one the record recommends.
- It says nothing about why. A tighter clip at scale is consistent with larger gradient-norm outliers, with a longer unstable early phase, or with the clip standing in for a learning-rate problem. Nothing here distinguishes them, and SOTA-071’s dynamic threshold is a different answer to what may be the same question.