Xiong et al. (2020) — ARXIV-2002.04745
Key takeaways
- Warmup had been a required, carefully hand-tuned stage with no account of why: it slows optimization down and adds hyperparameters, and everyone used it anyway. The paper’s contribution is to explain it and then show when it can be dropped.
- The mean-field result: in the original Post-LN Transformer, which puts layer normalization between the residual blocks, the expected gradients of parameters near the output layer are large at initialization. A large learning rate applied to those gradients is what makes early training unstable, and warmup is the workaround.
- Move the normalization inside the residual block — Pre-LN — and the gradients are well-behaved at initialization, so the reason for warmup goes away. Pre-LN Transformers trained with no warmup stage reach comparable results in significantly less time and with less hyperparameter tuning.
- The general lesson the record cares about: a schedule that everyone treats as universal can turn out to be compensating for one architectural choice.
Standing in the anthology
The source of SOTA-100, warmup proportional to model size. Read the practice against the paper: this work says the need for warmup is a property of where the normalization sits, and that under Pre-LN — which every model in the record’s modern half uses — it can be removed entirely. LIT-152 reaches a compatible conclusion for a different schedule parameter, finding that Muon plus the architecture make batch-size warmup unnecessary. SOTA-100 deserves a re-reading in that light; this note files the evidence rather than making the call.