Source
Xiong et al. (2020), LIT-114 — ARXIV-2002.04745, the mean-field analysis. Nguyen and Salazar (2019), LIT-651, the first systematic evaluation, four months earlier.
This practice has no introduced_by, and the emptiness is a claim.
Pre-norm was in OpenNMT, tensor2tensor and Sockeye, and was used inside larger
systems, before anybody published an argument for it; the two earliest papers
that argue for it both attribute it to others. So there is no work that first
made this recommendation in the sense ADR-029 and ADR-030 mean, and
ADR-053 is what lets the record say so instead of promoting its best
citation into the slot. The frontmatter comment names what was searched, which
is how a reader refutes it.
The original Transformer normalizes after the residual addition (Post-LN). Xiong et al.’s mean-field analysis shows that this leaves the expected gradients of parameters near the output layer large at initialization, which is what makes early training fragile under a large learning rate. Moving the normalization inside the residual branch, applied to the sublayer’s input (Pre-LN), removes that.
The condition worth carrying: the paper’s argument is about initialization, and its headline consequence is that Pre-LN models can be trained with no warmup stage at all and reach comparable results in less time. The record still recommends warmup in SOTA-100, which was written against a different account of why warmup exists. LIT-114 flags that tension; this practice does not resolve it.
The trade Pre-LN makes is representational rather than numerical: the residual stream is never renormalized, so later layers see a stream whose magnitude grows with depth. That is the cost people cite when they revisit Post-LN or hybrid placements, and it is why this is a placement recommendation rather than a law.
Independent of the statistic — see SOTA-182. Pre-LN with centered LayerNorm is what GPT-2 does.
Known implementations
- llama2
Pre-norm, and why it changed what training needs
Placing the normalisation inside the residual branch — normalise, then sublayer, then add — leaves the residual stream itself unnormalised, so there is a clean additive path from the embedding to the output that nothing rescales. The original arrangement normalised after the addition, which puts a normalisation on every step of that path.
The consequence is about gradients at initialisation. Post-norm gives expected gradients at the output layer that grow with depth, which is what makes a large learning rate diverge early and what a warmup schedule exists to survive. Pre-norm bounds them, and LIT-114’s result is that with it the warmup stage can be removed entirely.
That is why this is one of the few architecture practices whose consequence is a training practice: SOTA-100’s warmup-proportional-to-model-size is the compensation the post-norm arrangement needed, and pre-norm is what made it optional.
The cost, which is real and shows up at scale
Pre-norm trades trainability for some final quality: the unnormalised residual stream grows in magnitude with depth, and deep pre-norm models can see later blocks contributing proportionally less — the representation collapse argument. Sandwich and peri-layernorm variants exist because of it.
That argument now has a source and a rate. Xie et al.
(LIT-639) derive it: the per-layer change in the normalised hidden
state decays as O(1/√k), and adding a block to an N−1 block model moves
the output by O(1/√N). THEORY-096 holds the account, and the
shape worth carrying is that the benefit and the cost are one mechanism
described twice — the residual path being unrenormalised is what lets
gradients reach the early blocks, and what lets the stream outgrow any single
block’s contribution.
Two qualifications on that rate, both in the theory document: it is derived
under an independence assumption that a trained network does not satisfy, and
O(1/√k) is slow — at 24 layers the per-layer change is about a fifth of its
value at the first, not a thousandth. “Collapse” names the direction and
overstates the speed.
Every large model in this record is pre-norm nonetheless, which is the honest summary: the stability is worth more than the margin, and the alternatives are refinements of pre-norm rather than returns to post-norm.
What “warm-up can be removed” costs
The headline consequence above — that Pre-LN models train without a warm-up stage — is Xiong et al.’s and is stated here without a number. LIT-639 supplies one, on IWSLT:
| method | warm-up | E6D6 | E12D12 |
|---|---|---|---|
| Post-LN | yes | 35.37 | fail |
| Post-LN | no | fail | fail |
| Pre-LN | yes | 35.12 | 35.18 |
| Pre-LN | no | 32.28 | 31.82 |
Removing warm-up costs 2.84 BLEU at E6D6 and 3.36 at E12D12. Pre-LN trains without it, which is what Xiong et al. claimed and what that paper tested; it does not train as well, which nobody had measured here.
That matters for the tension this practice flags with SOTA-100 and does not resolve. The tension is not “pre-norm made warm-up unnecessary and the record still recommends it” — on this evidence warm-up is still paying for itself under pre-norm, and the thing pre-norm removed is the requirement rather than the benefit. One dataset and one architecture family, so it narrows the claim rather than settling the question.
Worth noting where the number came from: Xie et al.’s own prose says Pre-LN “can train effectively without” warm-up, in the paragraph beneath a table showing a three-point drop. The table is the finding and the sentence repeats the received view.
The regime where post-norm wins, and how much of the rest is initialisation
Everything above is about stability and about low-resource or deep settings.
LIT-651 ran the comparison at high resource — base Transformer,
WMT’14 English-German, newstest2014, tokenized BLEU — and the result goes
the other way:
| BLEU | |
|---|---|
| PostNorm + LayerNorm | 27.58 |
| PostNorm + FixNorm + ScaleNorm | 27.57 |
| PreNorm + FixNorm + ScaleNorm | 27.07 |
| PreNorm + LayerNorm | 26.83 |
Post-norm by 0.75 BLEU, which the paper flags as surprising and reports Wang et al. (2019) observing too. Their summary is the one to carry: “while PostNorm performs better for high-resource NMT in the original base Transformer regime, PreNorm is both more stable and more competent in low-resource settings.”
Their speculation about the mechanism is not the collapse argument above — they suggest identity residual networks act like shallow ensembles and thereby “undermine the learning of the longest path”, and say further study is required. Two accounts of the same cost, neither adjudicated here.
The second qualification is sharper, because it is about the headline failure.
On low-resource en→vi, post-norm with the default Xavier initialization
fails to converge at 4k and 8k warm-up steps and reaches 5.76 BLEU at 16k.
Reduce the attention layers’ initialization (SmallInit) and the same
post-norm model reaches 28.17 at 4k, against pre-norm’s 28.52.
So the dramatic version of this practice’s motivation — post-norm diverges, pre-norm does not — is partly an initialization artefact. The paper’s own phrasing is the accurate one: smaller initialization “partly reclaims PostNorm’s stability”, and pre-norm is “less sensitive to this magnitude”. Less sensitive is a real advantage and a smaller one than “does not diverge”.
What survives all of this unchanged is depth without warm-up, where the placement difference is categorical rather than marginal: at five and six encoder/decoder layers with no warm-up, post-norm fails and pre-norm sits at 28.13 and 28.32.