Source
Shoeybi et al. (2019), LIT-022 — ARXIV-1909.08053.
Megatron-LM, LIT-022, states the recipe in exactly this form: weights drawn
from N(0, 0.02), then the weights immediately before each residual scaled
by 1/sqrt(2N), N the number of transformer layers — and trains GPT-2-style
models to 8.3B parameters with it. That is a report of what was used, not a
measurement of what it buys; the paper runs no arm without it. Narayanan et
al. (LIT-043), cited here before, says nothing about initialization.
What the smaller variance is protecting against
The residual stream accumulates the output of every layer, so its variance grows with depth unless something holds it down. Initialising the output projection of each block with a smaller standard deviation — scaled down by the number of layers — keeps each block’s contribution small relative to what is already in the stream, so the signal at the top of a deep model is not dominated by initialisation noise.
That is the same reasoning that puts a 1/√(2·n_layers) factor on the output projections in GPT-2-style initialisations, and the same problem ReZero (SOTA-051) attacks by starting the residual branch at literally zero.
That factor has an author, and it is GPT-2. Radford et al. (2019),
LIT-733 §2.3, uses “a modified initialization which accounts for the
accumulation on the residual path with model depth”, scaling the weights of
residual layers by 1/sqrt(N) with N the number of residual layers — the
same quantity as 2·n_layers, two residual layers per block. Child et al.
(2019), LIT-225 §5.2, two months later, writes it as 1/sqrt(2N) over
blocks and states the invariant it is protecting: the ratio of
input-embedding scale to residual-block scale, held constant across values of
N. Child’s base initialisation is not 0.02. GPT-2 says it “largely
follows” GPT, whose report initialises every weight from N(0, 0.02), and
Megatron-LM writes the two down together. Named in prose and cited nowhere
until ADR-029 gave the record a field for it, and attributed to Child until
the earlier report was checked.
What the title used to get wrong
Until version 4 it said “layer norms with smaller variance (0.02)”, and the initialisation that matters here is the output projections of the attention and MLP blocks. LayerNorm’s own parameters are conventionally initialised to weight 1 and bias 0 — which is what SOTA-025 and SOTA-026 say, and 0.02 is not that.
0.02 is also a familiar number for a different reason: it is the standard deviation of the normal distribution GPT-2 and its descendants use to initialise all weights, before the depth scaling is applied on top. Two distinct conventions have been merged into one sentence.
This was flagged rather than rewritten for two versions, because fixing it meant deciding which of the two the practice is about. Megatron-LM (LIT-022) decides it: it states both together, as one recipe, so the title now names both. Neither it nor GPT-2 ablates the recipe. The evidence for it is that it is the setup large runs report using, not a measurement that it beats an alternative.