Source
Ba et al. (2016), LIT-005 — ARXIV-1607.06450, where the gain is introduced.
What the initialisation controls
LayerNorm’s learned gain multiplies the normalised activation, so its initial value sets how much of the block’s output reaches the residual stream at step one. Starting at 1 is the identity: the normalised signal passes through unchanged, and the network begins as the architecture without the learned adjustment rather than with an arbitrary one.
It is a convention, not a result
No paper in the record argues for it, including this one. LIT-005 introduces the gain and bias — “each neuron its own adaptive bias and gain” — and states no initialisation; the string “set to 1” does not appear in it. Every framework initialises the gain to 1 because identity is the obvious default, and nobody published that.
The record files it anyway, because a reader who does not know it is a convention will go looking for the paper that established it, and the useful thing to say is that there isn’t one.
What the former source actually argues
This practice cited LIT-025 until #114, and its title carried a range — “0.97–1.0” — attributed there. That paper contains the string “0.97” zero times. What it argues is that the gain and bias increase the risk of over-fitting and do not work in most cases, and that removing them outperforms keeping them on four datasets. It is a case for deleting the
parameter this practice initialises, and it is now filed as SOTA-191.
So the record briefly held a recommendation about how to set a parameter, sourced to the paper arguing that parameter should not exist. The range was the tell — the body already called it a constant nobody had grounds for, alongside SOTA-078’s buffer and SOTA-089’s alignment — but flagging an unsourced number is not the same as checking whether the source says anything at all.
Where it stands now
RMSNorm (SOTA-182) is the current default and keeps this gain while dropping the centering and the bias, so the practice applies to it unchanged and SOTA-026 does not.
The shrink-below-identity idea the old range gestured at is real and lives elsewhere with sources: SOTA-060’s scaled output projections and SOTA-051’s near-zero final layer both damp the residual branch at initialisation. That is the instinct the 0.97 was reaching for, and those are the practices that carry it.