Source
Ba et al. (2016), LIT-005 — ARXIV-1607.06450, where the bias is introduced.
Zero is the identity here too
The bias is added after the normalised activation is scaled, so initialising it to zero means the layer starts as pure normalisation-and-scale with no offset. Any other starting value asserts a preferred direction in feature space before training has said anything, which is the thing initialisation schemes generally try to avoid.
There is no interesting trade here, which is worth stating plainly: this is a convention with no live alternative, and the practice’s value is as a statement of what the default is rather than as advice between options.
What the former source actually argues
This cited LIT-025 until #114. That paper’s finding is that the bias and gain increase the risk of over-fitting and do not work in most cases, and that LayerNorm with both removed beats LayerNorm on four datasets. It is a case for deleting the parameter this practice initialises, not for setting it, and it
is now filed as SOTA-191.
LIT-005 introduces the bias and states no initialisation for it, so the zero is a convention rather than a result — the same standing as SOTA-025’s gain, and said plainly for the same reason.
Where it stops applying
RMSNorm has no bias at all. It normalises by root-mean-square without centring, on the finding that the mean subtraction contributes little, and drops both the centring and the offset — so in the models this record mostly concerns itself with (SOTA-006), there is no parameter for this practice to initialise.
Kept rather than retired because BatchNorm and LayerNorm are both still in use and the convention is real. But it is a practice about a component the frontier has largely moved past, which the record should say rather than leave the reader to infer from its absence elsewhere.