Source
Xu et al. (2019), LIT-025 — ARXIV-1911.07013.
The finding
The paper sets out to explain why LayerNorm works and arrives somewhere its authors did not expect. The received account is forward normalization — controlling the distribution of a layer’s inputs. Their analysis says the benefit is in the backward pass instead: the derivatives of the mean and variance re-center and re-scale the gradients, and that is what buys the smoother training.
The consequence for the learnable parameters is the part that becomes a
practice. If the forward normalization is not doing the work, the bias and
gain that adjust it are not either — and worse, they increase the risk of
over-fitting. LayerNorm-simple, with both removed, outperforms LayerNorm
on four of the datasets tried and reaches state of the art on En-Vi machine
translation. The paper’s own proposal, AdaNorm, replaces them with an
input-dependent transformation and beats LayerNorm on seven of eight.
Both halves of that come from LIT-025, and so does the recommendation. The backward-pass account rests on its detaching experiment, which keeps the forward computation identical, removes only the mean and variance derivatives from the backward pass, and loses performance. The case for removal is LayerNorm-simple, a result the authors call “beyond our expectation”; it removes the bias and gain together throughout, so the paper cannot say which of the two carries the effect.
Why this is Proposed, and why it is filed at all
The field went the other way. Every model in this record keeps a learnable gain, and RMSNorm (SOTA-182) — now effectively universal — keeps the gain while dropping the centering, which is the opposite half from the one this paper calls expendable. Two papers, one conclusion each, pointing in different directions about which part of LayerNorm is load-bearing. The other paper is LIT-023, RMSNorm’s source, the same year: it hypothesises that re-centering is dispensable and keeps the re-scaling, gain included, which disputes this practice’s claim that the gain is the part to remove.
The evidence here is 2019-era sequence models and machine translation, not decoder-only language models at scale, and no adopter in the record has tested the removal. That is enough to file it and not enough to recommend it.
It is filed rather than dropped for a specific reason. Three practices — SOTA-025, SOTA-026 and SOTA-027 — cited this paper for a decade of the record’s life while recommending how to set the two parameters it argues should be deleted. Leaving the note with no practice would leave the record with no statement of what it actually says, which is how that inversion survived being read.
A second data point at scale, on the other side
LIT-088 (ViT-22B) applies its LayerNorms “without bias and centering”,
citing RMSNorm — so it keeps the gain and drops the centering, the opposite
half from the one this practice calls expendable. That is one more model at 22B
taking SOTA-182’s side rather than this one, and it does not meet the
promote_when above, which asks for a decoder-only language model with the
gain removed.
It does carry a qualification worth having. ViT-22B drops the biases from the QKV projections and the LayerNorms for a measured 3% accelerator utilisation gain with no quality loss — and keeps the biases on the MLP dense layers, explicitly departing from PaLM, having “observed improved quality and no speed reduction”.
So bias removal is not uniform even inside one model, and two groups disagree about the MLP biases with neither showing the experiment. Whatever settles this practice will need to be more specific than “remove the learnable parameters”.
Known implementations
- None in this record. The paper’s own LayerNorm-simple and AdaNorm are the only reported uses.