LIT-023: Root Mean Square Layer Normalization

Zhang et al. (2019) — ARXIV-1910.07467

Key takeaways

• LayerNorm is credited with handling re-centering and re-scaling of both inputs and the weight matrix, and its computational overhead “significantly slows the underlying network, e.g. RNN in particular”
• The central claim is a hypothesis, in the paper’s own word: “we hypothesize that re-centering invariance in LayerNorm is dispensable”
• RMSNorm normalizes by root mean square alone, keeping re-scaling invariance and an implicit learning-rate adaptation — the effective step on a weight vector scales inversely with its norm
• 7–64% less time per step, the range depending on how much of the step the normalization was, at comparable quality
• pRMSNorm estimates the RMS from p% of the summed inputs, cheaper again and with its own uncharacterised error

Standing in the anthology

The source of SOTA-182.

Read in full for #114; the reading is NOTE-022. The practice is confirmed.

One thing the reading adds: the paper frames its claim as a hypothesis it tests, not a result it proves. What re-centering does is never measured directly. SOTA-182 is right that a centered LayerNorm is now the choice needing justification — but that is the field’s verdict, arrived at after this paper proposed the question and was not contradicted.

Its near-twin is LIT-025, same year, which also concludes part of LayerNorm is unnecessary and picks the other part.

Rerun by Narang et al. (LIT-711), in a 223M T5 encoder–decoder with hyperparameters held fixed. The loss gain reproduces: 2.167 ± 0.008 against LayerNorm’s 2.182 ± 0.005 early, and 1.821 against 1.838 final. The speed gain is 3.68 against 3.50 steps per second, about 5% rather than the 7–64% here. With learned absolute positions in place of relative biases, RMSNorm still lowers loss but loses to the baseline on SuperGLUE and WebQuestions.