Xu et al. (2019) — ARXIV-1911.07013
Key takeaways
- The received explanation for LayerNorm is forward normalization — controlling the distribution of layer inputs. The paper’s central claim is that this is not where the benefit comes from
- What matters instead is the derivatives of the mean and variance, which re-center and re-scale the backward gradients. The effect is in the backward pass, not the forward one
- The bias and gain increase the risk of over-fitting and do not work in most cases — the paper’s own “beyond our expectation” result
- LayerNorm-simple, with the bias and gain removed entirely, outperforms LayerNorm on four of the datasets tried, and reaches state of the art on En-Vi machine translation
- AdaNorm is the proposal: replace the bias and gain with a transformation function that adapts to the input. Better than LayerNorm on seven of eight datasets
Standing in the anthology
The source of SOTA-191.
Read in full for #114; the reading is NOTE-001. Three practices had been drawn from the replaced takeaways, and all three are about setting the bias and gain — a 0.97–1.0 initialisation range, a zero bias, a reduced learning rate for both. The paper argues those two parameters should be removed. It contains the string “0.97” zero times and “smaller learning rate” zero times.
This is the second note found this way, after LIT-052, and it fails
differently in a way worth naming. LIT-052’s bullets came from some
other paper entirely. These bullets are in the right neighbourhood —
LayerNorm, initialisation, gradients, optimization — and are wrong about
the direction of the finding. That is harder to catch by skimming, because
nothing looks out of place until the paper is open.