Shen et al. (2020) — ARXIV-2003.07845
Key takeaways
- Diagnoses why batch normalization underperforms layer normalization in NLP, by instrumenting training: the per-batch mean and variance fluctuate far more than in vision, so the running estimates used at inference do not match what training saw
- Centring is the fragile half. The variation is in the batch mean, and
the gradient contribution routed through
σ_Bcarries many more outliers than the one throughψ_B - Power Normalization drops the zero-mean step entirely and normalizes by a
running quadratic mean
ψ = √(E[x²])rather than a per-batch variance - +0.4/+0.6 BLEU on IWSLT14/WMT14 and −5.6/−3.0 PPL on PTB/WikiText-103 against LN, with no change of hyperparameters — and +1.5/+2.0 BLEU, −7.7/−3.4 PPL against BN
- Reframes the normalization question from which axis to which statistic is stable enough to normalize by
Standing in the anthology
Read — NOTE-056. PowerNorm did not displace LayerNorm or RMSNorm, and a 0.4-BLEU margin at 2020 scale is not a reason to revisit that.
The value is the connection the reading surfaced. This is the measurement
behind a claim LIT-023 states as a hypothesis: RMSNorm also drops centring
and normalizes by a root-mean-square, but along the feature axis and without
measuring why re-centring is dispensable. PowerNorm measures it on the batch
axis, forward and backward. LIT-025 then acts on the same intuition from a
third direction by removing LayerNorm’s learned gain and bias. Three papers
converging on LayerNorm does more than it needs to, and the record did not
previously connect them.
Nothing is sourced to this paper and the reading files no practice.