Source
Ioffe et al. (2015), LIT-002 — ARXIV-1502.03167.
Why the alternative wins for sequence models
Batch normalization normalises each feature across the batch, which makes an example’s representation depend on the other examples in its batch and requires running statistics at inference (SOTA-005). Both are awkward for sequence models and one is close to fatal: sequences vary in length, so the set of examples contributing to a position’s statistics varies with the batch’s padding, and at inference with batch size one there is nothing to normalise across.
Both properties are stated in LIT-002 itself, which is what this practice can take from it: Ioffe and Szegedy note that BN “does not independently process the activation in each training example” but depends on “the other examples in the mini-batch”, and they need population statistics to make inference deterministic. The paper was about convolutional image classifiers; it lists recurrent networks only as future work, and it does not propose LayerNorm, which came later.
LayerNorm normalises across the feature dimension of a single example. No cross-example dependence, no running statistics, no train/eval divergence, and nothing that changes when the batch does.
The recommendation to prefer it for sequence models is LIT-005’s. Ba, Kiros and Hinton introduced LayerNorm as a transposition of batch normalisation that computes the statistics from one training case, so it “performs exactly the same computation at training and test times” and works at batch size one. They name the sequence problem directly: batch normalisation applied to an RNN needs separate statistics per time step, which breaks when a test sequence is longer than any training sequence. On an attentive reader trained on the CNN question-answering corpus, LayerNorm inside the LSTM trained faster and reached a better validation result than recurrent batch normalisation. That was 2016 and about recurrent networks; the transformer (LIT-008) adopted LayerNorm a year later without arguing the choice, and no source here argues it for transformers specifically.
The practice this one sets itself against is SOTA-004, BatchNorm placed after the linear map and before the activation in convolutional networks. The comparison is argued rather than measured in the record: no source here trains the same transformer under both. And the two do not trade like for like. Moving to LayerNorm does not carry SOTA-004’s placement rule along; the placement question becomes whether normalisation sits inside or outside the residual branch.
Where this has gone since
Further than the title says, which is worth recording. RMSNorm drops the mean-centring and the bias and keeps only the scaling by root-mean-square, on the finding that the centring contributes little — cheaper, and now the common choice in large language models.
The placement question also moved: SOTA-032’s pre-norm arrangement, rather than the post-norm original, is what makes deep transformers trainable without a warmup schedule tuned to depth (SOTA-100).
So “consider alternatives like LayerNorm for transformers” understates a settled outcome. Transformers do not use BatchNorm; the live question in this area is which normalisation and where, not whether to consider one.