Source
Ioffe et al. (2015), LIT-002 — ARXIV-1502.03167.
Why before the activation
Batch normalization standardises its input to zero mean and unit variance, then applies a learned scale and shift. Placed between the linear map and the nonlinearity, it acts on the pre-activation distribution — which is what keeps the nonlinearity operating in its useful range rather than saturated, and is the placement LIT-002 introduces and evaluates.
Placed after the activation it standardises a distribution the nonlinearity has already shaped — a ReLU’s output is non-negative, so forcing zero mean undoes what the unit just did.
Condition, and what happened to this practice since
It is a claim about convolutional networks with batch normalization, which is the setting LIT-002 is about, and both halves of that have moved. Transformers use LayerNorm or RMSNorm (SOTA-006), where the placement question is a different one — inside or outside the residual branch (SOTA-032) — and the modern answer there is pre-norm, which is not the analogue of this rule.
The bias term in the preceding linear layer also becomes redundant, since normalisation removes any constant offset before the learned shift restores one. Omitting it costs nothing and is the usual convention.
SOTA-022 states this same rule from a different source and is superseded by this practice.