Source
Ioffe and Szegedy (2015), LIT-002 — the paper that introduced the technique, in which this account is not an aside but the title.
The account, as it was stated
Each layer’s input distribution moves while the layers beneath it train. The paper names this internal covariate shift and argues it is a first-order obstacle: a layer is forever re-adapting to a moving input, small learning rates are what keep the movement survivable, and saturating nonlinearities make it worse. Normalising each layer’s inputs to a fixed mean and variance holds the distributions still, so the re-adaptation stops being the bottleneck and larger rates become usable.
It is a good explanation. It predicts the right things — faster convergence, higher learning rates, less sensitivity to initialization, all of which LIT-002 demonstrates — and it explains them by a mechanism that is easy to picture, which is most of why it was repeated in every subsequent treatment for three years.
Why this is rejected
Santurkar et al. (THEORY-003) tested the mechanism rather than the predictions, by putting the covariate shift back: noise injected after each BN layer, non-zero-mean and time-varying, restoring exactly the instability BN was supposed to be removing. The network trained anyway. Their conclusion is flat — “distributional stability of layer inputs has little to do with the success of BatchNorm.”
What is rejected is the account, and only the account. Every effect LIT-002
reported is real and reproduces, the technique is standard, and the practices
drawn from it are Active. This document exists so that the record can hold
those two facts at once, which it could not while an explanation had nowhere
to live but the practice that rests on it (DP-003).
What this does not say
It does not say internal covariate shift is not a real phenomenon — layer input distributions do move during training, and nothing here disputes that. The rejected claim is causal: that the movement is what limits training, and that arresting it is what BN’s benefit consists of.
It also does not say the authors were careless. The explanation was the natural reading of the evidence they had, and what unseated it was an experiment nobody had thought to run for three years. The instructive part is how long a plausible mechanism survives when every prediction it makes keeps coming true for a different reason.