Bjorck, Gomes, Selman and Weinberger (2018) — ARXIV-1806.02375
Key takeaways
- The practical claim: BN’s central benefit is that it permits much larger learning rates than an unnormalised network tolerates, and most of the observed improvement in accuracy and training speed follows from that rather than from normalisation as such.
- Without BN, large learning rates at initialisation produce activations and gradients that grow with depth, which is what makes those learning rates divergent. BN bounds them, so the same rate becomes usable.
- Larger learning rates carry their own regularising effect, which is part of why BN is often described as a regulariser.
Not this paper
The internal-covariate-shift challenge, the loss-landscape smoothing argument and the Lipschitzness result belong to Santurkar, Tsipras, Ilyas and Madry — How Does Batch Normalization Help Optimization? — filed separately. This note carried those as its takeaways under this title for as long as the record has existed.
Worth stating how it survived: the title and the arXiv identifier agreed with each other, so nothing mechanical could catch it. Only the author field and the content came from the other paper, and neither is checkable against the other today.
Standing in the anthology
The source for SOTA-020 (use larger learning rates with batch normalization), which is exactly this paper’s claim and is the practice that was correctly sourced all along.