Source
Santurkar et al. (2018), LIT-015 — ARXIV-1806.02375.
This is LIT-015’s actual claim
Bjorck et al.’s argument is that permitting a much larger learning rate is batch normalization’s central benefit, and that most of the accuracy and speed improvement follows from it rather than from normalisation as such. Without BN, large rates at initialisation drive activations and gradients that grow with depth, which is what makes them divergent; BN bounds them, so the same rate becomes usable.
The practice is therefore not “BN lets you be sloppy about the learning rate” but “BN moves the stable range upward, so a rate tuned for an unnormalised network is leaving most of the benefit unused.”
Condition, and a correction to the record around it
Larger rates also carry their own regularising effect, which is part of why BN is described as a regulariser — the regularisation is downstream of the rate, not a separate property.
This practice was correctly sourced all along, which is the reason worth recording: LIT-015 carried the wrong author and the wrong takeaways until this contribution, but the paper the identifier names really does argue this, and SOTA-020 really does follow from it. The two practices beside it in the
cluster did not fare as well — SOTA-021 turns on a mechanism belonging to a different paper, and SOTA-022 duplicates SOTA-004.