anthology-of-the-sota

LIT-002: Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift

1 min read

Properties4
StatusActive
Tagsmodel-stability
Cited as “introduced by” bySOTA-004 — Place BatchNorm after linear/conv layers but before activation functions, SOTA-005 — Use running statistics for inference
Cited as “source” bySOTA-004 — Place BatchNorm after linear/conv layers but before activation functions, SOTA-005 — Use running statistics for inference, SOTA-006 — Consider alternatives like LayerNorm for transformers, THEORY-001 — Batch normalization works by reducing internal covariate shift, THEORY-003 — Batch normalization helps by smoothing the optimization landscape, not by reducing internal covariate shift

Graph View

Ioffe et al. (2015) — ARXIV-1502.03167

Key takeaways

  • Introduces normalization of layer inputs during training
  • Enables higher learning rates and faster convergence
  • Reduces dependence on careful initialization
  • Acts as regularization

Historical impact

  • Fundamental technique enabling deep network training

Backlinks