anthology-of-the-sota

LIT-005: Layer Normalization

1 min read

Properties4
StatusActive
Tagsmodel-stability
Cited as “introduced by” bySOTA-006 — Consider alternatives like LayerNorm for transformers, SOTA-025 — Initialize the LayerNorm gain to 1, SOTA-026 — Initialize LayerNorm bias to 0
Cited as “source” bySOTA-025 — Initialize the LayerNorm gain to 1, SOTA-026 — Initialize LayerNorm bias to 0

Graph View

Ba et al. (2016) — ARXIV-1607.06450

Key takeaways

  • Introduces normalization across features instead of batch
  • Works well with recurrent neural networks
  • Stabilizes hidden state dynamics
  • Reduces training time for deep networks

Backlinks