anthology-of-the-sota

LIT-017: An Empirical Model of Large-Batch Training

2 min read

Properties6
StatusActive
Version2
Tagstraining-optimization
Cited as “introduced by” bySOTA-092 — smaller batch sizes are more sample efficient (i.e., better loss as a function of tokens seen) earlier in training, SOTA-093 — larger batch sizes are beneficial later in training due to better gradient estimates, SOTA-198 — Measure the gradient noise scale instead of sweeping batch size, and expect it to grow during the run
Cited as “paper” byNOTE-062 — An Empirical Model of Large-Batch Training
Cited as “source” bySOTA-061 — Use largest batch that maintains >80% sample efficiency, SOTA-062 — Scale batch size with model size but sub-linearly, SOTA-092 — smaller batch sizes are more sample efficient (i.e., better loss as a function of tokens seen) earlier in training, SOTA-093 — larger batch sizes are beneficial later in training due to better gradient estimates, SOTA-097 — Optimal batch size scales approximately with compute budget - B ∝ C^(1/4), SOTA-198 — Measure the gradient noise scale instead of sweeping batch size, and expect it to grow during the run, THEORY-013 — The SGD noise scale g = eps*N/B sets generalization, so the optimal batch size grows with the training set

Graph View

McCandlish et al. (2018) — ARXIV-1812.06162

Key takeaways

  • Defines the gradient noise scale B_noise = tr(HΣ)/(GᵀHG), and the Hessian-free estimator actually used, B_simple = tr(Σ)/|G|² — the summed per-component gradient variance over the squared gradient norm
  • Per-step progress depends on batch size only through 1/(1 + B_noise/B). Below the noise scale, doubling the batch nearly doubles progress; above it, almost nothing. At B = B_noise, training speed is 50% of maximum — the noise scale is the knee, not the ceiling
  • The steps/examples tradeoff is a hyperbola, (S/S_min − 1)(E/E_min − 1) = 1, and critical batch size is defined off that fit: B_crit = E_min/S_min. The record inherits B_crit from here
  • Verified on eight tasks across supervised learning, RL and generative modelling — MNIST through ImageNet, Billion Word, Atari, Dota, a VAE. The noise scale is independent of dataset size, which is what lets it transfer between domains
  • The noise scale rises as the loss falls, so the right batch size grows during a run; and model size affects it only through the loss the model reaches, not through parameter count

Standing in the anthology

Read — NOTE-062. Order-of-magnitude prediction, which the paper says plainly: it tells you whether 10³ or 10⁵ is the ballpark, not whether 8K beats 16K.

Until the correction pass after #395, nothing in the record was sourced to this paper. SOTA-092 and SOTA-093 now are: small batches early and large batches late is this paper’s Fig. 5 caption, which PaLM later repeated with a citation. Elsewhere the anthology still carries the consequences of the noise scale without carrying its origin — B_crit appears in the scaling-law practices as an inherited quantity, and this is where it is defined and given a measurement procedure. A practice recommending that the noise scale be measured rather than the batch size swept is a real candidate and is deliberately not filed from a note-repair pass; it needs a session that can check what the registry already says about batch-size selection.

Backlinks