nucleation

THEORY-735: Under a local quadratic model of the loss with the greedy optimal step, per-step progress at batch size B is ΔL_max/(1 + B_noise/B), and at fixed B the steps and examples needed satisfy (S/S_min − 1)(E/E_min − 1) = 1, with E_min/S_min the progress-weighted mean of B_noise

2 min read

Properties4
StatusActive
Tagslearning-theory, mathematics
SourceLIT-348 — An Empirical Model of Large-Batch Training
Extended byTHEORY-033 — The measured noise scale B_simple tracks the critical batch size within about 10× on six of eight tasks and about 20× on two generative models, so the gradient's noise-to-signal ratio sets the critical batch only as far as the local quadratic model with an optimal step holds

Graph View

Source

McCandlish, Kaplan, Amodei & the OpenAI Dota Team (2018), LIT-348, eqs. 2.2–2.12 and App. D.1, as read in NOTE-293.

What was actually shown

A derivation. Write G for the gradient, H for the Hessian and Σ for the per-example gradient covariance.

  1. Let V be the batch gradient, with E V = G and Cov V = Σ/B. To second order, L(θ − εV) = L(θ) − ε GᵀV + ½ ε² VᵀHV.
  2. Take expectations. E[GᵀV] = |G|², and E[VᵀHV] = GᵀHG + tr(HΣ)/B. The expected decrease is ΔL(ε) = ε|G|² − ½ ε² (GᵀHG + tr(HΣ)/B).
  3. The maximising step is ε* = |G|²/(GᵀHG + tr(HΣ)/B), and ΔL* = ΔL_max/(1 + B_noise/B), with ΔL_max = ½|G|⁴/GᵀHG and B_noise = tr(HΣ)/GᵀHG.
  4. At fixed B and constant B_noise, a given decrease takes S = S_min(1 + B_noise/B) steps and E = SB = S_min(B + B_noise) examples, so E_min = S_min·B_noise. Then S/S_min − 1 = B_noise/B and E/E_min − 1 = B/B_noise, whose product is 1.
  5. When B_noise varies over the run, apply step 4 to each short stretch and sum. With ⟨B_noise⟩ the mean of B_noise weighted by the decrease each stretch contributes, S/S_min − 1 = ⟨B_noise⟩/B and E/E_min − 1 = B/⟨B_noise⟩. The hyperbola is exact (NOTE-293).
  6. E|V − G|²/|G|² = tr(Σ)/(B|G|²) = B_simple/B. So B_simple is the batch at which the gradient’s expected squared error equals its squared norm, and if H ∝ I then B_noise = B_simple.

What could have come out otherwise: nothing in the algebra. Its content is that, under these assumptions, examples substitute for steps almost one-for-one below B_noise and are mostly wasted above it.

What this does not say

  • That the assumptions hold in training. The authors call them “many unfounded assumptions” (§2.2). The fit across eight tasks is evidence for the form, and it belongs to THEORY-033.
  • That B_crit is predicted. S_min and E_min are fitted per goal, so B_crit is a fitted parameter. How well B_simple tracks it is THEORY-033’s measured claim.
  • That the knee is a channel-capacity effect. Progress per step saturates at ΔL_max. A Shannon–Hartley log(1 + SNR) does not saturate.
  • Anything about generalisation. The derivation concerns training loss only.

Backlinks