nucleation

LIT-348: An Empirical Model of Large-Batch Training

1 min read

Properties5
StatusActive — read in full 2026-09-29 (NOTE-293); worth reading as the origin of the batch-size/gradient-SNR/critical-batch/time–compute result that row 12 of the owner's map attributes to it. The theory is elementary and local, and the evidence is empirical. It has nothing on information, channels or energy, so it owns row 12's formulas for progress and trade-off and none of its information-theoretic framing.
Version2
Tagslearning-theory
Cited as “paper” byNOTE-293 — An Empirical Model of Large-Batch Training
Cited as “source” byTHEORY-033 — The measured noise scale B_simple tracks the critical batch size within about 10× on six of eight tasks and about 20× on two generative models, so the gradient's noise-to-signal ratio sets the critical batch only as far as the local quadratic model with an optimal step holds, THEORY-735 — Under a local quadratic model of the loss with the greedy optimal step, per-step progress at batch size B is ΔL_max/(1 + B_noise/B), and at fixed B the steps and examples needed satisfy (S/S_min − 1)(E/E_min − 1) = 1, with E_min/S_min the progress-weighted mean of B_noise

Graph View

Sam McCandlish, Jared Kaplan, Dario Amodei, OpenAI Dota Team (2018), arXiv preprint — ARXIV-1812.06162

Standing in the record

Registered on 2026-09-29 at the owner’s request, because the owner’s prior-art novelty map (a private working document on concepts as projectors, the linear representation hypothesis and their neighbours) cites it: row 12 and §4: owns gradient-as-channel / batch = SNR / critical batch that §5 re-derives. It is seeded from metadata and published descriptions and has not been read here.

It is also held in the Anthology of the SOTA as ANTH-LIT-017. Holding it here too follows ADR-013: this record holds it for the owner’s prior-art map, the anthology for ML practice.

Identification notes: Held in anthology (ANTH-LIT-017).

It was filed Deferred, unread. NOTE-293 is the close reading of 2026-09-29, and it placed the work: Active — worth reading as the origin of the batch-size/gradient-SNR/critical-batch/time–compute result that row 12 of the owner’s map attributes to it. The theory is elementary and local, and the evidence is empirical. It has nothing on information, channels or energy, so it owns row 12’s formulas for progress and trade-off and none of its information-theoretic framing.

Backlinks