Source
Kaplan et al. (2020), LIT-028 — ARXIV-2001.08361.
Filed against Chinchilla (LIT-068) until version 2. Chinchilla fits model size and token count against compute and does not derive a batch-size exponent; Kaplan does, and this is his.
McCandlish et al. (2018), LIT-017 — ARXIV-1812.06162. Where the
critical batch size Kaplan fits an exponent for is defined, as E_min/S_min
off the steps/examples tradeoff, and given a measurement procedure; it also
finds that model size affects the right batch size only through the loss
reached, which is the property Figure 10 relies on.
Where the exponent comes from
Kaplan’s equation 1.7 gives the compute-optimal allocation as a set of power laws sharing one numerator:
N ∝ C^(α_C/α_N), B ∝ C^(α_C/α_B), S ∝ C^(α_C/α_S)
with α_C ≈ 0.050 and α_B ≈ 0.21, so the batch exponent is 0.24 — which is what the title rounds to 1/4. Underneath it is the critical batch size, the point past which extra parallelism stops buying speed (equation 1.4):
B_crit(L) = B* / L^(1/α_B), B* ≈ 2×10⁸ tokens, α_B ≈ 0.21
The property that makes this usable is in Figure 10: B_crit depends on the loss, not directly on the model size, and approximately doubles for every 13% decrease in loss. Two models of different sizes at the same loss want the same batch size.
The recommendation starts in LIT-028. McCandlish et al. defined the critical
batch size, but the step from it to a compute budget is Kaplan’s: his fits of
B_crit against loss and of loss against compute, on WebText2, combine in
equation 1.7 into B ∝ C^0.24. Reading the paper in full confirmed the
exponent as 0.050/0.21.
Conditions
The exponent is small, which matters more than its exact value: a thousand-fold increase in compute asks for roughly a five-fold increase in batch size. Most of a growing budget goes to model size, not to batch.
B_crit is a speed-against-efficiency boundary rather than a quality one.
Training below it wastes wall-clock; above it wastes compute for the same loss.
Which side to err on is a scheduling decision, not a modelling one.
And the fit is Kaplan’s regime: dense decoder-only transformers on WebText2, 2020 hardware, a fixed schedule. Kaplan’s model-size exponent was later shown wrong by Chinchilla (SOTA-096) — the same paper, the same fitting method, a different exponent that did not survive. That the batch exponent has not been re-derived by anyone since is a reason to hold it loosely, not a reason to trust it.
Known implementations
- chinchilla
- llama2