Kaplan et al. (2020) — ARXIV-2001.08361
Key takeaways
- Loss is a power law in model size, dataset size and compute, over more
than seven orders of magnitude:
α_N ≈ 0.076,α_D ≈ 0.095,α_C ≈ 0.050 - Shape barely matters — width, depth, heads have “minimal effects within
a wide range” at fixed non-embedding parameter count. True of upstream
loss;
LIT-052contradicts it downstream - Larger models are significantly more sample-efficient, so
compute-optimal training means very large models stopped well short of
convergence, with data growing only as
D ∼ C^0.27 - Compute-optimal allocation (Eq. 1.7):
N ∝ C^(α_C/α_N),B ∝ C^(α_C/α_B),S ∝ C^(α_C/α_S)— the batch exponent being0.050/0.21 = 0.24 - Critical batch size
B_crit(L) = B*/L^(1/α_B),B* ≈ 2×10⁸tokens, depends on the loss, not directly on model size, and roughly doubles per 13% decrease in loss - On learning rates, and this is where the record went wrong: the shape
of the schedule is “mostly irrelevant” given warmup and a final decay —
measured on a 3M-parameter model — but “larger models require a
smaller learning rate to prevent divergence”, implemented as
LR(N) ≈ 0.003239 − 0.0001395·log(N), which “breaks down forN > 10¹⁰” - The schedule was held fixed across runs of different length, which Chinchilla later proposed as the source of the wrong allocation exponent. Its two halves behave differently. Porian et al. (LIT-690) find the decay half is not a main cause: matching it per run moves the exponent 0.03. The warmup half, too long for small models, is one of three causes. The other two are the uncounted head FLOPs and batch, learning rate and β₂ not tuned per size (THEORY-108). LIT-688 finds the counting effect independently at tiny scale. The allocation exponent was wrong; the reason usually given for it is only a quarter of the one that survived.
Standing in the anthology
The source of SOTA-040 and, until 2026-09-20, of SOTA-097.
The batch-size exponent has been corrected. Equation 1.7 gives
B ∝ C^0.24, and two groups measuring off the Chinchilla line —
LIT-443 and LIT-445 — find that optimal and critical batch
size scale in the token budget D, not in compute. The fit is not an
arithmetic error: along that line C = 6ND moves with D, so a power law in
either describes the data, and equation 1.7 measured a projection. The
allocation exponent has since been corrected too (LIT-688). The
mechanism is a related one, not the schedule Chinchilla named: a fit read the
slope of one region of a curve and reported it as the whole curve’s. Neither
correction retires the paper, which stays Active. The method
generalized past its own fits in a way its authors could not have checked
with runs only on the compute-optimal line.
Read in full for #114; the reading is NOTE-017.
SOTA-041 — “lr tuning less important for larger models” — is Rejected. It
states the reverse of Appendix D.6, and a paper that computes the learning
rate from the parameter count is not one claiming the rate matters less as
that count grows.
Chen et al. (LIT-685) fit this paper’s functional form alongside Chinchilla’s and their own modified law, over runs from 20M to 7B parameters. Their own Table 4 gives the Kaplan form the lowest fit error at 50M, 100M and 300M, which contradicts their claim that the new law wins at every size. The extra terms in their law depend only on tokens per parameter, so the better in-sample fit it gets elsewhere is what two added constants would be expected to buy.