anthology-of-the-sota

THEORY-108: Kaplan's and Chinchilla's compute-optimal exponents differ because of how small-scale runs were counted, warmed up and tuned, not because of the learning-rate decay

2 min read

Properties4
StatusProposed
Tagstraining-optimization, analysis-and-evaluation
SourceLIT-690 — Resolving Discrepancies in Compute-Optimal Scaling of Language Models, LIT-688 — Reconciling Kaplan and Chinchilla Scaling Laws
ExplainsSOTA-096 — num_tokens ~ 20 * num_params, SOTA-413 — In a scaling-law study, count the output head in parameters and FLOPs, and fit loss against compute with an irreducible-loss offset

Graph View

Source

Porian et al. (2024), LIT-690, primary. Pearce and Song (2024), LIT-688, corroborating on counting and on decay.

The account

At small scale, three things make a compute-optimal sweep favour parameters over tokens:

  1. Uncounted head FLOPs. Excluding the output layer under-counts compute by up to 90% at the smallest sizes, and by less and less as models grow. The under-count is size-dependent, so it tilts the fitted slope.
  2. A warmup longer than a small model’s optimal run. With 1.57B warmup tokens, small models are “forced” to train longer than is optimal to escape it. That pushes their optimal token counts up at small compute and flattens the growth of D*, which steepens N*.
  3. Hyperparameters fixed across sizes. One batch size and learning rate, and β₂ = 0.95 at small batch, suit some sizes worse than others.

Each moves the exponent by about 0.1, and together they account for the gap from about 0.84 to 0.5. The learning-rate decay moves it by 0.03.

What it explains

  • SOTA-096: why 20:1, equal proportion, replaced Kaplan’s parameter-heavy allocation. The earlier exponent was a small-scale measurement artefact, not a different law.
  • SOTA-413: why counting the head is a correction and not a convention. The under-count is size-dependent.

Where it stops

  • The sources disagree on the shares. Pearce and Song attribute “much of” the gap to counting, against the adjusted 0.73. Porian finds counting is about 38% of the gap to the raw 0.88, and explains the difference by the warmup factor Pearce and Song did not model.
  • Order matters in principle. The attribution is sequential. One reordering (tuning first, 0.835 → 0.717) is roughly additive. No factorial design was run.
  • The reproduction is Kaplan-like, not Kaplan’s. Llama-style, context 2048, different data and tokenizer.
  • It does not say decay is irrelevant to loss. Decay matters, increasingly with scale. It barely moves the fitted allocation.
  • The warmup factor has one source. That is the main reason this is Proposed.

Backlinks