Source
Hoffmann et al. (2022), LIT-068 — ARXIV-2203.15556.
Porian et al. (2024), LIT-690 — ARXIV-2406.19146. An independent, direct measurement of the equal-proportion allocation: once Kaplan’s discrepancies are removed it lands on an exponent of 0.497 from 5M to 901M on two datasets, within 15% of Chinchilla’s model size at Chinchilla’s compute. It evidences the exponent, not the 20: its optimal ratio is 14–16 on RefinedWeb and 11–22 on OpenWebText2.
The finding
Given a fixed compute budget, previous practice (LIT-028) allocated it toward model size and stopped well short of convergence. Fitting the loss surface over 400 runs from 70M to 16B parameters and 5B to 500B tokens, Hoffmann et al. found instead that model size and training tokens should scale in equal proportion: double the compute, double both. The ratio that falls out is about 20 tokens per parameter.
Both the recommendation and the ratio come from LIT-068, though the paper
never writes the number 20. Its three fitting methods give exponents of 0.50,
0.49 and 0.46 for N_opt ∝ C^a, against Kaplan et al.’s 0.73, and its Table 3
projects compute-optimal training at 20.2B tokens for a 1B-parameter model,
205.1B for 10B and 1.5T for 67B: about 20 tokens per parameter, read off the
table.
The demonstration was the point. Chinchilla at 70B, trained on 1.4T tokens, outperformed Gopher at 280B trained on 300B tokens — a model four times smaller, on the same compute, beating it across the board. Gopher, GPT-3, Jurassic-1 and Megatron-Turing NLG were all substantially undertrained.
What the ratio is and is not
It is the compute-optimal allocation: the split that minimises loss for a fixed training budget. It is not the allocation that minimises the cost of using the model, and the record’s practice has diverged from it for that reason. Inference cost scales with parameters and not with training tokens, so a model that will serve many requests is worth training past 20:1 — every frontier release in this record trains far beyond it, and none of them is violating this finding. They are optimising a different objective.
The ratio also assumes the data is there to spend. Past the point where a corpus must be repeated, the trade changes
(SOTA-124’s question), and the scaling law was fit in a regime where fresh tokens were assumed.
The batch-size claim that used to sit beside this one under the same citation (SOTA-097) is Kaplan’s, not Chinchilla’s, and now says so.
Known implementations
- chinchilla
- llama2