LIT-040: Scaling Laws for Autoregressive Generative Modeling

Henighan et al. (2020) — ARXIV-2010.14701

Key takeaways

• Extends the Kaplan-era scaling laws to four more domains — generative image, video, multimodal image↔text, and mathematical problem solving — and finds the same power-law-plus-constant form, the constant being the data’s irreducible entropy
• N_opt(C) ∝ C^β with β ≈ 0.7, nearly the same in every domain. The cross-domain universality of the exponent is the headline
• That exponent implies D ∝ N^0.4 — sub-linear data scaling, and the paper says so: “strongly suggests sub-linear scaling of dataset size with model size”, conditioned on one epoch. Chinchilla later measured D ∝ N. Do not take this number from here
• Section 6 names the inconsistency that undid it: the L(C) and L(D) trends, extrapolated, cross — and cannot, since every learning curve is bounded below by L(D). The authors also warn their projections are extremely sensitive, moving visibly under a ±5% change in the exponent
• Model size buys no strong generalisation. On mathematical extrapolation and on off-distribution image loss, performance depends predominantly on in-distribution loss and is otherwise independent of model size

Standing in the anthology

Read — NOTE-072. The allocation conclusion is superseded and the record already sources the successor, so nothing is filed from this reading.

Its value is as a reading instruction for a superseded paper. This document sits in the anthology as Active while containing a prominently derived D ∝ N^0.4 that the record elsewhere says is wrong. The takeaways it carried — “universal scaling behaviours”, “compute-optimal model sizing”, “resource allocation guidance” — advertised exactly that conclusion without stating it, so a reader could neither use the paper nor know to distrust it. They now name the number and say it was overturned. Not retiring a superseded paper, but stopping it being quietly wrong.

Third point on an arc the record now holds end to end: LIT-028 and this (sub-linear, 2020) → Chinchilla (1:1, 2022) → LIT-099 (1:1 confirmed independently at larger compute, 2023).