nucleation

THEORY-784: Language models trained on limited data show a test loss falling as P^(−γ/(2β)) because P training tokens resolve token correlations only out to a horizon growing as P^(1/(2β)), so that more data mainly lengthens the context a model can use, when the model learns within that horizon faster than it grows, as Cagnetta, Raventós, Ganguli and Wyart propose

2 min read

Properties5
StatusProposed
Tagslearning-theory, linguistics, information-theory
SourceLIT-888 — Deriving Neural Scaling Laws from the statistics of natural language, LIT-883 — Towards a theory of how the structure of language is acquired by deep neural networks
ExtendsTHEORY-204 — On TinyStories and WikiText-103, the data-limited exponent of transformer test loss, fitted to the lower envelope of learning curves across context lengths at up to about 10^8 training tokens, overlaps γ/(2β), where β is the decay with lag of the top singular value of the token–token covariance and γ the decay of next-token conditional entropy with context length; under assumed per-lag scaling forms, a horizon n*(P) ~ P^(1/(2β)) gives the exponent min(δ, γ/(2β)), δ being the decay rate of the within-horizon excess losses
AnswersQUESTION-233 — Do language models show the data-limited exponent γ/(2β) because data limits the usable horizon: does the exponent track γ/(2β) when one corpus statistic is moved and the other held, and does the match survive at horizons of thousands of tokens with γ estimated outside the model class?

Graph View

Source

Cagnetta, Raventós, Ganguli and Wyart (2026), LIT-888, §§3–6, as read in NOTE-689, with the horizon argument from Cagnetta and Wyart (LIT-883), whose account is THEORY-201.

What is asserted

That data limits a language model mainly by limiting how far back it can use context. With P training tokens, the correlation between tokens n apart can be told from sampling noise only while its top singular value, ~ n^(−β), exceeds ~ P^(−1/2), so the usable horizon grows as P^(1/(2β)). The loss is then the conditional entropy at that horizon, which falls as n^(−γ), and the data-limited exponent is γ/(2β), when the excess losses inside the horizon fall faster. The paper offers this as the mechanism of the measured scaling law, not only as a formula that fits it.

What the Active account establishes toward it

THEORY-204 holds the derivation under the paper’s assumed per-lag scaling forms, re-derived, and the measured match: on two corpora, fits of the lower envelope of learning curves overlap γ/(2β). That a formula built on the horizon fits is evidence for the horizon, but γ is estimated from the trained models’ own n-gram losses, so the match is partly internal.

Also proposed by the source, not filed here

The paper’s §6 conjectures that kernel methods and shallow networks fall outside this “universality class”, with worse exponents, by analogy with the Random Hierarchy Model (NOTE-689, C7, weak). It is a separate thesis, argued by analogy and not tested there.

What would confirm or reject it

The promote_when above, which is QUESTION-233’s test: move one corpus statistic with the other held and see whether the exponent follows, or reach horizons of thousands of tokens with γ estimated outside the model class.

Provenance

Filed on 2026-10-20 by the narrowing-loss sweep (ADR-047 v4, batch 068), from the “because” clause THEORY-204’s title carried before it was narrowed on 2026-10-20.

Backlinks