nucleation

QUESTION-233: Do language models show the data-limited exponent γ/(2β) because data limits the usable horizon: does the exponent track γ/(2β) when one corpus statistic is moved and the other held, and does the match survive at horizons of thousands of tokens with γ estimated outside the model class?

1 min read

Properties4
StatusOpen
Tagslearning-theory, linguistics, information-theory
Linelearned-structure
Answered byTHEORY-784 — Language models trained on limited data show a test loss falling as P^(−γ/(2β)) because P training tokens resolve token correlations only out to a horizon growing as P^(1/(2β)), so that more data mainly lengthens the context a model can use, when the model learns within that horizon faster than it grows, as Cagnetta, Raventós, Ganguli and Wyart propose

Graph View

Why it is a question

THEORY-204 now states two things. Under assumed scaling forms for each lag’s learning curve, a horizon n*(P) ~ P^(1/(2β)) gives a data-limited loss exponent of min(δ, γ/(2β)). And on two corpora, at up to about 10^8 tokens and horizons of tens of tokens, the exponent fitted to transformer learning curves overlaps γ/(2β). Cagnetta et al. (LIT-888) offer the horizon as the reason. The match does not show that it is the reason: another route could give the same number at that scale.

An answer is one of the following:

  • An intervention. Text whose correlations beyond some lag are weakened while its short-range statistics are kept, or a synthetic language with tunable β and γ, with the measured exponent tracking γ/(2β) as one is moved and the other held. The exponent should depart from γ/(2β) when the within-horizon excess loss is made to decay slowly.
  • Scale. The same match where the usable horizon reaches thousands of tokens, with γ estimated without a neural model of the same class.

More collapses at the scale already reported, with γ read off the trained models’ own n-gram losses, cannot settle it.

Backlinks