nucleation

THEORY-809: Language models trained by gradient descent on a mixture of random facts and other data reach the capacity-optimal allocation, so the model size and the mixing ratio at which they begin to learn the facts are those of an optimal capacity-bounded learner

2 min read

Properties5
StatusProposed
Tagslearning-theory, information-theory
SourceLIT-984 — Data Mixing Can Induce Phase Transitions in Knowledge Acquisition
ExtendsTHEORY-293 — For an optimal capacity-bounded learner on an orthogonal mixture of equally frequent random facts and a second domain, the facts are learned not at all below a capacity threshold set by the mixing ratio and in full once capacity exceeds it by the facts' total entropy, and Pythia models trained on synthetic biographies mixed into web text show a threshold in model size and in mixing ratio
AnswersQUESTION-176 — Do networks trained by gradient descent on a mixture of random facts and a second domain reach the capacity-optimal allocation, so that their threshold size and mixing ratio are the optimal learner's?

Graph View

Source

Gu, Lyu, Li and Zhang (2025), LIT-984, §§3–5, read in NOTE-796. The optimal learner’s threshold is Theorem 4.3; the Pythia runs are §3 (Figs. 1–4, Table 1).

What is asserted, and by whom

The thesis is that SGD-trained language models behave as the optimal capacity-bounded learner does: they allocate capacity optimally between the facts and the other domain, so their threshold model size scales as (1/(r·p))^(1/(α+1)) and their recall rises within about Htot past it. The authors assert it (NOTE-796, C5, weak). They also extend it, more weakly still, to the frequency at which open model families learn a fact (C6); that extension is not filed here.

What the record holds toward it

  • THEORY-293 (Active): Theorem 4.3 for the optimal learner, and a threshold in model size and in mixing ratio in Pythia models from 14M to 6.9B on synthetic biographies mixed into web text, mostly single-seed.
  • Against it, so far: the measured size exponent, 1.152, is about 10% below the predicted 1.283, with no interval (NOTE-796, Limitations). Below the threshold ratio, much longer training barely helps (C4, moderate), which fits allocation rather than dynamics but was extrapolated from fits over r in [0.2, 0.55].

What would confirm or reject it

Thresholds predicted in advance and matched across seeds and mixing ratios confirm it; a systematic departure from the predicted exponent, or networks below the threshold that learn the facts with enough further training, reject it. QUESTION-176 asks the same thing as a question; this entry is the answer the authors propose.

Backlinks