nucleation

LIT-984: Data Mixing Can Induce Phase Transitions in Knowledge Acquisition

2 min read

Properties5
StatusActive — read 2026-10-14 (NOTE-796) in full; worth reading for QUESTION-039 as a threshold of a different kind from the record's others: whether a set of facts is learned at all is set by competition for capacity with the rest of the data, through the ratio of their marginal loss reductions, not by the facts' own statistics. Read it knowing that the theorem is about an idealised optimal learner, says nothing about when in training the facts arrive, and predicts an onset over a width of Htot rather than a jump; that trained networks follow it rests on the shape of the curves and one exponent. Its practical advice belongs to the anthology's reading, ANTH-LIT-451. Filed as THEORY-293.
Version2
Tagslearning-theory
Cited as “paper” byNOTE-796 — Gu et al. — Data Mixing Can Induce Phase Transitions in Knowledge Acquisition
Cited as “source” byTHEORY-293 — For an optimal capacity-bounded learner on an orthogonal mixture of equally frequent random facts and a second domain, the facts are learned not at all below a capacity threshold set by the mixing ratio and in full once capacity exceeds it by the facts' total entropy, and Pythia models trained on synthetic biographies mixed into web text show a threshold in model size and in mixing ratio, THEORY-809 — Language models trained by gradient descent on a mixture of random facts and other data reach the capacity-optimal allocation, so the model size and the mixing ratio at which they begin to learn the facts are those of an optimal capacity-bounded learner

Graph View

Xinran Gu, Kaifeng Lyu, Jiazheng Li, Jingzhao Zhang (2025), NeurIPS 2025; arXiv:2505.18091.

Key takeaways

From the reading, NOTE-796.

  • No transition without mixing. On equally frequent facts alone, the best loss under capacity M is C + p·max{Htot − M, 0}: linear in M (Theorem 4.2).
  • With mixing, an onset. On an orthogonal mixture with ratio r, an optimal capacity-bounded learner learns none of the facts while the web domain’s marginal loss reduction −F′(M) exceeds r·p/(1−r), and all of them once M passes that point by Htot (Theorem 4.3).
  • Power-law case. M_thres ~ (1/(r·p))^(1/(α+1)), r_thres ~ 1/(p·M^(α+1)), and a fact’s threshold frequency f_thres ~ M
  • In trained models. 70M Pythia on SynBio-320k + FineWeb-Edu: 0.4% recall at r ≤ 0.2, 58.1% at r = 0.45 (three seeds). Thresholds in size from 14M to 410M, and in r for 410M, 2.8B and 6.9B models.
  • Longer training barely helps. At r = 0.2, 4–16 times more tokens leave recall near zero; steps to 40% accuracy outgrow an exponential fit in 1/r.
  • One exponent checked. Threshold frequency against size has slope −1.152; the theory predicts −1.283 from the fitted web-loss exponent.
  • What it does not show. That SGD reaches the optimal allocation, or when in training acquisition happens: the theory is static.

Standing in the record

Filed 2026-10-12 from the owner’s papers-feed for QUESTION-039, and read in full on 2026-10-14 in NOTE-796. It gives the question a threshold of a different kind from those it holds: competition for capacity between domains, set by a ratio of marginal values, where THEORY-086, THEORY-187 and THEORY-229 time acquisition within one task. The reading is filed as THEORY-293, which answers the question in part. It is also held in the Anthology of the SOTA, as ANTH-LIT-451, under ADR-013: the anthology reads it as a result about data mixtures and training recipes, including the subsampling and rephrasing strategies of its §6, and this record for QUESTION-039.

Backlinks