nucleation

LIT-371: How do language models learn facts? Dynamics, curricula and hallucinations

1 min read

Properties5
StatusActive — read in full 2026-09-30 (NOTE-319); worth reading as a clean, intervention-backed decomposition of a loss plateau into circuit formation followed by knowledge storage, in a setting where knowledge is countable; its curriculum and fine-tuning results are single-seed and synthetic, and it measures no information quantity, so it bears on phases of training but not on the IB account.
Version2
Tagslearning-theory, representation-learning
Cited as “paper” byNOTE-319 — How do language models learn facts? Dynamics, curricula and hallucinations
Cited as “source” byTHEORY-039 — The later phases reported in neural-network training differ in kind: after the transition a network may simplify its weights, acquire item-specific information, lose generalisation, confine its tangent kernel or saturate its units, and none of these is a measured compression of I(X;T)

Graph View

Zucchet et al. (2025), Conference on Language Modeling (COLM 2025), per the arXiv v2 comment “Accepted at the 2nd Conference on Language Modeling (2025)”; arXiv preprint 2503.21676 — arXiv:2503.21676

Standing in the record

Filed on 2026-09-30 at the owner’s request, as evidence on phases of training: whether they corroborate the information-bottleneck account of a fitting then a compression phase (LIT-324), and what phase structure they show. published: is the first appearance (ADR-002).

Also held in the Anthology of the SOTA as ANTH-LIT-450 (ADR-013): the anthology reads it for ML practice, this record for what it shows about phases of training and the information-bottleneck account.

It was filed Deferred, unread. NOTE-319 is the close reading of 2026-09-30, and it placed the work: Active — worth reading as a clean, intervention-backed decomposition of a loss plateau into circuit formation followed by knowledge storage, in a setting where knowledge is countable; its curriculum and fine-tuning results are single-seed and synthetic, and it measures no information quantity, so it bears on phases of training but not on the IB account.

Backlinks