nucleation

LIT-369: Grokking and Generalization Collapse: Insights from HTSR theory

1 min read

Properties5
StatusProposed — read in full 2026-09-30 (NOTE-317); worth knowing as the report of a late generalisation collapse after grokking at WD = 0, but it is one architecture on one 1k-sample MNIST subset, with the seed count unstated, and its central diagnostic (α < 2 ⇒ collapse) is contradicted by its own WD = 0.01 control. What would settle it: several seeds and architectures, with α and traps reported for runs that reach α < 2 without collapsing, and a comparison against the ℓ² norm under the Omnigrok LU account, which the paper does not cite.
Version2
Tagslearning-theory, anthology-candidate
Cited as “paper” byNOTE-317 — Grokking and Generalization Collapse: Insights from HTSR theory
Cited as “source” byTHEORY-039 — The later phases reported in neural-network training differ in kind: after the transition a network may simplify its weights, acquire item-specific information, lose generalisation, confine its tangent kernel or saturate its units, and none of these is a measured compression of I(X;T)

Graph View

Prakash & Martin (2025), arXiv preprint (v1 only; the PDF is marked “Preprint. Under review.”; no journal reference on the arXiv record as of 2026-09-30) — arXiv:2506.04434

Standing in the record

Filed on 2026-09-30 at the owner’s request, as evidence on phases of training: whether they corroborate the information-bottleneck account of a fitting then a compression phase (LIT-324), and what phase structure they show. published: is the first appearance (ADR-002).

It was filed Deferred, unread. NOTE-317 is the close reading of 2026-09-30, and it placed the work: Proposed — worth knowing as the report of a late generalisation collapse after grokking at WD = 0, but it is one architecture on one 1k-sample MNIST subset, with the seed count unstated, and its central diagnostic (α < 2 ⇒ collapse) is contradicted by its own WD = 0.01 control. What would settle it: several seeds and architectures, with α and traps reported for runs that reach α < 2 without collapsing, and a comparison against the ℓ² norm under the Omnigrok LU account, which the paper does not cite.

Backlinks