nucleation

THEORY-782: In networks that learn addition from summed embeddings, the critical training fraction of grokking is the fraction at which the training set's same-answer pairs come to fix the generalising representation up to translation and scale, at every group size, as Liu, Kitouni, Nolte, Michaud, Tegmark and Williams propose

2 min read

Properties5
StatusProposed
Tagsrepresentation-learning, learning-theory, anthology-candidate
SourceLIT-858 — Towards Understanding Grokking: An Effective Theory of Representation Learning, LIT-341 — Grokking: Generalization Beyond Overfitting on Small Algorithmic Datasets
ExtendsTHEORY-181 — In a model that decodes the sum of two learned embeddings, at zero training loss with an injective decoder, a training set fixes the generalising representation of addition when its same-answer pairs leave only translation and scale free; for non-modular addition with p = 10 and one-dimensional embeddings, generalisation sets in near the training fraction, about 0.4, at which that becomes likely
AnswersQUESTION-199 — Does the training fraction at which networks summing two learned embeddings generalise follow the fraction at which the training set's same-answer equations leave only translation and scale free, across group sizes?

Graph View

Source

Liu, Kitouni, Nolte, Michaud, Tegmark and Williams (2022), LIT-858, §3 and Appendices C–F and H, as read in NOTE-662; Power et al. (2022), LIT-341, for the critical fraction in the original grokking setting.

What is asserted

That grokking’s critical training fraction is set by the training data’s algebra: a network that sums two learned embeddings generalises once the same-answer pairs in its training set impose enough equations to leave the embedding free only up to translation and scale, and the critical fraction is where that first becomes likely. The paper offers this as the explanation of the critical fraction, not only as a match at one size.

What the Active account establishes toward it

THEORY-181 holds Propositions 1–2 for the ideal model (zero training loss, injective decoder), re-derived, and the one measurement: for non-modular addition with p = 10 and 1D embeddings, generalisation sets in near the fraction, about 0.4, at which that becomes likely.

Why it is held to addition

The pre-narrowing wording, and the sweep’s first suggestion, spoke of the model or of arithmetic in general. For S₃ the paper’s predicted accuracy is only a lower bound and the networks generalise by something beyond it (NOTE-662, App. H, Fig. 18e). So the source’s own results do not support the thesis beyond addition, and this entry does not claim it.

What would confirm or reject it

The promote_when above, QUESTION-199’s test across group sizes.

Provenance

Filed on 2026-10-20 by the narrowing-loss sweep (ADR-047 v4, batch 068), from the critical-fraction clause THEORY-181’s title carried before it was narrowed on 2026-10-19.

Backlinks