Source
- Liu, Kitouni, Nolte, Michaud, Tegmark and Williams (2022), LIT-858, §3 and Appendices C–F and H, as read in NOTE-662.
- Power et al. (2022), LIT-341, for the critical training fraction in the original grokking setting, as read in NOTE-287.
What was actually shown
The model is (a, b) ↦ Dec(E_a + E_b) with learned embeddings E_k. Take an “ideal” model: zero training loss and an injective decoder. Then two training pairs with i + j = m + n force E_i + E_j = E_m + E_n (Proposition 2). Conversely, any such coincidence must respect the arithmetic (Proposition 1). An unseen pair (i, j) is answered correctly when E_i + E_j equals the sum for some trained pair. So the accuracy a representation allows can be computed from the parallelograms it realises. For 1D embeddings and addition with p = 10, that predicted accuracy matches the measured accuracy across training fractions and seeds (Fig. 3).
The equations a training set imposes form a linear system. Its null space always contains translation and scale. When it contains nothing else, the only solution is E_k = a + kb, which answers every pair. The probability that a random training set reaches this point jumps near a training fraction of 0.4 for p = 10. The number of steps for trained networks to reach 95% of the possible parallelograms diverges at the same place (Fig. 4). An effective loss on the normalised embeddings, the mean squared parallelogram defect over the training set divided by Σ|E_k|², has exactly those ground states. It conserves Σ E_k and Σ|E_k|² (proved), and its third Hessian eigenvalue, zero below the threshold and rising above it, tracks the time to structure qualitatively (Fig. 5). The same counting extends to S₃ with matrix embeddings, where three parallelograms are needed to deduce a fourth and the threshold is near 0.5 (App. H).
What could have come out otherwise: the networks could have generalised below the nullity threshold, or failed above it. In 1D toy addition, they did neither.
What this does not say
- That networks generalise this way in general. The operation is built into the architecture: a sum of embeddings, or a product of 3 × 3 matrices for S₃. The transformer and MNIST evidence in the same paper is PCA pictures and hyperparameter sweeps, with no parallelogram count. For S₃ the measured accuracy exceeds the parallelogram prediction (Fig. 18e), so even in a toy something else contributes.
- That the threshold is a phase transition. It is a jump at one size, p = 10. No scaling with p is reported, and the paper’s “phases” are defined by accuracy thresholds.
- That the effective loss is the network’s loss. It is posited, and derived only for a linear decoder with the decoder frozen (App. L). The quantitative gaps are attributed to the decoder without a test.
- That the injectivity assumption holds where grokking was found. A classifier’s decoder is not injective, and the original result is a classification task.
- Anything about which hyperparameters to use. The paper’s phase diagrams address that, and it is the anthology’s subject, not this claim’s.
Connections
THEORY-022 states what the trained modular-addition transformer computes: characters of ℤ/p at a few frequencies. This THEORY is about why a training set suffices to fix a structured embedding at all, in a simpler model. Both put the group in the task and the architecture, not in anything the network discovers.
Note of 2026-10-19: the theorem re-derived, and the measurement held to one size
The promotion audit (batch 043, ADR-047 v2 and v3) re-derived the theorem and held the measurement to what was measured.
The theorem. The model is (a, b) ↦ Dec(E_a + E_b), with embeddings E_k ∈ R^m. At zero training loss with an injective decoder, two training pairs with the same answer, i + j = u + v, force E_i + E_j = E_u + E_v (Proposition 2: equal outputs, and Dec is injective). Each such equation is linear in (E_0, …, E_(p−1)) and acts on each of the m coordinates separately. Every solution space contains the constants (E_k = a) and the scalings (E_k = kb), since both make the sum depend on i + j alone. If the nullity is 2, the solutions are exactly E_k = a + kb with a, b ∈ R^m. Then E_i + E_j = 2a + (i + j)b depends on the answer alone, so every unseen pair whose answer occurs in training gets that answer’s sum, which the decoder maps to the answer. That is the sense of “fixes the generalising representation”: unique up to translation and scale. A pair whose answer no training pair has is not answered by the representation alone; the extreme sums 0 and 2p − 2 each come from one pair.
The measurement. NOTE-662 read Liu et al. in full. For non-modular addition with p = 10 and one-dimensional embeddings, the probability that a random training set has nullity 2 jumps near a fraction of 0.4, a value the text states. The steps trained networks need to realise 95% of the parallelograms diverge at the same fraction (Fig. 4a–b). That is one study at one size, so the title is held to it (ADR-047 v3). That the critical fraction tracks the nullity threshold at other sizes, which the former promote_when asked, is QUESTION-199. Power et al.’s critical fraction comes from a different task, modular arithmetic with a transformer, and the title no longer names it.
Note of 2026-10-20: the wider claim, filed separately
When this account was narrowed, its wider form went nowhere a later result could confirm or reject it. Under ADR-047 v4 it is now the Proposed THEORY-782: that grokking’s critical training fraction is where the training set fixes the representation, for addition at every group size, as the paper proposes. This account is that one’s evidence, and nothing here changes.