LIT-538: Grokking: Generalization Beyond Overfitting on Small Algorithmic Datasets

Power, Burda, Edwards, Babuschkin and Misra (2022) — ARXIV-2201.02177. Read as NOTE-281.

Key takeaways

• The phenomenon. On binary operation tables a ∘ b = c with every symbol an unrelated token, validation accuracy can rise from chance to perfect long after training accuracy saturates. Division mod 97, 50% training data: training accuracy near-perfect at < 10³ steps, validation reaching the same level near 10⁶, with very little generalization before 10⁵.
• The data dependence is in this paper. Converged performance stays at 100% across a range of training-set sizes while the time to get there grows sharply as the set shrinks. On the product in S₅, in the vicinity of 25–30% data a 1% decrease raises median steps-to-99%-validation by 40–50%. Steps to 99% training accuracy trends the other way and stays at 10³–10⁴.
• Weight decay is the strongest intervention tried, more than halving the samples needed against most alternatives; toward the origin beats toward the initialization. Gradient and weight noise help. Learning rate has to be tuned within about one order of magnitude.
• Not every operation groks. x³ + xy² + y (mod 97) failed to generalize at any fraction up to 95% within the budget — the model memorized and the data was effectively random to it. Symmetric operations need less data than their asymmetric counterparts.
• Validation loss double-descends, and learned symbol embeddings sometimes show the structure of the underlying object — a circular “number line” for modular addition.

Standing in the anthology

Active, and it is the trunk the record was missing. LIT-085 (Nanda et al.) was the only grokking paper here, and SOTA-200 staked its third check on that paper’s ~60%-data finding without the work it sharpens being present.

Its own framing is more careful than the phenomenon’s reputation. The paper proposes algorithmic datasets as a testbed — reproducible on one GPU, with data efficiency and memorization cleanly separable — and reports the regime dependence as a headline result rather than a caveat. The mystery belongs to the reception (DP-010): the striking left panel of Figure 1 is what travelled, and the centre panel of the same figure is the one that says the effect is a function of how much data you withheld.