LIT-085: Progress measures for grokking via mechanistic interpretability

Nanda et al. (2023) — ARXIV-2301.05217

Key takeaways

• Reverse-engineers the network rather than fitting a curve to its loss. A one-layer transformer on a + b mod 113 is shown to implement a Fourier multiplication algorithm: the embedding maps inputs to sines and cosines at a sparse set of key frequencies, attention and MLP combine them by trigonometric identities into sin/cos(w_k(a+b)), and the output matrices recombine. Addition by rotation on a circle
• The unembedding is approximately rank 10 — five frequencies × sine and cosine — and individual neurons are well approximated by degree-2 polynomials of sines and cosines at a single frequency, with the matching output direction carrying only that frequency. Computation is localised by frequency and mostly aligned with the neuron basis
• Progress measures defined from the recovered algorithm (restricted and excluded loss) show grokking is three phases: memorization, circuit formation, cleanup — continuous progress under an apparent discontinuity
• Grokking disappears above about 60% data. Sufficiently large data fractions generalize immediately; smaller fractions grok more slowly. The delayed generalization is a data-starved-regime phenomenon, not a fundamental one
• One task, P = 113, one layer, weight decay λ = 1, 40,000 epochs. The authors call the generalisation to emergence “a proof of concept”

Standing in the anthology

Read — NOTE-052. Promoted from Rejected to Active, and the reading that promoted it is the one that first declined to.

That reading said “the document is Rejected and this reading does not argue with that”, and then immediately gave the reason to keep it: grokking is the most-cited mysterious training phenomenon of its period and it goes away above 60% data. Those two sentences do not sit together. A finding that any claim about delayed generalization needs to know is not “not interesting enough to carry”, which is what Rejected means in this vocabulary.

The inconsistency became unavoidable when the same pass filed SOTA-200 — check whether an emergent capability is a metric artefact — and needed this paper for half of it. ADR-002 does permit an attic paper to source a live practice, and that permission is for a paper whose standing has moved while its result holds. This is not that case: nothing about this result moved. It was filed in the attic because its setting is small, and a small setting is a limitation to state, not a reason to retire.

What the deference cost: a reading that recorded the right finding and drew the wrong conclusion from it, because the status was already set and arguing with it felt like somebody else’s task.

None of the takeaways it carried said what the network was found to be doing — which is the result, and the thing that makes the progress measures definable at all.