Source
Olsson et al. (2022), LIT-1538, read in full in NOTE-1173: Arguments 1, 2, 4 and 6, the full-scale rows of the Model Analysis Table, and the Discussion.
What is asserted
That the circuit shown to carry in-context learning in small attention-only models (THEORY-973) carries most of it in large transformers too. In large models the same heads that copy literal random sequences are taken to do “fuzzy” or “nearest neighbor” completion over abstract representations, such as translation and template matching.
What the record holds toward it
-
Co-occurrence at every size. In the full-scale series (4 to 40 layers, 13M to 13B parameters) the score jumps in the same early window, induction heads form in it, and the after-score is about 0.4 nats at every size. There are 14 or 15 snapshots per model and no ablations.
-
One architectural intervention, the smeared keys, run only at one and two layers.
-
Three heads in the 13B model that are induction heads by the definition and also translate or match patterns. Anecdotal by design.
-
The mechanism is reverse-engineered only for attention-only models.
-
An earlier statement, with no added evidence. Elhage et al.’s framework (LIT-1534, NOTE-1189), written before the induction-heads paper, already calls induction heads “a central driver of in-context learning in language models of all sizes”.
-
No test from attribution graphs. Attribution graphs freeze attention patterns, so they cannot see an induction head’s QK step. Circuit Tracing’s Limitations section (LIT-1552, NOTE-1185) says that on “Aunt Sally … Aunt” the graph shows only that “Sally” came earlier, “a kind of regression” against the attention-only analyses. The record’s attribution-graph studies of a large model therefore neither support nor undercut this thesis.
THEORY-973 holds the small attention-only case as measured, and THEORY-977 the two-layer model’s mechanism in its weights. Nothing in the record measures the large case causally.
Why it stays Proposed
The authors’ own grading: “For small attention-only models, we present strong, causal evidence; for larger models with MLPs, we present correlational evidence.” And their alternative: “If all ‘composition heads’ form simultaneously during the phase change, then it’s possible that above some size, non-induction composition heads could together account for more of the phase change and in-context learning improvement than induction heads do.”
What would confirm or reject it
The promote_when above.
Note of 2026-10-27: the small-model anchor, qualified
This account argues from the small attention-only case (THEORY-973, THEORY-977). Causal scrubbing’s induction post (LIT-1579, NOTE-1196), read on 2026-10-27, qualifies that anchor: in a two-layer attention-only model, the textbook induction circuit accounts for only 35% of the two induction heads’ contribution on induction-favouring tokens, and 86% needs three-token matching and three layer-0 heads that attend to copies of the current token (THEORY-1020). Even where the mechanism is best understood, “induction heads” as the textbook describes them are a part of what those heads do. It adds no evidence about large models, so the account stays Proposed.