Source
Olsson, Elhage, Nanda, Joseph et al. (2022), In-context learning and induction heads, LIT-1538, read in full in NOTE-1173 from the arXiv print of the Transformer Circuits article. The account rests on Arguments 1–3, the smeared-key models, Model Details, the data-collection appendix (pattern-preserving ablation) and the small attention-only rows of the Model Analysis Table.
What was actually shown
The models. Attention-only transformers of one to six layers, d_model 768, 12 heads per layer, trained once each for 10,000 steps (about 10¹⁰ tokens) with 200 snapshots, and one- and two-layer “smeared key” variants trained the same way. Everything is measured on the same 10,000 examples of 512 tokens.
The measures. The in-context learning score is the mean loss at token 500 minus the mean loss at token 50. An induction head is a head that, on repeated random tokens, attends to the token that followed an earlier copy of the present one (prefix matching) and raises that token’s logit (copying).
Co-occurrence. In every model with two or more layers the score falls from above −0.15 to about −0.4 nats within one early window and then stays there. In the same window heads acquire high prefix-matching scores, the training loss has its one non-convex stretch, and the models’ trajectories in the PCA of per-token losses turn. The one-layer model shows none of the four.
Co-perturbation. A smeared key, k_j = σ(α)k_j + (1 − σ(α))k_{j−1} with α trained per head, lets a head attend by the previous token without a second layer. With it the one-layer model’s score drops abruptly too, and the two-layer model’s drop comes earlier than in the vanilla model.
Ablation. Zeroing one head’s output with all attention patterns frozen shows the induction heads carrying the score: “almost all the in-context learning in small attention-only models appears to come from these induction heads! This begins at the start of the phase change, and remains true through the end of training.” In an attention-only model with frozen patterns, the logits are a sum of per-head terms and a direct path term up to LayerNorm rescaling, so single-head ablations are close to an exact decomposition. That is why the authors treat these ablations as dispositive here and not for models with MLPs.
What could have come out otherwise. The score could have grown gradually, or with depth. The jump could have appeared in one-layer models, or stayed put under smeared keys. Ablating the induction heads could have left most of the score, with other copying heads carrying it; the authors note that basic copying heads exist in one-layer models, which never make the jump.
What this does not say
- Not about models with MLPs, or large models. There the evidence is correlational, anecdotal or by analogy, and the authors say so. That thesis is THEORY-990.
- Not that induction drives the transition rather than accompanying it. The authors name a shared cause they cannot exclude: the transition may be “the point at which the model learns how to compose layers through the residual stream”, of which induction heads are one product. The ablations show that induction heads carry the score afterwards, not that their formation caused the bump.
- Not about few-shot task learning. The score measures loss improvement with position. A model can raise it by getting worse at ordinary prediction: ablating most non-induction heads increases it (footnote 17).
- Not robust to data order. Each model was trained once, and all models saw the same examples in the same order. The window also overlaps the learning-rate warm-up, which ends at 1.5 × 10⁹ tokens.
- Not that sequential onset comes from the optimizer. No optimizer is varied. The timing evidence here is a dependency: the jump waits on a composition, and removing the need to compose moves it.
Connections
- QUESTION-039: answers in part. For in-context learning in small attention-only transformers, the moment of acquisition is set by when a two-head composition becomes available. It does not say what sets that, and it is one model family.
- THEORY-253 and THEORY-783. Those hold the optimizer as a source of sequential onset, beside dependence between skills. This account is a measured case of the second kind: removing a dependency moves the onset.
- QUESTION-275. This is the setting the question names first, and its circuit-only measurements (per-head scores and ablations at every snapshot) are a baseline the question’s coupled model has to beat.
- THEORY-169. The in-context learning here is conditioning on fixed weights, the case THEORY-169 describes for single layers.
- THEORY-977. The same mechanism read from the weights of one trained two-layer model (Elhage et al., LIT-1534): K-composition with a previous-token head. This account says when the mechanism forms; that one says what it is once formed.
- Refinements of the mechanism, not of its timing. Causal scrubbing’s induction result, 35% of the loss recovered by the naive previous-token-key hypothesis and 86% once three pathways are added, held only from a summary (NOTE-1168); and induction at the level of sparse-autoencoder features, in pieces, in Pythia-70M (NOTE-1187). Neither measures when the composition becomes available.
Note of 2026-10-27: how much of induction the textbook circuit explains
Causal scrubbing’s induction post (LIT-1579, NOTE-1196), read on 2026-10-27, measures in a two-layer attention-only model how much of the two induction heads’ contribution on induction-favouring tokens the standard circuit accounts for: 35%, and 86% once three-token matching, layer 0’s average effect and three self-attending layer-0 heads are added (THEORY-1020). It is compatible with this entry, which measures that induction heads carry in-context learning, not that the textbook circuit is all of what they do.