nucleation

LIT-1538: In-context Learning and Induction Heads

1 min read

Properties5
StatusActive — read 2026-10-26 (NOTE-1173) in full, from the arXiv print of the Transformer Circuits article; worth reading as the first mechanistic account of a transition in training, for the in-context learning score, and for the way it grades its own evidence by model class. Read it knowing that the case is causal only for small attention-only models (THEORY-973), that the large-model thesis is correlational and by analogy (THEORY-990), that every model was trained once on the same data order, and that its dates for the window disagree between sections.
Version2
Tagsrepresentation-learning, learning-theory, anthology-candidate
Cited as “paper” byNOTE-1173 — Olsson et al. — In-context learning and induction heads
Cited as “source” byTHEORY-973 — In small attention-only transformer language models of two to six layers, in-context learning, measured as the loss at token 500 minus the loss at token 50, appears abruptly in one early window of training in which induction heads form, and ablating those heads at test time removes nearly all of it; one-layer models show neither, and letting a single layer attend by the previous token brings the jump to one-layer models and moves it earlier in two-layer ones, THEORY-990 — In transformer language models of every size, including large ones with MLPs, induction heads, extended to fuzzy matching of similar tokens and abstract patterns, are the mechanism of most in-context learning, as Olsson et al. propose

Graph View

Olsson, C., Elhage, N., Nanda, N., Joseph, N., et al. (2022). In-context learning and induction heads. Transformer Circuits Thread, 8 March 2022. arXiv:2209.11895.

Read in full on 2026-10-26 (NOTE-1173). The summary is checked against the reading.

Why this is here

Cited at T24 for the setting where the reply would first test the record’s perspective framework on training dynamics: which partial capacities exist before the induction circuit forms. The record held it only as a pointer in NOTE-283 and in the reading of LIT-153 (NOTE-106). It is the empirical anchor of QUESTION-275, and bears on QUESTION-039 (what sets the order and moment at which a network acquires a structure).

Backlinks