nucleation

LIT-1534: A Mathematical Framework for Transformer Circuits

1 min read

Properties5
StatusActive — read 2026-10-26 (NOTE-1189) in full, from the Transformer Circuits article; worth reading for the exact decomposition of an attention-only transformer into end-to-end paths and of each head into separable QK and OV circuits (THEORY-988, re-derived and checked by the record), for the three kinds of head composition, and as the paper in which induction heads are found, by K-composition with a previous-token head, in a two-layer attention-only model (THEORY-977). Read it knowing that the models are toys without MLPs, that the framework ignores layer norm and biases, that the composition diagram was corrected after publication, and that the interactive figures carry much of the evidence.
Version2
Tagsrepresentation-learning, compositionality, anthology-candidate
Cited as “paper” byNOTE-1189 — Elhage, Nanda, Olsson et al. — A mathematical framework for transformer circuits
Cited as “source” byTHEORY-1342 — In Transformer attention the weights each query puts on the positions depend only on query–key compatibility, softmax(QKᵀ/√d_k), and the output is those weights applied to the values, so for a fixed input what a head selects and what it transports are separate factors; because queries, keys and values are all projections of the same input, the separation holds only with the input, and so the pattern, held fixed, THEORY-977 — In the two-layer attention-only transformer that Elhage et al. reverse-engineer, the second-layer heads that compose substantially with first-layer heads do so through their keys, mainly with a head that attends to the previous token and, after the published correction, also with one that attends over the last several tokens; these heads are induction heads, attending from a token to the one after its earlier occurrence, including on repeated random tokens, with positive-eigenvalue copying OV and same-matching QK circuits, and the second-order virtual-head terms add little to the loss reduction, THEORY-988 — In an attention-only transformer without layer norm or biases, once every attention pattern is held at its value on the input, the logits are exactly a sum of end-to-end path terms, a direct path from embedding to unembedding, one term per head and one per chain of heads composed through their values, and each head's attention scores are a bilinear form in the tokens through which earlier heads can enter on the query side, the key side or both

Graph View

Elhage, N., Nanda, N., Olsson, C., Henighan, T., Joseph, N., Mann, B., et al. (2021). A mathematical framework for transformer circuits. Transformer Circuits Thread, 22 December 2021. https://transformer-circuits.pub/2021/framework/index.html

Read in full on 2026-10-26 (NOTE-1189). Two accounts were drawn from it: THEORY-988 and THEORY-977.

Why this is here

Registered and read at the owner’s request of 2026-10-26, after an answer to “how’s the record on neural circuit theory?” found that the record held almost none of the core circuits literature. The record’s transformer circuits line rested on this paper without holding it. The induction-head paper (LIT-1538) takes its induction head and its K-composition mechanism from here, and THEORY-973 and the readings of Wang et al.’s refined learning coefficients (THEORY-975, THEORY-967) and of Musat et al. (THEORY-981, THEORY-1009) use its two-layer attention-only setting and its vocabulary of QK and OV circuits. Its observation that the residual stream has no privileged basis is the setting of THEORY-802 and THEORY-018.

Backlinks