nucleation

LIT-1581: Attribution Patching: Activation Patching At Industrial Scale

1 min read

Properties6
StatusActive — read 2026-10-27 (NOTE-1201) in full, from the author's own page; worth reading as the origin of attribution patching, for its derivation of the estimate and the identities that make it cheap and additive (THEORY-1023), for the account of where a first-order estimate should fail (THEORY-1021), and for its framework of what patching can and cannot show: sufficiency against necessity, specific against general understanding, and parallel against serial redundancy. Read it knowing that the author calls it an incomplete project, that its empirical evidence is one task in GPT-2 Small shown only in plots and a notebook, and that the systematic test is Kramár et al. (LIT-1531).
Version2
Tagsrepresentation-learning, causality, anthology-candidate
Cited as “extends” byLIT-1531 — AtP*: An efficient and scalable method for localizing LLM behaviour to components
Cited as “paper” byNOTE-1201 — Nanda — Attribution patching
Cited as “source” byTHEORY-1021 — In transformer language models, attribution patching agrees well with activation patching for small patches of single heads and their queries, keys, values and attention patterns, for nodes late in a circuit relative to the metric, and for circuits that route information through attention, and agrees badly for large patches such as whole residual streams, for the first MLP layer and the early layers of deep circuits, where nonlinearities move out of saturation, and where an effect needs several features at once, as Nanda proposes, THEORY-1023 — Attribution patching estimates the effect of patching an activation from a clean run into a corrupted run as the difference between the two activations dotted with the gradient of the metric on the corrupted run, so the estimates for every activation come from two forward passes and one backward pass and add over sub-components; the estimate is exact when the metric is affine in the activation, gives no effect for a nonlinearity in saturation whatever the true change, and through LayerNorm ignores the component of a patch along the vector being normalized

Graph View

Nanda, N. (2023). Attribution patching: Activation patching at industrial scale. Blog post, 4 February 2023, https://www.neelnanda.io/mechanistic-interpretability/attribution-patching.

Read in full on 2026-10-27 (NOTE-1201). Two accounts were drawn from it: THEORY-1023 (Active) and THEORY-1021 (Proposed).

Why this is here

Registered and read at the owner’s request of 2026-10-27, as one of the works the circuits readings of 2026-10-26 named and did not register. The record held attribution patching only through Kramár et al.’s test of it (LIT-1531, NOTE-1190), and through the work built on it: its edge version with integrated gradients, by which LIT-1539 and MIB’s circuit track (LIT-1550) find circuits, and sparse feature circuits (LIT-1532), whose nodes are selected by linear estimates of indirect effect. This is the method’s own statement: what is estimated, why it is cheap, and where its author expected it to fail. LIT-1531 was preferred as the record’s entry because it tests the method against ground truth. This post gives the derivation and the reasons for its failure modes, which LIT-1531 then measured.

Backlinks