nucleation

THEORY-1000: For fine-grained nodes of transformer language models generally, attribution patching corrected for attention saturation is a reliable prefilter for activation patching: nodes it ranks low rarely have large single-node patching effects, so verifying nodes in its order finds the important ones at a small multiple of the cost of an oracle, as Kramár et al. expect

1 min read

Properties5
StatusProposed
Version2
Tagsrepresentation-learning, causality
SourceLIT-1531 — AtP*: An efficient and scalable method for localizing LLM behaviour to components
ExtendsTHEORY-996 — On Pythia language models from 410M to 12B parameters, in three single prompt pairs and two small prompt distributions, verifying attention or neuron nodes by activation patching in the order given by attribution patching, a first-order gradient estimate of each node's noising effect, found the nodes with the largest effects at about 2 to 50 times the cost of an oracle ordering, below unordered patching, subsampling and block search in every setting; its misses were concentrated in queries and keys behind saturated attention and in nodes whose direct and indirect effects nearly cancel

Graph View

Source

Kramár, Lieberum, Shah and Nanda (2024), LIT-1531, read in full in NOTE-1190: §5.1 (limitations) and §5.4 (recommendations).

What is asserted

That what THEORY-996 measured in Pythia holds for transformer language models of other families and larger sizes, on tasks beyond the five tried: gradient-ordered verification, with the softmax corrected, misses few large-effect nodes.

What the record holds toward it

  • THEORY-996 (Active): four Pythia sizes, five settings.
  • The failure modes are understood mechanistically (saturation, cancellation), which is a reason to expect transfer and also names where it could fail: models with sharper attention or more cancellation.
  • Nothing in the record measures another family.

What would confirm or reject it

The promote_when above.

Note of 2026-10-27: the method author’s account

Nanda’s post (LIT-1581, NOTE-1201), read on 2026-10-27, gives the method author’s own account of where attribution patching agrees and disagrees with activation patching, filed Proposed as THEORY-1021. Its promote_when is compatible with this one’s; it adds the size of a patch and a node’s depth in the circuit as conditions to vary. The estimate itself is THEORY-1023.

Backlinks