nucleation

THEORY-1021: In transformer language models, attribution patching agrees well with activation patching for small patches of single heads and their queries, keys, values and attention patterns, for nodes late in a circuit relative to the metric, and for circuits that route information through attention, and agrees badly for large patches such as whole residual streams, for the first MLP layer and the early layers of deep circuits, where nonlinearities move out of saturation, and where an effect needs several features at once, as Nanda proposes

2 min read

Properties4
StatusProposed
Tagsrepresentation-learning, causality
SourceLIT-1581 — Attribution Patching: Activation Patching At Industrial Scale
ExtendsTHEORY-1023 — Attribution patching estimates the effect of patching an activation from a clean run into a corrupted run as the difference between the two activations dotted with the gradient of the metric on the corrupted run, so the estimates for every activation come from two forward passes and one backward pass and add over sub-components; the estimate is exact when the metric is affine in the activation, gives no effect for a nonlinearity in saturation whatever the true change, and through LayerNorm ignores the component of a patch along the vector being normalized

Graph View

Source

Nanda (2023), LIT-1581, read in full in NOTE-1201: Does This Work In Practice, Does This Make Any Sense, and My Guesses for Where Attribution Patching Works.

What is asserted

From the experiments: on IOI in GPT-2 Small, the estimate is “a bad approximation for ‘big’ activations like residual stream patching and for MLP0, a good approximation for all other layer outputs, head outputs, and more zoomed in head activations”. From the author’s guesses: it works “towards the end of circuits”, for “circuits to do with routing information”, for “incremental changes in key activations” and for “small, targeted patches”, and badly for “MLP heavy circuits”, “early layers of circuits with many layers composing”, “contexts with multi-variable dependence”, attention weights “very close to zero or one”, and “when we patch a ‘big’ activation”.

What the record holds toward it

  • The mathematics (THEORY-1023): the estimate’s error is second order in patch size, it vanishes in saturation, it cannot see interactions, and through LayerNorm it ignores the component along the residual stream. Those are reasons to expect the pattern, not measurements of it.
  • One task in the post, IOI in GPT-2 Small, with scatter plots not seen here. The post states no agreement figure.
  • Kramár et al. (THEORY-996), on Pythia 410M to 12B: misses concentrated behind saturated attention and in nodes whose direct and indirect effects cancel. The first is predicted here. The second is not among the post’s failure cases; the post treats cancellation in backup heads as a source of false positives. Only fine-grained nodes (heads’ queries, keys, values and outputs, and neurons) were tested, so the residual-stream half is untouched.
  • MLP0 is explained in the post by its acting “as an extended embedding”, with discrete structure. That is a conjecture.

Why it stays Proposed

One model and one task for the measured part, and none of the rest measured, as the author says. The one systematic test held agrees on saturation, adds a failure mode the post does not name, and does not test large patches.

What would confirm or reject it

The promote_when above.

Backlinks