nucleation

LIT-1549: Causal Scrubbing: a method for rigorously testing interpretability hypotheses

2 min read

Properties6
StatusActive — read 2026-10-26 (NOTE-1168) in full, the main post of the sequence from the Alignment Forum; worth reading for the method that turns a circuit hypothesis into a mechanical test, for the argument for resampling ablation that later circuit work adopts, and for its own list of the ways a hypothesis can pass while wrong. Read it knowing that the test sees only the partitions of the data a hypothesis induces (THEORY-978), that its two applications are summarized here and detailed in companion posts the record has not read, and that hypotheses recovering 72% to 86% are accepted under a test that permits only exact swaps.
Version3
Tagsrepresentation-learning, causality, anthology-candidate
Cited as “extends” byLIT-1579 — Causal scrubbing: results on induction heads, LIT-1580 — Causal scrubbing: results on a paren balance checker, LIT-1583 — Causal scrubbing: Appendix
Cited as “paper” byNOTE-1168 — Chan, Garriga-Alonso, Goldowsky-Dill, Greenblatt et al. — Causal scrubbing
Cited as “source” byTHEORY-978 — Causal scrubbing scores a hypothesis through the partitions of the dataset that its interpretation nodes induce, so two hypotheses that assign different features to a node but induce the same partition on the dataset, such as two features perfectly correlated on it, receive the same score

Graph View

Chan, L., Garriga-Alonso, A., Goldowsky-Dill, N., Greenblatt, R., Nitishinskaya, J., Radhakrishnan, A., Shlegeris, B. & Thomas, N. (2022). Causal scrubbing: a method for rigorously testing interpretability hypotheses. AI Alignment Forum, 3 December 2022.

Read in full on 2026-10-26 (NOTE-1168); the companion posts were not read. One account was drawn from it: THEORY-978.

Why this is here

The owner asked on 2026-10-26 how the record stood on neural circuit theory; the answer found it held almost none of the core circuits literature, and the owner said “go ahead and register and read those”. This is the method by which a circuit hypothesis is tested rather than illustrated, and the source of the resampling-ablation convention that ACDC (LIT-1547) and AtP* (LIT-1531) use. The record’s circuits line held induction heads (LIT-1538), grokking (LIT-345) and the causal-abstraction readings (LIT-961, LIT-968, LIT-1310), but no account of how a stated circuit is checked. It bears on QUESTION-234 and THEORY-802: its score depends only on the partitions a hypothesis induces, so basis-related hypotheses can tie by construction.

Note of 2026-10-27: the companion posts, now held

The status note’s “detailed in companion posts the record has not read” and the body’s “the companion posts were not read” were true when written. The three posts were registered and read in full on 2026-10-27, at the owner’s request, each as its own work extending this one: the paren balance checker results (LIT-1580, NOTE-1195), the induction results (LIT-1579, NOTE-1196) and the appendix (LIT-1583, NOTE-1202). They are separate posts with their own addresses, and the two results posts add an author (Tao Lin).

Two corrections to how the summary reads, from those posts:

  • The two sets of figures are on different scales. The appendix says both results posts use the randomized-loss baseline, but they do not do so alike. The paren figures (93% down to 72%) are fractions of a 4.30-nat gap to a label-shuffled baseline; the induction figures (35% and 86%) are fractions of the 0.053 nats the two induction heads contribute on induction-favouring tokens. Set side by side, as in the summary, they are not comparable.
  • “86% refined” is hypothesis 4. The induction post’s own headline, 89%, is a later bonus hypothesis with a different rule for the previous-token head.

The measured results are THEORY-1029 (paren) and THEORY-1020 (induction), and the appendix’s maximum-entropy theorem is THEORY-1025.

Backlinks