nucleation

LIT-961: A Reply to Makelov et al. (2023)'s "Interpretability Illusion" Arguments

3 min read

Properties5
StatusActive — read 2026-10-14 (NOTE-809) in full; worth reading for the point it shares with the paper it answers: interchange interventions on one set of variables can be satisfied equally by alignments related by a change of basis, so the mediating variable is fixed only relative to a choice of units. Read it knowing that its central toy computation, which claims Makelov et al.'s test rejects their own faithful direction, holds only for an unnormalized patch; patching the subspace the rowspace component spans recovers the full effect. Its own GPT-2 experiments agree that MLP8 carries almost no name-position information. The dispute is over the label, the metric (FLDD against IIA) and how DAS is trained, not over constitution against correlation. Filed, with LIT-968, as THEORY-243.
Version2
Tagscausality, representation-learning, anthology-candidate
Cited as “paper” byNOTE-809 — Wu et al. — a reply to Makelov et al.'s interpretability illusion arguments
Cited as “source” byTHEORY-243 — In the three-unit identity network x ↦ x(1, 0, 1) ↦ (0, 2, 1)·h, aligning the input with the third unit and aligning it with the direction (h₁ + h₂)/√2, two alignments related by an orthogonal change of basis, pass every interchange intervention on the input equally, so interchange interventions on that variable alone do not fix which internal direction mediates it, THEORY-282 — A subspace intervention can reproduce the output effect of changing a variable without localizing it, by pairing a direction that varies with the variable but cannot reach the output with one that can reach the output but does not vary on the data, THEORY-802 — In trained networks as in constructed ones, interchange interventions on a fixed set of variables can be satisfied equally by alignments related by a change of basis, so which internal direction mediates a high-level variable is fixed only relative to a chosen decomposition into units, and complex models will generally have several distinct, fully correct abstractions

Graph View

Zhengxuan Wu, Atticus Geiger, Jing Huang, Aryaman Arora, Thomas Icard, Christopher Potts, Noah D. Goodman (2024), arXiv:2401.12631.

Key takeaways

From the reading, NOTE-809.

  • The illusion restated (§2, Eq. 7–8). The effect of a patch splits into a null-space term and a rowspace term, and the “illusion effect” is the difference between patching v and patching its rowspace part.
  • The toy counterexample (§3.2) does not hold as the test is specified. The reply finds that h₃, Makelov et al.’s faithful direction, keeps only 20% of its effect when patched along the unnormalized rowspace vector. Patching the subspace that vector spans, which is Makelov et al.’s test and, by the reply’s footnote 1, their code, gives the full effect. The toy cannot discriminate either way, because its readout is one-dimensional.
  • The general point (§3.3) stands. A faithful direction need not be orthogonal to the null space of downstream weights, because the data-induced submanifold generally is not.
  • Multiple abstractions (App. A). A rotation makes (h₁ + h₂)/√2 carry the input as h₃ did. Under the interventions performed the two alignments tie; aligning further variables separates them.
  • Metrics and set-up (§4). A 0.505 → 0.5025 change gives about 50% FLDD with no behavioural change. Low IIA (4.2%) is read as no relevant structure. The factual-recall DAS was trained and tested on single fact pairs.
  • Their GPT-2 results (§5). Name position is written by the layer-8 attention block and spread over heads (no single head above 0% IIA; heads 6 and 10 matter most). The IO name appears at layer 9, with head 9.9 most important but about 11 heads needed.

Standing in the record

Filed on 2026-10-12 for QUESTION-030, and read on 2026-10-14 after Makelov et al. (LIT-968), in NOTE-809. The registration said the dispute “turns on whether a subspace interchange intervention finds the variable that constitutes a behaviour or one that only correlates with it”. It does not. Both sides agree on the facts at MLP8 and in the residual stream, and neither speaks of constitution or correlation. They disagree over whether an effect running through components the next layer cannot read discredits a found subspace, over the metric, and over how DAS should be trained. What bears on QUESTION-030 is the point both papers grant, filed as THEORY-243: interventions on a fixed set of variables do not by themselves pick the mediating variable from alignments related by a change of basis. That is the network-level form of the concession the question records, that causal tests do not settle constitution without a theory of the act. It supports the premise of CLAIM-150.

The reply does not undercut THEORY-282, filed from LIT-968: it contests the label and the toy, not the dependence of the MLP8 effect on the null space. Its advice on training and evaluating DAS is for the anthology. An anthology topic on interpretability could hold it, so it keeps the anthology-candidate flag, and it is held here for its own question under ADR-013.

Backlinks