Aleksandar Makelov, Georg Lange and Neel Nanda (2023), arXiv:2311.17030 (NeurIPS 2023 Workshop on Attributing Model Behavior at Scale).
Key takeaways
From the reading, NOTE-782.
- The mechanism (§3, Eq. 3). Patching along v = (v_disconnected + v_dormant)/√2 changes the MLP output by ½(v_disconnectedᵀact_A − v_disconnectedᵀact_B) W_out v_dormant. The intervention converts variation the model cannot use into a push along a direction it does not use on the data. Equal weights are optimal (Lemma A.1).
- IOI in GPT-2 Small (Table 1). A DAS direction in MLP8 gives 46.7% FLDD and 4.2% interchange accuracy; without its null-space part, 13.5%; the whole MLP patch, −8%. MLP8 does not write the feature on the data (Figure 4), and a random MLP gives an even stronger direction (B.5).
- The success case (§5). The residual-stream direction after layer 8 gives 140.7% FLDD and 74.8% accuracy, keeps 127.5% without the part the name movers cannot read, and is written only by the S-inhibition heads.
- Factual recall (§6). DAS fact patches in GPT2-XL MLPs depend on the null space of W_out. Any such patch has a rank-1 edit with the same output at that token (Lemma D.2), and ROME edits are approximated by subspace interventions in layers 20–35 (§6.3).
- Prevalence (§7). Argued, not measured: any unused MLP between two components that communicate a feature should contain such a direction.
- Its own limits (§8). No definition of a ground-truth explanation; a cancelling-subspaces reading of MLP8 is rejected on parsimony; the verdict depends on treating heads and MLP layers as the units of explanation, and in a rotated basis the toy’s illusory direction becomes the faithful one (A.3).
Standing in the record
Filed on 2026-10-12 for QUESTION-030, and read on 2026-10-14 with the reply (LIT-961) and Huang et al. (LIT-987) as one dispute, in NOTE-782. The registration said the paper shows that “interventional success does not identify the constituting relation”. The reading narrows that: it shows that interventional success does not identify the mediating variable, and it gives a test that tells the two apart in an MLP. It says nothing about constitution. For QUESTION-030 it is a worked case, in a network, of an intervention that creates the relation it was meant to detect by setting a variable to values the system never takes. That is a caution for do(S) in CLAIM-130’s sense, and a concrete instance of the underdetermination premise of CLAIM-150.
Filed from the reading: THEORY-282, the illusion; and, from the point this paper and the reply share, THEORY-243, that alignments related by a change of basis can tie under interchange interventions.
The paper’s closing advice to interpretability practitioners (search in
bottlenecks, validate beyond end-to-end effects) is an instruction for
machine-learning practice and is not filed here. An anthology topic on
interpretability could hold the paper, so it keeps the
anthology-candidate flag; it is held here for its own question under
ADR-013.
Note of 2026-10-27: THEORY-1011 rejected; this argument now in THEORY-1039’s test
The argument this reading holds, that an edit’s success at a layer does not show the layer held the fact, now carries an outcome. THEORY-1011, which inferred localized storage from ROME’s traces and edits, is Rejected after Hase et al. (THEORY-1024) met its refutation clause. THEORY-1039, the equal-storage conjecture split out of it, takes Makelov et al.’s Lemma D.2 into its promote_when: more successful edits cannot settle it.