Jing Huang, Junyi Tao, Thomas Icard, Diyi Yang and Christopher Potts (2025), arXiv:2505.11770 (ICML 2025).
Key takeaways
From the reading, NOTE-765.
- Two methods (§4.3). Counterfactual simulation scores whether a test input’s value of the located variable, transplanted into k verified examples, makes the outputs follow the high-level model (Eq. 5–6). Value probing classifies that value linearly (Eq. 7–8).
- Out of distribution (Table 3). Counterfactual simulation is best on 8 of 10 shifts, e.g. IOI with Chinese names 0.997 against 0.767 for the best confidence score. Confidence wins once (UnlearnHP, longer passage) and value probing once (MMLU, changed option symbols).
- Causal against background features. A probe of the located variable beats a probe of its orthogonal complement on 7 of 10 shifts and loses on IOI rephrased, MMLU with a distractor and UnlearnHP rephrased.
- In distribution (Table 2). Causal features are best or tied. The last-token probe scores identically to the causal-variable probe on four of five tasks.
- Confidence degrades with shift (Table 7). On RAVEL, as accuracy falls from 89% to 72%, confidence falls from 0.903 to 0.693 AUC-ROC while counterfactual simulation stays near 0.9.
- Interventional fit is not enough (Figure 4). AUC-ROC rises with IIA until IIA nears 1. Some early-layer representations with high IIA predict poorly.
- Scope. One main model, shifts selected for lowering accuracy, given high-level models, linear localization.
Standing in the record
Filed on 2026-10-12 for QUESTION-030 and QUESTION-028, and read on 2026-10-14 with Makelov et al. (LIT-968) and the reply (LIT-961), in NOTE-765. It does not cite either; it belongs to the dispute because it uses DAS-located residual-stream variables, validated by IIA, for prediction. The registration said “Intervention here separates constitutive structure from diagnostic structure”. The reading corrects that. The paper separates variables located by intervention as mediating from features chosen without causal analysis and from the rest of the same representation, and it claims nothing about constitution. Nor are its comparison features “merely correlated” ones.
For the empirical half of QUESTION-030 it is a measured case, in a
network, of intervention-identified variables predicting behaviour across
changes the mechanism ignores. For QUESTION-028 it is an instance of
intervention-relevant structure that generalizes, with nothing said about
real kinds. Filed as THEORY-287. Its purpose, correctness estimation
for deployed models, and its guidance on which method to use when are
practice for the anthology. An anthology topic on interpretability could
hold it, so it keeps the anthology-candidate flag; it is held here for
its own question under ADR-013.