nucleation

LIT-1578: Does Localization Inform Editing? Surprising Differences in Causality-Based Localization vs. Knowledge Editing in Language Models

1 min read

Properties5
StatusActive — read 2026-10-27 (NOTE-1194) in full, from the arXiv copy (v2, the NeurIPS 2023 version); worth reading as the direct test of whether causal tracing tells where to edit a fact: in GPT-J, fact by fact, it does not. The tracing effect at a layer adds at most 3.2 points of explained variance in edit success once the edit layer is known, and ROME's rewrite score is above 96% at every layer but the last (THEORY-1024). Read it knowing that its own account of why, that tracing and editing answer different questions, is offered and not tested (THEORY-1026), that "where a fact is stored" means where causal tracing peaks, and that its Table 3 and Figure 6 disagree on one cell.
Version2
Tagsrepresentation-learning, causality, anthology-candidate
Cited as “paper” byNOTE-1194 — Hase, Bansal, Kim and Ghandeharioun — Does localization inform editing?
Cited as “source” byTHEORY-1024 — In GPT-J, for 652 CounterFact facts the model knows, the per-fact causal-tracing effect at an MLP layer adds at most 3.2 points of explained variance in the success of editing that fact at that layer, beyond what the choice of layer alone explains, across ROME, MEMIT and constrained fine-tuning and five editing objectives; ROME rewrites the fact at over 96% at every layer but the last, including layers where tracing does not place it, and the only reliable link appears when the edit is made on the noised input that tracing uses, THEORY-1026 — In transformer language models, locating where a forward pass carries information about a fact and finding where a weight edit best changes the expressed fact are different questions, because an edit at one layer can override information accumulated across other layers, so that many layers could hold a fact and some happen to, as Hase, Bansal, Kim and Ghandeharioun propose, THEORY-1039 — In autoregressive transformer language models, factual associations are recalled through middle-layer MLP modules acting as key–value memories, and any fact could be stored equally in any one of the middle MLP layers, as Meng, Bau, Andonian and Belinkov conjecture

Graph View

Hase, P., Bansal, M., Kim, B. and Ghandeharioun, A. (2023). Does localization inform editing? Surprising differences in causality-based localization vs. knowledge editing in language models. NeurIPS 2023. arXiv:2301.04213.

Read in full on 2026-10-27 (NOTE-1194). Two accounts were drawn from it: THEORY-1024 and THEORY-1026.

Why this is here

Registered and read at the owner’s request of 2026-10-27, to fill a gap the circuits readings of 2026-10-26 named. The reading of ROME (LIT-1536, NOTE-1166) filed the storage hypothesis THEORY-1011 Proposed, and its promote_when asks for exactly this test: whether the layer where causal tracing finds a fact predicts, fact by fact, where an edit of that fact succeeds. NOTE-782, the reading of Makelov et al. (LIT-968), already cited the finding without the record holding it. It is the empirical half of the argument whose formal half is Makelov et al.’s Lemma D.2: a successful edit at a layer does not show that the fact is there.

Backlinks