Source
Hase, Bansal, Kim and Ghandeharioun (2023), NeurIPS 2023, arXiv:2301.04213, LIT-1578, read in full in NOTE-1194.
What was actually shown
Setting. GPT-J (6B, 28 layers), 652 CounterFact facts it completes correctly. Causal tracing as in Meng et al. (LIT-1536): Gaussian noise on the subject’s embeddings, MLP outputs restored over a window of five layers, and the effect at a layer taken as the maximum over tokens, as a fraction of the probability lost. Edits at layers {1, 5, 6, 9, 13, 17, 21, 25 (or 24), 28} by ROME (one layer), MEMIT (five), and ℓ∞-constrained fine-tuning (one or five).
Measure. For each method and objective, the R² of a regression of edit success on the edit layer (categorical), and the gain when the per-fact tracing effect at that layer is added, with an F-test.
Results. Error Injection with ROME: ρ = −0.13 at layer 6; R² 0.947 from the layer, 0.016 from tracing alone, 0.948 from both. Across the other objectives (Tracing Reversal, Fact Erasure, Fact Amplification, Fact Forcing) and methods, the gain is at most 0.032, and is significant and largest for fine-tuning under Fact Forcing (0.027 and 0.032), the one objective that edits on the noised input. ROME’s mean rewrite score is 99% at layer 6 and above 96% at every layer but 28. The same holds for paraphrase, neighbourhood and overall scores, for a tracing window of 10, for the tracing effect at the last subject token, for GPT-2 XL, for ZSRE, for representation zeroing, and for the 74 facts whose tracing effects are concentrated in a few layers.
What could have come out otherwise. Facts traced to a layer could have been edited best at that layer, and facts traced elsewhere edited worse at the default layer. Editing five layers could have aligned with a five-layer tracing window. The concentrated-effect subset could have shown the link that the full set hides. None did.
What this does not say
- Not that the layer is irrelevant. The layer explains most of the variance, and ROME is best on average at layer 6, near where tracing peaks on average. The result is about the per-fact deviation from that average.
- Not that facts are stored nowhere, or everywhere. “Stored” here means where causal tracing peaks. The finding is that this does not locate the edit. It does not say what would.
- Not that causal tracing is uninformative. It is a statement about tracing as a guide to editing. Meng et al.’s traces (THEORY-998) stand as measurements.
- Not beyond the setting. Two models up to 6B parameters, short English factual prompts, layer-level MLP edits, and denoising or zeroing as the localization.
- Not why. The authors’ account, that tracing and editing answer different questions and that an edit at one layer can override what others have accumulated, is THEORY-1026, Proposed.
Connections
- THEORY-1011. Its promote_when names this test and its refutation condition (“edits of equal quality are available at layers that tracing does not implicate”). This result meets that condition for the rewrite score. See NOTE-1194.
- THEORY-282. Makelov et al.’s equivalence of fact patches and rank-one edits gives a reason an edit can work at a layer whatever that layer carries. That is a mechanism for this result, from another direction.
- THEORY-998 is the average-over-facts result this one is consistent with.