Source
Meng et al. (2022), LIT-1536, read in full in NOTE-1166.
What is asserted
A mechanism, not a measurement: mid-layer MLPs at the subject’s last token act as linear associative memories, W_proj k ≈ v, from a subject key to an attribute value; “middle layer MLP outputs accumulate information, then the summed information is copied to the last token by attention at high layers”; and, from Zhao et al.’s finding that BERT’s layers can be reordered, “there is no further special role for the particular choice or arrangement of individual layers in the middle range”.
What the record holds toward it
- For: the traces and edits of THEORY-998. Restoring mid-layer MLP outputs at the last subject token recovers recall; editing there generalizes and stays specific; editing late attention only regurgitates.
- Against the inference: Makelov et al. (LIT-968, THEORY-282). Fact patches in GPT2-XL’s MLPs found by distributed alignment search depend on the null space of W_out, the illusion’s signature; any such patch has a rank-one edit with the same output at that token (their Lemma D.2); and ROME edits are approximated by subspace interventions across layers 20–35. An edit at a layer can work because the layer can write the needed output, whether or not it holds the fact.
- Untested: the conjecture that any middle layer could hold any fact.
Why it stays Proposed
The paper’s own warrant for “how” as well as “where” is that edits at the traced site work: “This evidence suggests that we have an accurate understanding not only of where factual associations are stored, but also how.” That step is the one LIT-968 shows does not follow. The tracing result is about sufficiency of restored states in a noised run, not about where recall happens in a clean one.
What would confirm or reject it
The promote_when above.
Note of 2026-10-27: Hase et al. meet the refutation clause for the localization part
Three works this entry leans on or names were registered and read on 2026-10-27, at the owner’s request (“get started on the seven works and markov IBP”).
Hase et al. (LIT-1578, NOTE-1194, THEORY-1024). The promote_when above says this account is “refuted if edits of equal quality are available at layers that tracing does not implicate”. Hase, Bansal, Kim and Ghandeharioun report exactly that, in GPT-J on 652 CounterFact facts the model knows:
- ROME’s mean rewrite score is 99% at layer 6 and above 96% at every edited layer but the last (28), including layers where causal tracing does not place the fact.
- For ROME at layer 6, the per-fact tracing effect and the rewrite score correlate at ρ = −0.13. A regression of edit success on the edit layer alone gives R² 0.947; adding the per-fact tracing effect gives 0.948, a gain of 0.1 point.
- Across ROME, MEMIT and constrained fine-tuning on one or five layers, and five editing objectives, the largest gain from tracing is 3.2 points of R² (fine-tuning on five layers under Fact Forcing, the one objective that edits on the noised input tracing uses).
So the localization part of this account (“stored in middle-layer MLP modules … at the subject’s last token”, read as the traced site being where the fact is) meets the account’s own refutation condition, for the rewrite score, in the one model and dataset measured (GPT-2 XL and zsRE are reported to agree). The layer still explains most of the variance, and ROME is best on average near where tracing peaks on average (THEORY-998 is not contradicted). What fails is the per-fact link from trace to edit.
The conjecture that “any middle layer could store a given fact equally” is not refuted. It is close to Hase et al.’s own reading that “many layers could store a fact”, filed Proposed as THEORY-1026, and the result is if anything friendly to it.
Status left for the owner. Under ADR-047 the result would support a disposition of split (the localization claim against the equal-storage conjecture) or needs-correction, rather than a flat rejection; which, if any, is the owner’s decision, and this note changes neither the status nor the title. The “Untested” line under “What the record holds toward it” now reads, for the conjecture, as tested indirectly and not refuted; and the localization part has a measured Against case beside LIT-968’s argument.
Geva et al. (LIT-1576, NOTE-1200). The key–value reading this account takes from Geva et al. is borrowed as an idea, not as a form. Geva’s memories are unit-level: each of a feed-forward layer’s hidden units has a key (a row of the input matrix) and a value (a row of the output matrix), and the values promote next tokens (THEORY-1033, measured in one WikiText-103 model). ROME’s memory is matrix-level: the whole output matrix W_proj as a linear associative memory keyed by an activation vector, holding a subject–attribute fact. Geva supports this account only at the level of “MLPs can be read as memories”.
MEMIT (LIT-1582, NOTE-1197, THEORY-1031) spreads an edit over layers 3–8 of GPT-J and inserts 10,000 facts at once. It adds no support toward promotion: its layer range is chosen from averaged traces, and its ablations compare averaged ranges, not facts.
Note of 2026-10-27 (later): rejected, and the conjecture split out
At the owner’s decision on the note above, this entry is Rejected. Its defining claim is that the middle layers causal tracing implicates are where a fact is stored, so that editing there succeeds and editing elsewhere does not. That claim met the refutation clause the entry wrote for itself: Hase et al. (LIT-1578, THEORY-1024) find edits of equal quality at layers tracing does not implicate.
The entry was also inconsistent with itself, which is why it is split rather than simply rejected. Its title includes Meng et al.’s conjecture that “any middle layer could store a given fact equally”, and that conjecture predicts exactly the result the promote_when counted as a refutation. The conjecture is not refuted, and the source asserts it, so under ADR-047 (kind 1) it is filed as its own Proposed entry, THEORY-1039. Its promote_when carries over this entry’s clean-input ablation test and adds a control for edits that create an association wherever they are made (LIT-968). Hase et al.’s own account, that localizing and editing are different questions, stays THEORY-1026.
What stands from the paper is the measurement, THEORY-998: the traces and the edits as measured. Only the inference from them to localized storage is rejected.