nucleation

LIT-1582: Mass-Editing Memory in a Transformer

1 min read

Properties6
StatusActive — read 2026-10-27 (NOTE-1197) in full, from the arXiv copy (v2, the ICLR 2023 version); worth reading as ROME's extension from one edit to thousands, by a closed-form least-squares update spread over the range of MLP layers that causal tracing implicates, and for the scaling curves that show sequential single edits failing where it holds (THEORY-1031). Read it knowing that it inherits ROME's storage reading (THEORY-1011) rather than testing it, that its layer range is chosen from traces averaged over facts, which Hase et al. (LIT-1578) show do not predict per fact where an edit works, and that its zsRE specificity hardly moves for any method.
Version2
Tagsrepresentation-learning, causality, anthology-candidate
ExtendsLIT-1536 — Locating and Editing Factual Associations in GPT
Cited as “paper” byNOTE-1197 — Meng, Sen Sharma, Andonian, Belinkov and Bau — Mass-editing memory in a transformer
Cited as “source” byTHEORY-1031 — In GPT-J, inserting 10,000 CounterFact counterfactuals at once by a closed-form least-squares update of the MLP output matrices of layers 3 to 8, spread so that each layer writes a share of each fact's required change to the subject's hidden state, gives an editing score of 85.8 (efficacy 98.9, paraphrase 88.6, neighbourhood 73.7) with fluency kept, against 67.6 for weight-decayed fine-tuning, which breaks generation, 50.3 for ROME applied one fact at a time and 23.1 for MEND; GPT-NeoX scores 82.0, and editing late MLPs or attention instead does worse

Graph View

Meng, K., Sen Sharma, A., Andonian, A., Belinkov, Y. and Bau, D. (2022). Mass-editing memory in a transformer. ICLR 2023. arXiv:2210.07229.

Read in full on 2026-10-27 (NOTE-1197). One account was drawn from it: THEORY-1031.

Why this is here

Registered and read at the owner’s request of 2026-10-27, to fill a gap the circuits readings of 2026-10-26 named: ROME’s extension to many edits, which the reading of ROME (NOTE-1166) cited without the record holding it. It is the method Hase et al. (LIT-1578) test alongside ROME, and its choice of a range of layers rather than one is the clearest case of a localization used to design an edit. The record’s question is what that design shows about where facts are (THEORY-1011), not how to edit models, which is practice and the anthology’s.

Backlinks