nucleation

LIT-972: On Linear Representations and Pretraining Data Frequency in Language Models

2 min read

Properties5
StatusActive — read 2026-10-14 (NOTE-786) in full; worth reading as the clearest measurement of when linear structure forms in trained language models: across 24 factual relations in OLMo-1B, OLMo-7B and GPT-J, the quality of a linear relational embedding tracks mean subject–object co-occurrence in pretraining (r = 0.82), with mean causality above 0.9 past about 4.4k, 2.0k and 1.1k co-occurrences, at any training step. Read it knowing that it is correlational by the authors' own statement, that an LRE is an affine map between hidden states for one relation and not an attribute direction, and that the structure of co-occurrence across attributes is not measured. Registered for QUESTION-025, where it bears only indirectly; THEORY-269 is filed from it.
Version2
Tagsrepresentation-learning, anthology-candidate
Cited as “paper” byNOTE-786 — Merullo et al. — Linear Representations and Pretraining Data Frequency
Cited as “source” byTHEORY-269 — In OLMo-1B, OLMo-7B and GPT-J, how well an affine map from subject to object representation reproduces a factual relation (its LRE causality) tracks how often the relation's subjects and objects co-occur in pretraining, with mean causality above 0.9 beyond about 4.4k, 2.0k and 1.1k co-occurrences respectively, and, across OLMo-7B's eight checkpoints, at whatever step that count is reached

Graph View

Jack Merullo, Noah A. Smith, Sarah Wiegreffe and Yanai Elazar (2025), ICLR 2025.

Key takeaways

From the reading, NOTE-786.

  • What “linear” means here. A linear relational embedding is F(s, c) ≈ W s + b, with W the mean Jacobian, from a subject’s middle-layer state to the final state, fitted per relation on 8 examples. Its causality is the share of edits through it that change the model’s answer as intended. It is a map for one relation, not a direction for an attribute.
  • Frequency tracks it. Across 24 relations, causality correlates with mean subject–object co-occurrence within training sequences at r = 0.82. That beats subject frequency (0.66), object frequency (0.59) and 5-shot accuracy (0.74).
  • Thresholds, at any step. Mean causality exceeds 0.9 above 1,097 (GPT-J), 1,998 (OLMo-7B) and 4,447 (OLMo-1B) mean co-occurrences, “regardless of pretraining step”. It does so “consistently (but not exclusively)”.
  • Recall without linearity. Rare relations can be recalled with no consistent linear map. Star–constellation reaches 84% 5-shot accuracy at 44% causality, with about 21 co-occurrences.
  • Inverting it. LRE features predict an object’s training frequency within 10× about 70% of the time on held-out relations. Across models they give 0.65 ± 0.12 on GPT-J against 0.31 for the mean baseline. Subject–object counts are barely predicted.
  • Not causal. “We can not draw causal claims” (Appendix A).

Standing in the record

Filed on 2026-10-12 for QUESTION-025, and read on 2026-10-14 in NOTE-786. It was registered as “an empirical test of the question’s premise that co-occurrence statistics give linear attribute directions”. The reading corrects that. The paper measures relation maps, not attribute directions. It counts how often terms co-occur, not how co-occurrence is structured across attributes. And its result is a correlation. Its bearing on the question is indirect: in trained models linearity is graded and depends on exposure, so a measurement of attribute directions would need to control for frequency. The frequency–linearity finding is filed as THEORY-269. It sits beside THEORY-185, whose noise account a count threshold fits loosely. The authors’ proposals, to shape training data to meet thresholds and to infer closed models’ data, are ML practice, hence the anthology-candidate flag.

Backlinks