Source
Merullo, Smith, Wiegreffe and Elazar (2025), “On Linear Representations and Pretraining Data Frequency in Language Models”, ICLR 2025, LIT-972. Read in full in NOTE-786: §3 (Eq. 1, counting), §4.2–4.3 (Fig. 2 and its table), §6, Appendices A, D and G.
What was actually shown
- The measure.
- Following Hernandez et al., each relation’s computation from the subject’s middle-layer state s to the final state is approximated by F(s, c) ≈ W s + b, with W the mean Jacobian over 8 examples.
- Its quality is causality: the share of edits through the map that make the model output a chosen other object.
- Subject–object co-occurrences within training sequences were counted exactly in OLMo’s reconstructed Dolma batches, up to each of eight checkpoints from 41B to 2T tokens. For GPT-J they were counted per document in the Pile.
- The correlation.
- Across 24 relations, causality correlates with the log of mean subject–object co-occurrence at r = 0.82.
- Against subject frequency alone r = 0.66, and against object frequency alone r = 0.59.
- It is also higher than causality’s correlation with 5-shot accuracy (0.74, OLMo-7B).
- The thresholds.
- Mean causality exceeds 0.9 above a mean count of 1,097 (GPT-J), 1,998 (OLMo-7B) and 4,447 (OLMo-1B).
- Relations past the threshold are linear at early checkpoints as well as late ones: “regardless of pretraining step”.
- Rare relations. A rare relation can be recalled without a single consistent map. Star–constellation has 84% 5-shot accuracy but 44% causality, at about 21 co-occurrences.
- What could have come out otherwise. Linearity could have tracked training stage, or accuracy, rather than exposure. The checkpoint design separates exposure from stage, and the accuracy comparison separates it from recall.
What this does not say
- Not that frequency causes linearity. “We can not draw causal claims about how exposure affects individual representations” (Appendix A). Frequent facts may differ in other ways.
- Not about attribute directions. An LRE is one affine map per relation between two hidden states. It is not a direction for a binary attribute in an embedding. The result does not transfer to probes for attributes without a further test.
- Not a per-fact threshold. The counts are means over a relation’s pairs, and “consistently (but not exclusively)”: some relations below them are linear.
- Not a scaling law. Three models give three thresholds. The authors say they “cannot draw conclusions from only three models”.
- Not about the structure of co-occurrence. Only how often subjects and objects co-occur is measured. Whether the statistics are additive across attributes, correlated or hierarchical is not.
- Not an instruction. The authors’ suggestion to shape training data to meet frequency thresholds, and their tool for inferring training data, are ML practice. They are left to the Anthology of the SOTA.
Connections
- QUESTION-025. No
answerslink. The question’s premise is that co-occurrence structure gives linear attribute directions. This result is about relation maps and raw exposure. It bears on the question only by showing that linearity in trained models is graded and depends on counts. A measurement of the kind the question asks for would need to control for frequency, or a missing direction could just mean a rare attribute. - THEORY-185. That account explains linear directions by additive PMI and treats departures as noise that buries the attribute modes. A count threshold fits that picture loosely, since low counts mean noisy PMI. This paper does not test it.