nucleation

THEORY-1004: In Pythia-1B, OLMo-1B and BLOOM-1B, the crosscoder features that most affect subject–verb agreement change over pretraining: at early checkpoints they detect particular tokens and subtokens, at later ones grammatical categories; in BLOOM, language-specific features give way to features shared across English, French, Spanish and Portuguese while Hindi and Arabic keep language-specific ones; and which features matter keeps shifting after task accuracy has stopped improving

2 min read

Properties3
StatusProposed
Tagsrepresentation-learning, linguistics
SourceLIT-1555 — Crosscoding Through Time: Tracking Emergence & Consolidation Of Linguistic Representations Throughout LLM Pretraining

Graph View

Source

Bayazit, Mueller and Bosselut (2025), LIT-1555, read in full in NOTE-1186 from arXiv v2: §§5–8, Tables 1–2 and 6–7, Figs. 2–5, and appendices D–G.

What is asserted

That in these three model families, at the middle layer, the features a model uses for subject–verb agreement are replaced over pretraining by more general ones, in two directions: from particular tokens to grammatical categories, and, in a multilingual model, from one language to several with similar agreement systems. And that the replacement continues after the task is learned.

What the paper shows toward it

  • Measured. Crosscoders reconstruct each checkpoint’s activations with a small rise in loss (Table 6). RelIE agrees with single-feature ablations on the top-10 features (ρ about 0.95). In Pythia’s three-way analysis, many of the top-100 features are specific to the 4B or the 286B checkpoint although accuracy is the same at both (Fig. 4, no counts). In BLOOM the overlap of top-10 features across the four Latin-script languages rises by 341B, and Hindi and Arabic overlap little (Fig. 5).
  • Annotated. That the early features are token-specific and the late ones categorical is the authors’ description of top-activating sequences for a few features per checkpoint (Tables 1, 2 and the appendix tables), done blind to RelIE but by the authors.

Why it stays Proposed

The central trend is an annotation. The post-plateau shift has no null: two crosscoders, or two nearby checkpoints, might also disagree on which features matter. And the analysis is one layer and a handful of checkpoints per model.

What this does not say

  • Not that a later feature descends from an earlier one. Shared features share an index in a jointly trained dictionary; that is resemblance, which CLAIM-329 distinguishes from continuity.
  • Not a mechanism. Features are scored one at a time; how they combine into a circuit is the authors’ future work.
  • Not a prediction. Checkpoints are chosen after the transitions are seen.

Connections

  • QUESTION-275. The feature-only description the question’s coupled model has to beat.
  • THEORY-279. Low-level structure first, at the level of features in large language models, by annotation.
  • CASE-165. Presence (decoder norm) and use (indirect effect) are measured separately here, which the case’s model could not do from its outputs.
  • LIT-1532. The indirect effects and their integrated-gradient approximation come from Marks et al.’s sparse feature circuits, which ran them on the same subject–verb agreement tasks in one checkpoint (THEORY-1007).

Backlinks