nucleation

LIT-1555: Crosscoding Through Time: Tracking Emergence & Consolidation Of Linguistic Representations Throughout LLM Pretraining

1 min read

Properties5
StatusActive — read 2026-10-26 (NOTE-1186) in full, from the arXiv PDF of v2 (30 April 2026); worth reading for the method of tracking features across pretraining checkpoints with one joint dictionary, and for separating a feature's presence (decoder norm) from its causal use (RelIE). Read it knowing that its developmental trends rest on the authors' annotation of a few top features (THEORY-1004, Proposed), that it covers one layer and two or three checkpoints at a time, and that a feature shared across checkpoints is shared by construction of the dictionary, not shown to be carried forward.
Version2
Tagsrepresentation-learning, linguistics, anthology-candidate
Cited as “paper” byNOTE-1186 — Bayazit, Mueller and Bosselut — Crosscoding through time
Cited as “source” byTHEORY-1004 — In Pythia-1B, OLMo-1B and BLOOM-1B, the crosscoder features that most affect subject–verb agreement change over pretraining: at early checkpoints they detect particular tokens and subtokens, at later ones grammatical categories; in BLOOM, language-specific features give way to features shared across English, French, Spanish and Portuguese while Hindi and Arabic keep language-specific ones; and which features matter keeps shifting after task accuracy has stopped improving

Graph View

Bayazit, D., Mueller, A. & Bosselut, A. (2025). Crosscoding through time: Tracking emergence & consolidation of linguistic representations throughout LLM pretraining. ACL 2026. arXiv:2509.05291.

Read in full on 2026-10-26 (NOTE-1186). The summary is checked against the reading.

Why this is here

Cited at T24 as the methods and baselines for tracking which features appear, persist and disappear over training, the comparator the reply’s dynamic latent-feature model must beat (“The dynamic IBP must earn its complexity”). The record had no crosscoder work. It bears on QUESTION-275 and CASE-165.

Backlinks