nucleation

THEORY-1014: In a language model, a behaviour that looks like one mechanism, whether picked out by one contrast or by one cluster of next-token predictions, is often carried by several distinct pathways of sparse features that converge on the output without fully intersecting, as in Pythia-70M's agreement across a relative clause, its prediction of "to" before an infinitival object, and its incrementing of sequences

2 min read

Properties4
StatusProposed
Tagsrepresentation-learning, causality
SourceLIT-1532 — Sparse Feature Circuits: Discovering and Editing Interpretable Causal Graphs in Language Models
ExtendsTHEORY-1007 — On four subject–verb agreement tasks in Pythia-70M and Gemma-2-2B, circuits of sparse-autoencoder features and autoencoder error nodes, selected by linear estimates of indirect effect, carried half or more of the behaviour on held-out pairs with about 100 and 500 nodes, where circuits of neurons needed about 1,500 and more than 30,000; with the error nodes removed, the same feature circuits carried little of it, under about a quarter in Pythia and almost none in Gemma

Graph View

Source

Marks et al. (2025), LIT-1532, read in full in NOTE-1187: §3.3 and Figure 4, §5 and Figure 5, and Appendix C (Figures 7–18, from captions and text).

What is asserted

That behaviours are typically unions of mechanisms at the feature level, so that a behaviour’s “circuit” is an arrangement of pathways rather than one chain. The authors make the point of their two clusters, and their agreement circuits have the same shape.

What the record holds toward it

  • Agreement across a relative clause (Pythia-70M, 86 nodes; Gemma-2-2B, 223 nodes; faithfulness 0.21 each): a direct pathway from subject-number features to verb-form features, and a second that moves the number to the end of the clause through boundary detectors; Gemma adds noun-phrase number trackers.
  • “To” as an infinitival object (cluster 475, Pythia-70M): two mechanisms, one on verbs, participles and predicate adjectives that take infinitival objects, one on nominal objects before an object complement, which “both influence the output in layer 5 without fully intersecting”.
  • Incrementing sequences (cluster 382): general succession features in layer-3 attention and narrow induction features for specific tokens, composing into a layer-5 feature that increments “3” to “4”.
  • Against or qualifying. The circuits are small, chosen at thresholds that keep annotation manageable, and the agreement circuits recover only 0.21 of the behaviour. No pathway was ablated on its own. The annotations are the authors’.

What it would take

See promote_when.

Connections

  • CLAIM-454. If behaviours are unions of feature pathways, then what does the work is an organization of features, which is the claim’s thesis transposed to a network.
  • QUESTION-275. A developmental account would have to track pathways, not features: whether the two agreement pathways form together or one after the other.
  • THEORY-973. Induction at feature level comes in token-specific pieces composed with succession, where Olsson et al. found heads.

Backlinks