nucleation

THEORY-984: In Claude 3.5 Haiku, on the prompt "Fact: the capital of the state containing Dallas is", which it completes with "Austin", an attribution graph shows Dallas features activating Texas features that, with "say a capital" features, promote Austin, beside a direct edge from Dallas to "say Austin"; inhibiting the Dallas features lowers the Texas features and leaves "say a capital" largely intact, and replacing the Texas features with those of California, Georgia, British Columbia, China or the Byzantine Empire makes the model name that place's capital

2 min read

Properties4
StatusActive
Tagsrepresentation-learning, cognition
SourceLIT-1541 — On the Biology of a Large Language Model
Extended byTHEORY-982 — Claude 3.5 Haiku routinely computes intermediate steps within a single forward pass and plans its outputs ahead, representing a target before the text that leads to it and writing towards it, as Lindsey et al. propose from their case studies

Graph View

Source

Lindsey et al. (2025), On the Biology of a Large Language Model, Transformer Circuits Thread, LIT-1541, read in NOTE-1181: the section “Introductory Example: Multi-step Reasoning”, with “Validation with Inhibition Experiments” and “Swapping Alternative Features”. The method is LIT-1552.

What was actually shown

The graph. With a 30M-feature cross-layer transcoder, the pruned graph for “Austin” contains supernodes the authors labelled from feature visualizations: Dallas, Texas (Texas-related features “not specific to a particular city”), capital (including multilingual capital features), say a capital, and say Austin. Paths: Dallas → Texas; capital → say a capital; Texas and say a capital → Austin, directly and via say Austin; and Dallas → say Austin directly.

Inhibition. Clamping each supernode to a negative multiple of its activation in the model: inhibiting Dallas lowers Texas and say Austin and leaves say a capital largely unaffected, and the model outputs other state capitals; inhibiting capital lowers say a capital and say Austin and leaves Texas largely unchanged, and the model gives non-capital completions.

Swaps. Texas features inhibited and features from an analogous prompt injected: from “Oakland”, the model outputs Sacramento; from “Savannah”, Atlanta; from “Vancouver” (province), Victoria; from “Shanghai” (country), Beijing; from “Thessaloniki” (empire), Constantinople. The non-state swaps needed larger injections.

What could have come out otherwise. The graph could have shown only the shortcut, with no Texas step. Inhibiting Dallas could have left Texas intact. Swapping the Texas features could have left “Austin” in place, which is what a completion carried by the shortcut alone would do.

What this does not say

  • Not that the model usually reasons in two steps. One prompt and its analogues, chosen from the prompts where the method gave a clear graph.
  • Not that the shortcut is negligible. Both paths are present; the article does not measure their shares of the output.
  • Not that “Texas” is a single representation. It is a hand-made group of several features, each with other facets (one is about law and government in Texas).
  • Not about the model’s attention. Why the “Dallas” information reaches the final position is not shown.

Backlinks