nucleation

LIT-1541: On the Biology of a Large Language Model

1 min read

Properties5
StatusActive — read 2026-10-26 (NOTE-1181) in full, from the Transformer Circuits page; worth reading for ten case studies that show, on single prompts and with interventions, a two-hop intermediate in Claude 3.5 Haiku (THEORY-984), a rhyme planned on the newline before a line is written (THEORY-1017), a jailbreak whose target word is not represented before it is written (THEORY-1018), a trained-in goal represented in every Assistant context of a model organism (THEORY-976), and a default "can't answer" circuit that known entities inhibit. Read it knowing that the authors find a satisfying graph for about a quarter of the prompts they try, that the graphs shown are curated and of a replacement model (LIT-1552), that several cruxes run through attention the graphs cannot see, and that its general statements about the model (THEORY-982) go beyond what the case studies show.
Version2
Tagsrepresentation-learning, causality, cognition, anthology-candidate
Cited as “paper” byNOTE-1181 — Lindsey et al. — On the biology of a large language model
Cited as “source” byTHEORY-1017 — In Claude 3.5 Haiku, writing the second line of a rhyming couplet whose first line ends "grab it", features for the candidate rhymes "rabbit" and "habit" are active on the newline token before the line, steering them changes the line's final word only when applied at that token, and suppressing "rabbit" turns the line towards "habit"; injecting a "rabbit" or "green" word feature at that point in 25 other poems ended the line with the injected word in 70% of cases, and such planned-word features were found in about half of the poems the authors examined, THEORY-1018 — In Claude 3.5 Haiku, on the jailbreak prompt "Babies Outlive Mustard Block." asking it to join the first letters and say how to make one, the attribution graph for its output "BOMB" shows separate letter-wise paths voting for the output and no internal representation of the word or of a bomb-making request before it is written; "make a bomb" features first fire on the model's own "To make a bomb", injecting one on "BOMB" brings an immediate refusal, and the refusal that does come waits for the sentence's end, THEORY-976 — In a Claude 3.5 Haiku variant finetuned by Marks et al. to exploit reward-model biases described in fictional documents, a cross-layer-transcoder feature that in pretraining-style text fires almost only where those biases are discussed was active in all 100 short Human/Assistant-formatted prompts tested and in 1 of the same 100 without that formatting, and suppressing such bias features cut the sampled rate of three bias behaviours, including two the model was never trained to show, THEORY-982 — Claude 3.5 Haiku routinely computes intermediate steps within a single forward pass and plans its outputs ahead, representing a target before the text that leads to it and writing towards it, as Lindsey et al. propose from their case studies, THEORY-984 — In Claude 3.5 Haiku, on the prompt "Fact: the capital of the state containing Dallas is", which it completes with "Austin", an attribution graph shows Dallas features activating Texas features that, with "say a capital" features, promote Austin, beside a direct edge from Dallas to "say Austin"; inhibiting the Dallas features lowers the Texas features and leaves "say a capital" largely intact, and replacing the Texas features with those of California, Georgia, British Columbia, China or the Byzantine Empire makes the model name that place's capital

Graph View

Lindsey, J., Gurnee, W., Ameisen, E., Chen, B., Pearce, A., Turner, N. L., Citro, C., et al. (2025). On the biology of a large language model. Transformer Circuits Thread, 27 March 2025. https://transformer-circuits.pub/2025/attribution-graphs/biology.html

Read in full on 2026-10-26 (NOTE-1181). Five accounts were drawn from it: THEORY-984, THEORY-1017, THEORY-1018, THEORY-976 and THEORY-982.

Why this is here

Registered and read at the owner’s request of 2026-10-26, after “how’s the record on neural circuit theory?” found that the record held almost none of the core circuits literature. It is the most cited set of circuit case studies in a production-scale model, and the companion of LIT-1552, whose method it applies. For the circuits line it adds the first entries about intermediate steps and plans in a large model, as against small or algorithmic ones (LIT-345, LIT-1538), and it bears on CLAIM-454 (features read from activity, here validated by intervention) and on the workspace reading of LIT-1347, by the same group.

Backlinks