nucleation

LIT-1545: Towards Monosemanticity: Decomposing Language Models With Dictionary Learning

1 min read

Properties5
StatusActive — read 2026-10-26 (NOTE-1191) in full, from the article's own page on the Transformer Circuits Thread; worth reading as the paper that made sparse-autoencoder features the field's unit of analysis below the neuron, for its case that a feature is a causal unit and not only a pattern in the data (downstream effects the autoencoder never saw), and for feature splitting, which makes the feature set a resolution chosen by dictionary size (THEORY-1016). Read it knowing that everything is one 512-neuron MLP layer of a one-layer model, that the four deep-dive features are cherry-picked by the authors' own account, that the human ratings are one author's, and that the 79% of loss recovered is measured against zero-ablation, a baseline the authors call an overestimate (THEORY-991).
Version2
Tagsrepresentation-learning, anthology-candidate
Cited as “paper” byNOTE-1191 — Bricken, Templeton, Batson, Chen, Jermyn et al. — Towards monosemanticity
Cited as “source” byTHEORY-1016 — The features a sparse autoencoder recovers from a language model are a resolution set by the dictionary's width and sparsity penalty, not a unique set of the model's units: as the dictionary widens, features split into families of finer features, and splits also merge, so that the number of features is not well-posed, THEORY-965 — Sparse-autoencoder features of a large language model's residual stream are units the model computes with: they track abstract concepts across languages, modalities and concrete or abstract instances, and intervening on one changes the model's behaviour in the way its label predicts, better than directions found from a few labelled examples, THEORY-991 — In a one-layer transformer whose MLP has 512 ReLU neurons, a sparse autoencoder with 4,096 features trained on that layer's activations yields features that a blinded rater and an automated pipeline judged far more interpretable than the neurons, that for the four features examined in detail correspond to no neuron, and whose effects on the output, which the autoencoder never saw, agree with their activations; splicing in the reconstruction recovers 79% of the loss reduction the MLP provides, measured against zero-ablating it

Graph View

Bricken, T., Templeton, A., Batson, J., Chen, B., Jermyn, A., et al. (2023). Towards monosemanticity: Decomposing language models with dictionary learning. Transformer Circuits Thread, 4 October 2023. https://transformer-circuits.pub/2023/monosemantic-features/index.html

Read in full on 2026-10-26 (NOTE-1191). Two accounts were drawn from it: THEORY-991 and THEORY-1016.

Why this is here

Registered on 2026-10-26 at the owner’s request, when the record was found to hold almost none of the core circuits literature. The record had the superposition hypothesis (LIT-323) and a benchmark that tests sparse-autoencoder features as causal units (LIT-1550, THEORY-968), but not the paper that proposed those features as the unit below the neuron. It fills that gap. It bears on CLAIM-454 (a recovered feature is at most a constituent), on THEORY-018 and THEORY-017 (a basis of features is the analyst’s choice), and on QUESTION-275.

Backlinks