nucleation

LIT-323: Toy Models of Superposition

2 min read

Properties6
StatusActive — read in full 2026-09-29 (NOTE-278); worth reading as the founding statement of the superposition hypothesis and of the vocabulary the field now uses: features as directions, privileged versus non-privileged bases, polysemanticity as interference, and feature dimensionality. It is also the cleanest demonstration that a nonlinearity plus sparsity makes it optimal to store more features than dimensions. Its evidence is toy models alone, which the paper says plainly. Its phase-change and polytope results are exact only for n = 2, m = 1 and uniform features.
Version3
Tagsrepresentation-learning, anthology-candidate
Cited as “extends” byLIT-1625 — Polysemanticity and Capacity in Neural Networks
Cited as “paper” byNOTE-278 — Toy Models of Superposition
Cited as “source” byTHEORY-018 — Softmax training identifies a language model's final-layer representation only up to an invertible linear map, so no inner product on it is intrinsic, and Park, Choe and Veitch's causal inner product is fixed by a stipulation, THEORY-021 — In the exact configurations of uniform superposition (regular polygons, the tetrahedron, antipodal pairs, and tegum products of these that span the hidden space, with equal norms within each factor) the features form a rank-one POVM compressed from the n-feature basis: their dimensionality-weighted projectors sum to the identity

Graph View

Nelson Elhage, Tristan Hume, Catherine Olsson, Nicholas Schiefer, Tom Henighan, Shauna Kravec, Zac Hatfield-Dodds, Robert Lasenby, Dawn Drain, Carol Chen, Roger Grosse, Sam McCandlish, Jared Kaplan, Dario Amodei, Martin Wattenberg, Christopher Olah (2022), Transformer Circuits Thread (Anthropic); arXiv 2022-09-21 — ARXIV-2209.10652

Standing in the record

Registered on 2026-09-29 at the owner’s request, because the owner’s prior-art novelty map (a private working document on concepts as projectors, the linear representation hypothesis and their neighbours) cites it: §4 must-cite (linear representation line, alongside Park 2023): features as (non-orthogonal) directions. It is seeded from metadata and published descriptions and has not been read here.

Identification notes: Not held in either record (anthology holds only Chen et al. 2023 on a toy model of superposition, ANTH-LIT-544). Author list per the Transformer Circuits page (16 authors; arXiv lists the same). Published = Transformer Circuits date (Sept 14, 2022); arXiv v1 is 2022-09-21.

It was filed Deferred, unread. NOTE-278 is the close reading of 2026-09-29, and it placed the work: Active — worth reading as the founding statement of the superposition hypothesis and of the vocabulary the field now uses: features as directions, privileged versus non-privileged bases, polysemanticity as interference, and feature dimensionality. It is also the cleanest demonstration that a nonlinearity plus sparsity makes it optimal to store more features than dimensions. Its evidence is toy models alone, which the paper says plainly. Its phase-change and polytope results are exact only for n = 2, m = 1 and uniform features.

Note of 2026-10-27: its first case and its sequels, now held

The core circuits works, registered and read on 2026-10-26 at the owner’s request, place this paper in its line.

  • The first case called superposition is older than this paper. Zoom In (LIT-1543, NOTE-1182) shows InceptionV1 building a car feature and spreading it over many dog neurons, and calls that superposition, the phenomenon this paper models in toy settings.
  • Its empirical sequels. Towards Monosemanticity (LIT-1545, NOTE-1191) takes the dictionary-learning strategy this paper named to a one-layer transformer, and Scaling Monosemanticity (LIT-1533, NOTE-1178) takes it to a production model.
  • A correction by the same team. Towards Monosemanticity’s Discussion says this paper’s isotropic picture of superposition was “missing something crucial”: real features clump, by correlated activation and by similar “output actions”.

Status and summary are unchanged.

Backlinks