nucleation

THEORY-1016: The features a sparse autoencoder recovers from a language model are a resolution set by the dictionary's width and sparsity penalty, not a unique set of the model's units: as the dictionary widens, features split into families of finer features, and splits also merge, so that the number of features is not well-posed

2 min read

Properties4
StatusProposed
Tagsrepresentation-learning
SourceLIT-1545 — Towards Monosemanticity: Decomposing Language Models With Dictionary Learning, LIT-1533 — Scaling Monosemanticity: Extracting Interpretable Features from Claude 3 Sonnet
ExtendsTHEORY-991 — In a one-layer transformer whose MLP has 512 ReLU neurons, a sparse autoencoder with 4,096 features trained on that layer's activations yields features that a blinded rater and an automated pipeline judged far more interpretable than the neurons, that for the four features examined in detail correspond to no neuron, and whose effects on the output, which the autoencoder never saw, agree with their activations; splicing in the reconstruction recovers 79% of the loss reduction the MLP provides, measured against zero-ablating it

Graph View

Source

  • Bricken et al. (2023), LIT-1545, read in full in NOTE-1191: Feature Splitting (the mathematics and physics example, the two “bugs”), Universality, and the Discussion’s “Theories of Superposition”.
  • Templeton et al. (2024), LIT-1533, read in full in NOTE-1178: Exploring Feature Neighborhoods and Feature Completeness.

What is asserted

That a sparse autoencoder’s features are not the model’s units read off once, but a decomposition at a granularity the analyst sets by width and penalty. In Bricken et al.’s words: “the features we find at one level may both split and merge to form refined features at the next”, “determining the ‘correct number of features’ for dictionary learning is less important than it might initially seem”, and in some theories “finding the ‘correct number of features’ may not be well-posed”.

What the record holds toward it

  • Observed splitting. In the one-layer model, base64 is one feature at 512, three at 4,096 (letters, digits, base64 encoding ASCII) and more at 16,384; the three cover the one. In Claude 3 Sonnet a San Francisco feature at 1M is two at 4M and eleven at 34M, and some 4M and 34M features (earthquakes) have no 1M analogue.
  • Splits that merge. The mathematics and physics example is “more complex than a tree”.
  • Coverage tracks width. In Claude 3 Sonnet a concept gets its own feature when its frequency passes about one over the number of live features (THEORY-992), so which concepts are units depends on dictionary size.

What it would take

See promote_when. The record holds examples read from feature browsers and neighbourhood plots, with a cosine or activation-similarity threshold chosen to make a graph legible. No measurement says what fraction of features split, merge or persist across widths.

Connections

  • THEORY-017. Its thesis, that parts and features are fixed by an extrinsic choice, here in a measured case of a learned network: the dictionary’s width and penalty are the choice.
  • THEORY-018. The softmax leaves the final layer free up to an invertible map; this account adds that even with the basis fixed by sparsity, the number of units is not.
  • CLAIM-454. If the features are a resolution, then whether a perspective or regime “coincides with one feature” depends on width as well as on the model.
  • QUESTION-275. A feature model fitted across checkpoints inherits this: a feature that appears to split during training may be the dictionary’s resolution moving, not the network’s.

Backlinks