Source
Templeton, Conerly, Marcus, Lindsey et al. (2024), Scaling monosemanticity, LIT-1533, read in full in NOTE-1178: Our SAE experiments (dead-feature fractions) and Feature Completeness.
What was actually shown
The procedure. For each concept, a prompt naming it is run and the five features most active on its last token are explained by Sonnet; a human judges whether an explanation specifically names the concept. The concept’s frequency is counted in a proxy of the training data by searching for the name between spaces, with false positives the authors note (the element “lead”).
The dictionaries. About 1M, 4M and 34M features, with about 2%, 35% and 65% dead, so roughly 1.0M, 2.7M and 11.7M live (computed here from the stated fractions; the paper says “about 12M” for the largest).
The result. For each dictionary and category, the share of concepts with a feature rises with frequency, and the point where it passes one half is “consistently slightly lower than the inverse of the number of alive features”. Rescaling frequency by the live-feature count makes the curves roughly coincide, a sigmoid in log frequency, with a similar threshold across categories. Separately, the 34M dictionary had features for about 60% of London boroughs.
What could have come out otherwise. Coverage could have depended on the category (places before foods), or on salience rather than frequency, or grown with size along curves that do not coincide.
What this does not say
- Not that a concept without a feature is absent from the model. The authors say the model “can use multiple related features compositionally to reference a specific concept”, and the model names every London borough.
- Not which frequency matters. The autoencoder’s training data resembles the model’s pre-training data, so the two frequencies are not separated.
- Not the extrapolation. That a once-in-a-billion concept needs about a billion live features is the authors’ projection.
- Not a measured detection error. Presence is judged from Sonnet’s explanations of the top five features by one human rule; a feature outside the top five, or mislabelled, would be missed.
Connections
- THEORY-1016. Which concepts are units depends on dictionary size: the measured part of the claim that the feature set is a resolution.
- THEORY-017. A decomposition whose parts appear as the dictionary grows is one fixed by the analyst’s budget, not by the model alone.
- CLAIM-454. A concept the model uses with no feature of its own is held by a combination of features, the claim’s case of something carried by an arrangement and not by a single recovered unit.