THEORY-991: In a one-layer transformer whose MLP has 512 ReLU neurons, a sparse autoencoder with 4,096 features trained on that layer's activations yields features that a blinded rater and an automated pipeline judged far more interpretable than the neurons, that for the four features examined in detail correspond to no neuron, and whose effects on the output, which the autoencoder never saw, agree with their activations; splicing in the reconstruction recovers 79% of the loss reduction the MLP provides, measured against zero-ablating it

Source

Bricken, Templeton, Batson, Chen, Jermyn et al. (2023), Towards monosemanticity, LIT-1545, read in full in NOTE-1191 from the article’s HTML: the detailed investigations of individual features, Global Analysis, and the appendices on proxies, ablations, the rubric and automated interpretability.

What was actually shown

The setting. A transformer with one attention block and one MLP of 512 ReLU neurons (residual stream 128), trained on 100 billion Pile tokens. The autoencoder A/1 has 4,096 features, of which 3,636 are live and not “ultralow density”; it was trained on 8 billion MLP activations with an MSE and L1 loss, seeing nothing downstream of the MLP.

Interpretability. One author, blinded, scored 412 activation intervals over 162 features and neurons on a 0–14 rubric: median 0 for neurons (“could not even form a hypothesis”), 12 for feature intervals. Claude 2’s explanations predicted held-out activations better for features than for neurons, and predicted held-out top logit tokens at 74% for features and 58% for neurons, against 50% chance.

Not neurons. The Arabic-script feature’s most correlated neuron responds to many languages, and its three largest neuron coefficients are negative. No neuron has Hebrew among its top examples. The base64 feature’s best-correlated neuron reaches 0.18.

Effects agree with activations. Logit weights, ablation of the feature over a context, and sampling with the feature pinned high all point the way its activations do, for the four detailed features and, by the authors’ report, in the feature browser for all. On a transformer with shuffled weights the autoencoder finds mostly single-token features whose other features the authors could not interpret.

Loss. Replacing the MLP’s activations with A/1’s reconstruction costs 21% of the loss that zero-ablating the MLP would. The authors call this baseline “especially bad” and the percentage an overestimate.

What could have come out otherwise. The features could have been rotated copies of neurons, or patterns of the data with no matching effects on the output. The shuffled-weights control shows that the second outcome happens when the model has not learned anything.

What this does not say

• Not about deeper models or other sites. One MLP layer whose effect on the logits is nearly linear; the authors say this makes linear features likelier here.
• Not that all features are interpretable. Lower activation intervals are less consistent, and the detailed cases are chosen to be easy.
• Not that the model is composed of these features. The authors are “agnostic” on that, and the feature set depends on width and penalty (THEORY-1016).
• Not that features carry known causal variables better than neurons. That is a different test, and in MIB they did worse (THEORY-968).

Connections

• CLAIM-454. The paper itself treats activation structure and role as separate evidence, and a control shows activation structure arising from data alone. A supporting case of the claim’s distinction, not a test of it.
• LIT-323. The empirical counterpart of the superposition hypothesis’s prediction that more features than neurons are stored.
• THEORY-968. Compatible: better interpretability and worse localization of supplied variables are both measured, by different tests.