nucleation

THEORY-965: Sparse-autoencoder features of a large language model's residual stream are units the model computes with: they track abstract concepts across languages, modalities and concrete or abstract instances, and intervening on one changes the model's behaviour in the way its label predicts, better than directions found from a few labelled examples

2 min read

Properties4
StatusProposed
Tagsrepresentation-learning, causality
SourceLIT-1533 — Scaling Monosemanticity: Extracting Interpretable Features from Claude 3 Sonnet, LIT-1545 — Towards Monosemanticity: Decomposing Language Models With Dictionary Learning
ExtendsTHEORY-987 — In Claude 3 Sonnet's middle-layer residual stream, on two prompts for which every active sparse-autoencoder feature was ablated, how strongly a feature was active predicted its ablation effect on the answer poorly (correlation about 0.12) while its gradient attribution predicted it well (about 0.81), and on the multi-step prompt only three of the ten most active features were among the ten with the largest ablation effect, THEORY-992 — In sparse autoencoders of about 1M, 4M and 34M features on Claude 3 Sonnet's middle-layer residual stream, whether a concept among chemical elements, cities, animals and foods had a dedicated feature depended on how often it occurs in the training data: the frequency at which a concept became more than 50% likely to have one was slightly below one over the number of live features, and the three dictionaries' curves roughly coincided when frequency was rescaled by that number

Graph View

Source

  • Templeton et al. (2024), LIT-1533, read in full in NOTE-1178: Assessing Feature Interpretability, Safety-Relevant Features, Comparison to Other Approaches, and the steering appendices.
  • Bricken et al. (2023), LIT-1545, read in full in NOTE-1191, for the one-layer version of the thesis (THEORY-991).

What is asserted

That the features are not only a description of the activations but the model’s working concepts. “We find this compelling evidence that our interpretations of features line up with how they are used by the model.” And that, in the few-shot regime, they are better handles than probe directions.

What the record holds toward it

  • For. Examples of abstract features (the Golden Gate Bridge feature across languages and in images; a code-error feature across programming languages; an addition feature that follows function composition) and steering in the labelled direction, for features the authors chose. 82% of a random sample match no neuron. In the one-layer model, effects on the output agree with activations (THEORY-991). Few-shot difference-of-means vectors steered as well in two of seven hand-picked cases and failed in five.
  • Against or qualifying. Steering typically needs a clamp beyond the feature’s observed range, which the authors attribute to correlated features acting together. The most active features are mostly not those that carry the answer (THEORY-987). In MIB, features of public autoencoders on Gemma-2 and Llama-3.1 residual streams localized known causal variables worse than the model’s own dimensions in all sixteen cells (THEORY-968). Residual-stream autoencoders are the site where Gemma Scope’s cost the most loss (THEORY-504).

What it would take

See promote_when.

Connections

  • CLAIM-454. The thesis is the strong reading the claim denies for organizations: that a recovered feature is the variable that does the work. Its evidence is examples of a feature moving behaviour when pushed hard, which the claim allows of a constituent.
  • THEORY-968. The present test against it.
  • THEORY-1016. If the features are a resolution, the thesis needs a statement of which resolution’s features are the units.

Backlinks