nucleation

THEORY-1005: On one MLP layer each of GPT2-small, Pythia-410M and Pythia-1.4B, transcoders trained to imitate the MLP lay on or above the sparsity–loss frontier of sparse autoencoders trained on its output, at one learning rate tuned on transcoders and one seed, and in a single rater's blind reading of 50 random features each from Pythia-410M layer 15, 41 transcoder and 38 SAE features were judged interpretable

2 min read

Properties3
StatusActive
Tagsrepresentation-learning
SourceLIT-1554 — Transcoders Find Interpretable LLM Feature Circuits

Graph View

Source

Dunefsky, Chlenski and Nanda (2024), arXiv:2406.11944, LIT-1554, read in full in NOTE-1174: §4.1–4.2, Table 1, Fig. 3, Apps. E and F.

What was actually shown

The design. For GPT2-small layer 8, Pythia-410M layer 15 and Pythia-1.4B layer 15, several transcoders (MLP input → MLP output) and several SAEs (on MLP output) were trained at different λ₁, all 32 × d_model wide, on 60M OpenWebText tokens, with Adam at 2 × 10⁻⁵ and seed 42. Each was spliced into the model in place of the MLP output and scored by next-token cross-entropy on the same 3.2M tokens, against mean L0. The unmodified model and the mean-ablated MLP bound the range.

The result. “In all cases, transcoders are equal to or better than SAEs. In fact, the gap between transcoders and SAEs seems to widen on larger models.” (The record read this from the text and the caption; Figure 3’s points were not legible in the extracted text.)

Interpretability. A transcoder (L0 44.04, 3.35 nats) and an SAE on MLP inputs (L0 47.85, 3.36 nats) at nearly the same point of the frontier: 50 random features each, shuffled, judged from top-activating examples before their source was revealed. Transcoder 41 interpretable, 8 maybe, 1 not; SAE 38, 8, 4. The counts sum to 50 each.

What could have come out otherwise. SAEs could have dominated the frontier, as Gemma Scope later found on Gemma 2 2B; the blind counts could have favoured the SAE.

What this does not say

  • Not a general ranking. The authors: the shared learning rate “might not be optimal for SAEs”, and “a different set of hyperparameters could have allowed SAEs to surpass transcoders”. One seed, one layer per model.
  • Not for other models. On Gemma 2 2B the order reverses (LIT-611, NOTE-482), for reasons not established.
  • Not an interpretability measurement in any strong sense. One rater, one layer, one model, top-activating examples only; the three-point gap in “interpretable” is within what a second rater could move.
  • Not about circuits. Fidelity here is the loss with one MLP replaced; it says nothing about how faithful the circuits read off the transcoder are.

Backlinks