nucleation

LIT-1554: Transcoders Find Interpretable LLM Feature Circuits

1 min read

Properties6
StatusActive — read 2026-10-26 (NOTE-1174) in full, from the arXiv copy (v2, the NeurIPS 2024 version); worth reading as the source of transcoder circuit analysis, for the factorization of a feature-to-feature attribution into an input-dependent activation times an input-invariant weight (THEORY-974), on which the attribution graphs of LIT-1552 are built, and for its blind case studies, which grade the method against hidden prompts and report the failures. Read it knowing that the comparison with SAEs is one learning rate, one seed and three models (THEORY-1005), that Gemma Scope found the reverse on Gemma 2 2B (NOTE-482), that attention patterns are frozen and QK circuits are not explained, and that its counts of case studies disagree between sections.
Version2
Tagsrepresentation-learning, causality, anthology-candidate
Extended byLIT-1552 — Circuit Tracing: Revealing Computational Graphs in Language Models
Cited as “paper” byNOTE-1174 — Dunefsky, Chlenski and Nanda — Transcoders find interpretable LLM feature circuits
Cited as “source” byTHEORY-1005 — On one MLP layer each of GPT2-small, Pythia-410M and Pythia-1.4B, transcoders trained to imitate the MLP lay on or above the sparsity–loss frontier of sparse autoencoders trained on its output, at one learning rate tuned on transcoders and one seed, and in a single rater's blind reading of 50 random features each from Pythia-410M layer 15, 41 transcoder and 38 SAE features were judged interpretable, THEORY-974 — When a transformer's MLP sublayers are replaced by transcoders and the attention patterns and LayerNorm scales on a prompt are held fixed, the contribution of an earlier transcoder feature at the same token to a later one's pre-activation is the earlier feature's activation times the dot product of its decoder with the later feature's encoder, scaled by one LayerNorm constant per token, so the weight factor is fixed by the trained weights and holds on every input

Graph View

Dunefsky, J., Chlenski, P., & Nanda, N. (2024). Transcoders find interpretable LLM feature circuits. NeurIPS 2024. arXiv:2406.11944.

Read in full on 2026-10-26 (NOTE-1174). Two accounts were drawn from it: THEORY-974 and THEORY-1005.

Why this is here

Registered and read at the owner’s request of 2026-10-26, after the question “how’s the record on neural circuit theory?” found that the record held almost none of the core circuits literature. The record’s circuits line had reverse-engineered circuits in small or algorithmic models (LIT-345, LIT-1538), sparse dictionaries as units (LIT-323, LIT-611) and a benchmark for localization (LIT-1550), but no method for tracing a circuit through MLP layers in terms of features. This is that method’s source, and the direct ancestor of the attribution graphs of LIT-1552. It bears on CLAIM-454 (a feature recovered from activity is not yet the variable that does the work) and on QUESTION-275 (what a circuit-only description over checkpoints could be).

Backlinks