nucleation

LIT-1625: Polysemanticity and Capacity in Neural Networks

1 min read

Properties6
StatusActive — read 2026-10-27 (NOTE-1205) in full, from arXiv v4; worth reading as the source of the toy model the causal scrubbing appendix (LIT-1583, §6) analyses, and as an exactly solvable case of superposition: in a one-hidden-layer model with quadratic activations, the expected loss depends on the weights only through each feature's capacity and embedding length, and the optimal allocation has a closed form in which features are ignored, shared or given a whole dimension according to their importance and the inputs' fourth moment (THEORY-1036, re-derived and checked numerically). Read it knowing that its other models are shown only as phase-diagram figures, and that its bearing on trained networks is the authors' hope, not a result (THEORY-1035).
Version2
Tagsrepresentation-learning, anthology-candidate
ExtendsLIT-323 — Toy Models of Superposition
Cited as “paper” byNOTE-1205 — Scherlis et al. — Polysemanticity and capacity in neural networks
Cited as “source” byTHEORY-1035 — In trained neural networks, polysemantic neurons arise from the loss-optimal allocation of a layer's limited capacity among features: a feature is shared between neurons when its marginal loss reduction from capacity equals that of the features it shares with, so that sparser, higher-kurtosis inputs, which make returns to capacity diminish, produce more polysemanticity, and the block structure of the embedding, which architecture shapes, sets how many features each neuron mixes, as Scherlis, Sachan, Jermyn, Benton and Shlegeris propose, THEORY-1036 — In a model that fits y = Σ v_i x_i² with ỹ = Σ_r (W_r·x)² + b, on independent inputs of zero mean, unit variance and common fourth moment k, the expected squared error at the optimal bias is (k − 1)Σ(n_i − v_i)² − 2Σn_i² + 2Σn_i²/C_i, in the squared embedding lengths n_i and capacities C_i = (w_i·w_i)²/Σ_j(w_i·w_j)², and its minimum with the capacities summing to the number of neurons D gives C_i = clip((k − 1)v_i/((k − 3)λ) − 2/(k − 3), 0, 1): for k ≤ 3 each feature is given a whole dimension or none, and for k > 3 a feature is shared between neurons exactly when 2λ/(k − 1) < v_i < λ

Graph View

Scherlis, A., Sachan, K., Jermyn, A. S., Benton, J. and Shlegeris, B. (2022). Polysemanticity and capacity in neural networks. arXiv:2210.01892 (v1 4 October 2022; v4 25 March 2025). Announced on the Alignment Forum, 7 October 2022.

Read in full on 2026-10-27 (NOTE-1205). Two accounts were drawn from it: THEORY-1036 (Active) and THEORY-1035 (Proposed).

Why this is here

Registered and read at the owner’s request of 2026-10-27 (“file the next readings”), as one of the works the circuits readings named and did not register. The causal scrubbing appendix (LIT-1583, NOTE-1202) shows how a hypothesis handles superposition on this paper’s toy model (NOTE-1202’s claim C4), and §6.1’s point about splitting the interference term is the mechanism THEORY-1029 finds in a real model. The record held the descriptive study it builds on, Toy Models of Superposition (LIT-323), but no model of superposition solved in closed form. Read against the circuits line, it supplies the case the appendix assumes: when two features of equal importance share one neuron, and why sparse inputs make them share.

Backlinks