nucleation

THEORY-1035: In trained neural networks, polysemantic neurons arise from the loss-optimal allocation of a layer's limited capacity among features: a feature is shared between neurons when its marginal loss reduction from capacity equals that of the features it shares with, so that sparser, higher-kurtosis inputs, which make returns to capacity diminish, produce more polysemanticity, and the block structure of the embedding, which architecture shapes, sets how many features each neuron mixes, as Scherlis, Sachan, Jermyn, Benton and Shlegeris propose

2 min read

Properties4
StatusProposed
Tagsrepresentation-learning
SourceLIT-1625 — Polysemanticity and Capacity in Neural Networks
ExtendsTHEORY-1036 — In a model that fits y = Σ v_i x_i² with ỹ = Σ_r (W_r·x)² + b, on independent inputs of zero mean, unit variance and common fourth moment k, the expected squared error at the optimal bias is (k − 1)Σ(n_i − v_i)² − 2Σn_i² + 2Σn_i²/C_i, in the squared embedding lengths n_i and capacities C_i = (w_i·w_i)²/Σ_j(w_i·w_j)², and its minimum with the capacities summing to the number of neurons D gives C_i = clip((k − 1)v_i/((k − 3)λ) − 2/(k − 3), 0, 1): for k ≤ 3 each feature is given a whole dimension or none, and for k > 3 a feature is shared between neurons exactly when 2λ/(k − 1) < v_i < λ

Graph View

Source

Scherlis et al. (2022), Polysemanticity and Capacity in Neural Networks, LIT-1625, read in full in NOTE-1205: §§1, 2, 3.7, 4.6 and 5, and the Alignment Forum post that announced it.

What is asserted

That polysemanticity can be understood “through the lens of feature capacity”: the model has a fixed amount of capacity to allocate, the optimum equalizes marginal returns among partially represented features, “polysemanticity will arise when there are diminishing marginal returns to capacity”, which sparse inputs produce, and “differences in architecture can lead to different sizes of blocks, which could provide a way to control the extent of polysemanticity in models”.

What the record holds toward it

  • The first-order condition (§2.2), which needs only that capacity never hurts and that every allocation is realizable. It is general, but it describes an optimum, and trained networks need not reach it.
  • One model solved (THEORY-1036), where the account holds exactly.
  • Six-feature numerical comparisons with Elhage et al.’s autoencoder (LIT-323) under three activations, shown as phase diagrams: the qualitative shape agrees, the block structure differs.

Why it stays Proposed

No trained network is examined, the comparisons are of toy models, and the authors themselves expect the story in large models to be messier. The architecture claim rests on one contrast between two toys.

What would confirm or reject it

The promote_when above.

Backlinks