nucleation

LIT-1535: Differentiation and Specialization of Attention Heads via the Refined Local Learning Coefficient

1 min read

Properties5
StatusActive — read 2026-10-26 (NOTE-1188) in full, from the arXiv PDF of v1; worth reading as the record's first developmental use of the learning coefficient, for per-head trajectories that separate attention heads by function (THEORY-975), and for its observations of heads changing role as circuits form (THEORY-967, Proposed). Read it knowing that it is one two-layer attention-only model with three appendix seeds, that the estimates are trusted only for their ordering, that cluster counts were chosen by eye, and that nothing is predicted before it happens.
Version2
Tagslearning-theory, loss-landscapes, representation-learning, anthology-candidate
Cited as “paper” byNOTE-1188 — Wang, Hoogland, van Wingerden, Furman and Murfet — Refined local learning coefficients of attention heads
Cited as “source” byTHEORY-967 — In a two-layer attention-only transformer, attention heads change function during training: heads in the first layer that predict simple n-grams early later become the previous-token heads of the induction circuit, a bracket-matching support head, or supporting members of a circuit with second-layer n-gram heads, giving up simple n-grams as they do, and particular n-grams pass from one head to another; so later circuits form partly by repurposing components that already had a role, THEORY-975 — In a two-layer attention-only transformer trained on the Pile, the weight-refined local learning coefficients of individual attention heads start alike and diverge over training, and clustering the curves groups the heads by the functional types that ablation assigns them at the end of training, previous-token, current-token, induction and n-gram-memorizing; among layer-1 heads, those running simple algorithms end with the lowest values and those memorizing the most n-grams with the highest

Graph View

Wang, G., Hoogland, J., van Wingerden, S., Furman, Z. & Murfet, D. (2024). Differentiation and specialization of attention heads via the refined local learning coefficient. ICLR 2025. arXiv:2410.02984.

Read in full on 2026-10-26 (NOTE-1188). The summary is checked against the reading.

Why this is here

Cited at T24 as prior art for components of a network becoming differentiated over training. The record held singular learning theory only through Watanabe (LIT-354, Unavailable) and the WBIC (LIT-616), and nothing from developmental interpretability. It bears on QUESTION-275 and on QUESTION-039.

Backlinks