nucleation

LIT-1530: Invariant Learning Dynamics of Transformers in Inductive Reasoning Tasks

1 min read

Properties5
StatusActive — read 2026-10-26 (NOTE-1177) in full, from the arXiv copy (v2); worth reading for its proof that symmetries of the data confine population gradient descent to a subspace of interpretable head actions (THEORY-981), for its theorem that a saturated memorized shortcut starves the in-context circuit of gradient on the data it solves (THEORY-1009), and for its scan in which initialization picks one of several induction circuits, read by the authors as cooperation through shared units and competition for layers (THEORY-1001). Read it knowing that the architecture is idealized (fixed orthogonal write maps, merged key-query, orthonormal embeddings), that invariance is proved and attraction is not, that the results are for gradient descent while its circuit-detection experiment uses AdamW, and that the competition scan was run in training projected onto seven coordinates.
Version2
Tagslearning-theory, representation-learning, anthology-candidate
Cited as “paper” byNOTE-1177 — Musat, Pimentel, Zucchet and Hofmann — Invariant learning dynamics of transformers in inductive reasoning tasks
Cited as “source” byTHEORY-1001 — In a transformer whose candidate circuits are sets of shared elementary coordinates, how circuits interact during training is set by what they share: circuits that share a unit cooperate, and circuits that need the same layers for different roles compete, so that which circuit wins is decided by the initialization of units beyond its own, with sharp boundaries between competitors and soft ones between cooperators, as Musat, Pimentel, Zucchet and Hofmann read their three-layer scan, THEORY-1009 — In Musat, Pimentel, Zucchet and Hofmann's two-layer attention-only transformer on in-context bigrams, once an in-weights circuit that maps each common token to its memorized partner has saturated, the population gradient on the induction circuit is what it would be without it multiplied by fr + (1 − fr)fv, the share of the data the memorized map does not solve, up to errors that vanish at large vocabulary and large in-weights margin when no distractor block carries the query's partner; the two circuits occupy orthogonal coordinates and still interact, because they solve overlapping parts of the data, THEORY-981 — When a data distribution is unchanged by a group of orthogonal transformations under which a network is equivariant, the weights fixed by that group form a subspace that population gradient descent never leaves; for attention-only transformers on block-list inductive tasks, with fixed orthogonal write maps, Musat, Pimentel, Zucchet and Hofmann compute this subspace as the span of a few elementary token and position actions composed through the heads, and an elementwise optimizer such as sign descent keeps it only when those actions are aligned with the parameter coordinates

Graph View

Musat, T., Pimentel, T., Zucchet, N. & Hofmann, T. (2026). Invariant learning dynamics of transformers in inductive reasoning tasks. arXiv:2607.11875.

Read in full on 2026-10-26 (NOTE-1177). Three accounts were drawn from it: THEORY-981, THEORY-1009 and THEORY-1001.

Why this is here

Cited at T24 as “particularly valuable prior art and a strong comparator — not merely background motivation” for the reply’s proposed study of induction-circuit formation. Its competition between circuit solutions decided by data statistics and initialization is what the record’s framework would have to predict better than to earn QUESTION-275’s prospective test. It bears on QUESTION-039 and on THEORY-253, the record’s toy-scope account of skills learned in order.

Backlinks