Source
Liu, Liu, Michaud, Gore and Tegmark (2025), LIT-951, read in NOTE-794. The account rests on §§2.2–2.6, Figs. 4, 5, 7 and 12, and Eqs. 1–3 and 6–9.
What was actually shown
The model. The Geometry model has loss ℓ = Σ_i p_i L(s_i), with skill level s_i = (θ − θ₀)·t_i and L(s) = (1 − s)² or −log σ(s). The t_i are random and orthogonalised when n_task ≤ n_dim, and p_i ∝ i
Two orthogonal tasks (Fig. 4). The setting is n_dim = 1000, batch 128, and p₁ + p₂ = 1, with SGD, SignGD and Adam.
- Under SGD, task 2’s convergence time grows roughly as 1/p₂, as the decoupled linear dynamics require.
- Under SignGD, task 2 makes rapid progress only once task 1 is done, and t₂/t₁ saturates at 2 however small p₂ is.
- Adam lies between, sub-linear in p₁/p₂.
The mechanism offered is sign(g₁ + g₂) ≈ sign(g₁) when ‖g₁‖ ≫ ‖g₂‖. The number of coordinates whose update sign agrees with each task’s gradient falls for task 1 as it rises for task 2 (Fig. 5).
A rotated quadratic (Fig. 12). Take ℓ = x₁² + 0.1x₂² + 0.01x₃² + 0.001x₄² with x = Rθ.
- SGD gives the same dynamics for R = I and for a Hadamard R.
- SignGD learns all four together when R = I, and in Domino sequence when R = H₄.
- So the sequence needs the loss’s eigendirections to be misaligned with the coordinates the optimizer acts on.
Many tasks (Fig. 5). Under SignGD skills are learned in frequency order. The clean Domino form holds “only for large α and the first few skills”.
The reduced models.
- Resource model (Eq. 1). du_i/dt = −η_eff p_i u_i/(Σ_j p_j u_j + N₀), with one fitted constant N₀. It matches the SignGD curves (Fig. 7). It conserves u_i^(1/p_i) = C exactly.
- Domino model. Its limit N₀ → 0, p₁ ≫ p₂ ≫ … gives strict sequence, with t_n ∝ n, against t_n ∝ n^α for SGD.
What could have come out otherwise.
- SignGD could have shown the same 1/p₂ scaling as SGD.
- The rotated quadratic could have shown a sequence under both optimizers, or under neither.
What this does not say
- Not proved. The Domino effect under SignGD is a simulation result. The Resource model is fitted to it, not derived from the optimizer, and the Domino model is a limit of that fit.
- Not about the order. Frequency sets the order under every optimizer tested. The optimizer changes whether onsets overlap and how the n-th onset scales.
- Not that real networks owe their stages to Adam. The real-network evidence is small. Two sparse parities with Adam show the effect (Fig. 2), and one two-variable regression shows t₂ ≳ 2t₁. Gradient descent was not run on these for contrast.
- Not the only cause of sequential onset. In the same paper, a parity that is the XOR of two others waits for them under Adam (Fig. 16), and an independent parity of the same order waits longer. That is dependency, a different cause, and this account does not hold it.
- Not an instruction. The paper’s reweighting, SignGD-against-grokking and optimizer comparisons are machine-learning practice and are left to the anthology.
Connections
- QUESTION-039. It answers in part, and it adds to the question’s list (data statistics, initial scale, depth, architecture) a factor the list lacks: the optimizer. In this model the optimizer decides whether acquisitions arrive together or one at a time. Read with the dependency experiment, the paper gives at least two kinds of event behind “sequential acquisition”.
- THEORY-229, THEORY-186 and THEORY-187. These get one-at-a-time
acquisition under gradient flow from small initialization. Each mode
waits near an unstable fixed point for a time set by its eigenvalue
and the log of the initial scale. The Geometry model has no such fixed
point, and under gradient descent it shows no sequence. So the two
mechanisms make opposite predictions under the same intervention:
- replace sign descent with gradient descent, and the Domino sequence disappears while THEORY-229’s steps remain;
- start from order-one scale, and THEORY-229’s steps shrink while the Domino sequence does not depend on scale.
- THEORY-086. Under SGD the Geometry model is the kernel picture there: every direction is fitted at once, at a rate set by its weight.
- THEORY-212. There a conserved quantity of the initialization decides between staged and simultaneous learning. Here the optimizer does. Neither leaves “staged or not” to the data alone.
- THEORY-039. Its caution that transitions differ in kind is borne out. Optimizer competition, dependency and small-initialization escape are three different routes to the same-looking staircase.
Note of 2026-10-19: the two-task and quadratic results, reproduced
Reproduced for the promotion audit (batch 045), with full-batch deterministic updates, n_dim = 1000, squared loss, and a skill counted as learned at s ≥ 0.9.
- Two orthogonal tasks. Under gradient descent, task 2’s time grows with 1/p₂. Under sign descent, t₂/t₁ was 1.00, 1.48, 1.76, 1.88 and 2.02 at p₁ = 0.5, 0.8, 0.9, 0.95 and 0.99.
- The rotated quadratic x₁² + 0.1x₂² + 0.01x₃² + 0.001x₄², with x = Rθ. Under sign descent with R = I, all four coordinates fell below half their start at the same step. With R = H₄ they did so at steps 250, 729, 1171 and 1730. Gradient descent was identical under both.
Both match the paper’s text. They are the paper’s model run again, not new evidence about trained networks.
Note of 2026-10-20: the wider claim, filed separately
When this account was narrowed, its wider form went nowhere a later result could confirm or reject it. Under ADR-047 v4 it is now the Proposed THEORY-783: that sign-descent-like optimizers such as Adam are one source of sequential skill onset in trained networks, as the paper proposes. This account is that one’s evidence, and nothing here changes.
Note of 2026-10-26: the misalignment condition in a transformer, and a dependency case
T24 of the exchange “Experimental Infrastructure Review” cited this account for transformers without its toy scope (QUESTION-275). Two works read on 2026-10-26 for that question bear on it.
- The misalignment condition in a transformer (THEORY-981). In Musat, Pimentel, Zucchet and Hofmann’s attention-only transformers on block-list inductive tasks, population gradient descent never leaves the subspace of weights the data’s symmetries fix. The record checked, when it read the paper, that sign descent keeps that subspace only when its basis is aligned with the parameter coordinates: with random orthonormal embeddings or write maps, sign(G) leaves it (relative residual 0.6 to 0.7) while G stays in it. That is this account’s rotated quadratic, carried to a transformer’s weight space: an elementwise optimizer changes the dynamics where the relevant directions are not coordinate-aligned. It is about which subspace training stays in, not about onset order, and nobody has yet run the comparison QUESTION-193 asks for in that setting.
- Dependency, measured (THEORY-973). Under “Not the only cause”, the record now holds a dependency case in trained networks: in small attention-only transformers, in-context learning waits on the composition of two heads, and removing the need for it moves the jump earlier.
Status and title are unchanged.