nucleation

LIT-951: Physics of Skill Learning

2 min read

Properties5
StatusActive — read 2026-10-14 (NOTE-794) in full; worth reading for one contrast in a toy model of independent skills, the Geometry model. Under gradient descent, orthogonal skills are learned together, each in time ∝ 1/p. Under sign descent they compete for coordinate signs, and each starts as the one before finishes. That is the Domino effect, and it makes the n-th skill take time ∝ n. A separate MLP experiment shows a second cause of sequential onset, dependency between skills. Read it knowing that the Domino effect is shown in simulation, that the Resource and Domino models are fitted to it rather than derived, that the order is frequency order under every optimizer, and that the real-network evidence is a few small MLPs trained with Adam. Held as THEORY-253.
Version2
Tagslearning-theory, anthology-candidate
Cited as “paper” byNOTE-794 — Liu et al. — Physics of Skill Learning
Cited as “source” byTHEORY-253 — In Liu, Liu, Michaud, Gore and Tegmark's Geometry model of independent skills with orthogonal task vectors, simulation shows skills progressing together under gradient descent, the weaker taking about 1/p of the time, and in sequence under sign descent, a weaker skill progressing fast only once a stronger is done and the ratio of their times approaching 2; on a rotated quadratic the sequence appears under sign descent only when the loss's eigendirections are misaligned with the coordinates, THEORY-783 — In neural-network training, sign-descent-like optimizers such as Adam make independent skills of unequal frequency compete for coordinate signs, so that they are learned one after another where gradient descent would learn them together; this is one source of sequential skill onset, beside dependence between skills, as Liu, Liu, Michaud, Gore and Tegmark propose

Graph View

Ziming Liu, Yizhou Liu, Eric J. Michaud, Jeff Gore and Max Tegmark (2025), arXiv:2501.12391 (preprint, under review).

Key takeaways

From the reading, NOTE-794.

  • The Domino effect is the optimizer’s. The Geometry model has loss Σ p_i L((θ − θ₀)·t_i) with orthogonal task vectors. Under SGD, task 2 takes about 1/p₂ as long as task 1. Under SignGD it waits for task 1 to finish and then needs a constant extra delay: t₂/t₁ → 2. Adam lies between.
  • Rotation matters. Take a hierarchical quadratic loss.
    • SGD behaves the same in any basis.
    • SignGD learns every direction together when the loss is aligned with the coordinates, and in Domino sequence when it is rotated.
  • The order is frequency order throughout. The optimizer changes whether onsets overlap and how the n-th onset scales: ∝ n under sign descent, against ∝ n^α if the time is ∝ 1/p.
  • A one-parameter ODE. The Resource model, du_i/dt = −η_eff p_i u_i/(Σ p_j u_j + N₀), fits the SignGD curves with N₀ ≈ 0.003. It conserves u_i^(1/p_i) exactly. Its strong-hierarchy limit is the strict-sequence Domino model.
  • Dependency is a second cause. A two-layer MLP learning x₁⊕x₂, x₃⊕x₄ and x₁⊕x₂⊕x₃⊕x₄ starts the third only after the first two are perfect. An independent 4-parity is delayed more.
  • Scaling claims rest on fits. At Zipf α = 1, the Geometry model gives α_N ≈ 0.34, which “happens to be” Chinchilla’s value. The reweighting and optimizer proposals are machine-learning practice.

Standing in the record

Filed on 2026-10-12 from the owner’s papers-feed for QUESTION-039, and read on 2026-10-14 in NOTE-794. Its optimizer result is held as THEORY-253, a partial answer to QUESTION-039. It names a factor the question does not list, the optimizer, which decides in this model whether acquisitions are sequential. That makes it a rival mechanism to the small-initialization staircase of THEORY-229 and THEORY-187: the two predict opposite effects of switching to gradient descent. The seeded reading, that the models “explain the order”, is corrected above. Carries the anthology-candidate flag: the anthology was checked when it was filed, and the paper’s practice proposals would belong there.

Backlinks