nucleation

LIT-943: From Lazy to Rich: Exact Learning Dynamics in Deep Linear Networks

2 min read

Properties5
StatusActive — read 2026-10-13 (NOTE-715): main text in full, and the statement of Appendix A.3 (random initializations are λ-balanced) without its proof; the proofs in Appendices B–C not checked. Its control parameter is the relative scale of the layers, not the absolute scale the seed entry named. The account filed from it is THEORY-212, and the question it raises under SGD is QUESTION-081.
Version2
Tagslearning-theory, representation-learning, anthology-candidate
Cited as “paper” byNOTE-715 — From Lazy to Rich: Exact Learning Dynamics in Deep Linear Networks
Cited as “source” byTHEORY-212 — In a two-layer linear network trained by gradient flow, the relative scale of the two layers, which gradient flow conserves, sets the learning regime: balanced layers learn in sigmoidal stages and both build task-specific representations, while strongly unbalanced layers learn exponentially with a static kernel and leave the larger layer task-agnostic

Graph View

Clémentine C. J. Dominé, Nicolas Anguita, Alexandra M. Proca, Lukas Braun, Daniel Kunin, Pedro A. M. Mediano and Andrew M. Saxe (2025), ICLR 2025 — ARXIV-2409.14623

Key takeaways

  • Relative scale sets the regime. With the absolute scale held fixed, the relative scale λ of the two layers moves the network from rich (λ → 0, sigmoidal singular-value transitions) to lazy (|λ| → ∞, exponential transitions, static NTK) (Theorem 5.1).
  • λ is conserved under gradient flow. The λ-balanced condition, if it holds at initialization, “will persist throughout training” (Assumption A2). That is the rescale conservation law.
  • Where the task lives. At convergence, the layer initialized large stays task-agnostic and near the identity. The layer initialized small carries the task structure, at a scale ∝ 1/|λ| (Theorems 5.2, C.4).
  • Architecture and sign. Funnel networks go lazy as λ → +∞ and inverted funnels as λ → −∞. In the opposite limits each has a lazy phase followed by a delayed rich phase.

Standing in the record

First filed for QUESTION-039, from the abstract. Read on 2026-10-13 because its control parameter is the quantity that gradient flow conserves and SGD’s minibatch noise drives toward balance (THEORY-211). The paper analyses full-batch gradient flow only. The reading is NOTE-715, the account is THEORY-212, and the open question under SGD is QUESTION-081.

Backlinks