nucleation

THEORY-212: In a two-layer linear network trained by gradient flow, the relative scale of the two layers, which gradient flow conserves, sets the learning regime: balanced layers learn in sigmoidal stages and both build task-specific representations, while strongly unbalanced layers learn exponentially with a static kernel and leave the larger layer task-agnostic

2 min read

Properties4
StatusProposed
Tagslearning-theory, representation-learning, anthology-candidate
SourceLIT-943 — From Lazy to Rich: Exact Learning Dynamics in Deep Linear Networks
Bears onQUESTION-039 — What sets the order and the moment, in training time or in examples, at which a network acquires a structure, and is acquisition one kind of event or several?

Graph View

Source

Dominé, Anguita, Proca, Braun, Kunin, Mediano & Saxe (2025), LIT-943, read in NOTE-715: §§1–7, with the proofs unchecked.

The claim

Hold the absolute scale of the initialization fixed and vary how it is split between the two layers, measured by λ in W₂ᵀW₂ − W₁W₁ᵀ = λI. Under gradient flow λ never changes, so the split at initialization governs the whole trajectory:

  • Balanced (λ → 0): rich. Each singular value of the input–output map is learned in a sigmoidal transition, in order of strength. The NTK changes, and both layers end with task-specific representations, whatever the absolute scale. This is the stage-like learning of THEORY-186.
  • Unbalanced (|λ| → ∞): lazy. Transitions are exponential, 1 − e^{−|λ|t/τ}, and all modes are learned together, as in a shallow network. The NTK is static. The larger layer stays near an identity projection. The smaller layer carries the task structure, but at a scale ∝ 1/|λ| that leaves it out of the kernel.
  • Architecture and sign. In funnel and inverted-funnel networks, one sign of λ gives lazy learning and the other gives a lazy phase followed by a delayed rich phase.

What it bears on

  • QUESTION-039 asks what sets the order and moment at which structure is acquired. Here a single conserved quantity of the initialization decides whether acquisition comes in ordered stages or all at once.
  • THEORY-186 and THEORY-187 are the balanced, aligned special case. Their condition of a balanced start is here a parameter, not a restriction.
  • THEORY-211. Under SGD λ is not conserved: minibatch noise drives it toward the point where the layers’ gradient noise balances. This account’s control parameter is therefore fixed only under gradient flow. QUESTION-081 asks what that does to the regimes.

What this does not say

  • It does not cover deep or nonlinear networks. The paper defers those to Kunin et al. (2024), not held, and expects the nonlinear case to differ.
  • It does not cover SGD. The paper never models minibatch noise.
  • It does not say which regime generalizes better in general. The transfer and fine-tuning results are on small constructed tasks.
  • It does not instruct. Initialization schemes for practice belong to the anthology.

Backlinks