The question
THEORY-212 (Dominé et al.) says that in a two-layer linear network the relative scale λ of the layers decides between rich and lazy learning. That holds because gradient flow conserves λ. THEORY-211 (Ziyin et al.) says that under SGD the same quantity is driven toward balance, d(‖u‖² − ‖w‖²)/dt = −T(uᵀC₁u − wᵀC₂w) with T = η/S, plus weight decay. The question is which wins.
- If the drift is fast compared with learning, an unbalanced, lazy start would move toward balance while it learns, and the regime would shift toward rich during training.
- If it is slow, learning finishes in the regime the initialization set, and balance arrives only later, at stationarity.
- If the noise-balanced point is not λ = 0, SGD would select its own regime, set by the data’s noise structure, regardless of initialization.
Why the record needs it
QUESTION-039 asks what sets when and how structure is acquired. THEORY-212 answers with the initialization. THEORY-211 implies that, under the optimizer actually used, the initialization’s say may be temporary. Which is true decides whether “the initialization sets the regime” is a fact about networks trained with SGD, or only about gradient flow.
What the record holds
- THEORY-212: the regime as a function of a conserved λ, under gradient flow.
- THEORY-211: the law of balance. Its two-layer linear example reaches uᵢ² = wᵢ² at stationarity. That is a statement about the end of training, not about the transitions.
- THEORY-186 and THEORY-187: the balanced case’s stage-like learning.
- LIT-1012 (Neural Mechanics): weight decay also drives the conserved quantity toward zero, as e^{−2λt} for weight decay λ, plus a gradient-norm term whose sign is not fixed.
What would answer it
- A measurement: two-layer or deeper networks trained with SGD from unbalanced starts, tracking λ(t) and the NTK’s movement against the learning curve, at several η/S.
- Or a derivation: comparing the drift rate TC₀ with the rate of the slowest singular-value transition.