nucleation

THEORY-792: In a deep linear network trained by gradient descent from generic initial weights, not only from decoupled ones, depth slows learning through the initial end-to-end strength of each mode, so that random orthogonal weights, whose product is an isometry, learn in a number of iterations that stays bounded as depth grows, as Saxe, McClelland and Ganguli argue from simulation

2 min read

Properties5
StatusProposed
Tagslearning-theory
SourceLIT-864 — Exact solutions to the nonlinear dynamics of learning in deep linear neural networks
ExtendsTHEORY-187 — In a deep linear network trained by gradient descent on whitened inputs from decoupled, balanced initial weights, depth slows learning through the initial end-to-end strength of each mode: counting iterations as learning time over the stable step size, from strength of order one the number of iterations stays bounded as depth grows, once the step size falls with depth as stability requires, while from small strength ε the plateau lengthens from ln(1/ε) with one hidden layer towards 1/ε at great depth
AnswersQUESTION-166 — Does the depth law of deep linear networks hold from starts off the decoupled set: learning time, counted in iterations at the stable step size, bounded in depth when every mode's initial end-to-end gain is of order one, and growing as a power of 1/ε when it is ε, for random orthogonal or small random weights not aligned with the task's singular vectors?

Graph View

Source

LIT-864. Filed by the promotion audit’s narrowing-loss sweep (batch 066, ADR-047 v4) on 2026-10-20, to carry the wider claim that the narrowing of THEORY-187 dropped. THEORY-187 is Active and is this THEORY’s evidence; this one stays Proposed.

What was actually shown

The paper solves the dynamics exactly when the initial weights are decoupled: aligned with the singular vectors of the input–output correlation and balanced across layers. THEORY-187 states that result, re-derived in the promotion audit. The paper then argues, from MNIST runs and from simulations of random matrix products, that random orthogonal layers behave like a decoupled start of order-one strength, because the product of orthogonal matrices is an isometry on every mode, and that scaled Gaussian layers preserve norms but not isometry and so learn slowly at depth (NOTE-667, §3 and the closing sections). That step is the argument this THEORY records. No derivation covers a start off the decoupled set.

What would settle it

The promote_when is inherited from THEORY-187’s before it was narrowed, and QUESTION-166 asks the same question.

What this does not say

It does not say that the ε law of THEORY-187 (a plateau growing from ln(1/ε) towards 1/ε) holds from small random weights; the paper argues that small Gaussian starts are slow but does not derive their rate. It says nothing about nonlinear networks, beyond the paper’s separate and also simulated claim about tanh networks near gain 1. It is not advice on how to initialise.

Backlinks