Source
Saxe, McClelland and Ganguli (2014), LIT-864, §2, §3 and Supplementary Appendices B–B.1, as read in NOTE-667.
What was actually shown
Take a linear network with N_l layers, whitened inputs and squared error. Start its weights on the decoupled set, where each layer’s singular vectors chain into the next and the first and last match the SVD of the input–output correlation. Then each singular mode evolves alone, as N_l − 1 scalars aᵢ descending (1/2τ)(s − Πaᵢ)². The differences aᵢ² − aⱼ² are conserved, and from equal starts the end-to-end strength u = Πaᵢ obeys τu̇ = (N_l − 1)u^{2−2/(N_l−1)}(s − u). This is exact.
Two consequences follow.
- Small start. With one hidden layer, learning from u = ε takes about (τ/s)ln(s/ε). As depth grows the factor u^{2−2/(N_l−1)} starves growth near zero. Integrating the equation gives a plateau of order ε^{−(N_l−3)/(N_l−1)}, and the paper’s infinite-depth solution, Eq. 17, has an s/ε term. The exponent at finite depth is my integration; the paper states only the end cases.
- Large start. Depth also raises the curvature. The top Hessian eigenvalue at the optimum is (N_l − 1)s^{(2N_l−4)/(N_l−1)}/τ, so the stable step size falls as 1/(N_l s²). Measured at that step size, the iterations an infinitely deep network needs exceed a three-layer network’s by about cs/ε, which is finite (Appendix B.1). When the initial strength is of order one, depth costs a bounded number of iterations.
The experiment that could have come out otherwise is Fig. 4. Deep linear networks on MNIST, 3 to 100 layers with width 1000, start on the decoupled set at u₀ = 0.001, and each depth gets the best of twenty step sizes. The iterations to a fixed training error saturate with depth, and the best step sizes fall as 1/N_l, as predicted.
The paper then argues that the same holds from random starts whose product of layers is close to an isometry. Gaussian layers scaled to preserve norm on average have a product whose singular values become heavy-tailed with depth, as computed numerically. Orthogonal layers have an exactly orthogonal product. On MNIST, learning time grows with depth from the first and not from the second, nor from pretraining (Fig. 6A).
What this does not say
- Not that random orthogonal weights are a decoupled start. They are not aligned with the task’s singular vectors, so the exact theory does not reach them. Their depth-independence is one MNIST experiment, and the explanation by dynamical isometry is an argument with no derivation of learning time from the Jacobian spectrum.
- Not that the stepwise, strongest-first schedule survives a start of order one. That schedule is the small-start corner (THEORY-186). From a near-isometric start the modes are already of order one, and the paper does not describe the order in which the residual is learned.
- Not that deep networks learn faster. Continuous-time learning time falls with depth at a fixed rate, which is an artefact. Counted at the stable step size, depth adds delay, bounded only when the start is strong. The count is in iterations, not computation.
- Not about nonlinear networks’ learning time. The paper’s nonlinear results concern signal propagation at initialisation (an order-to-chaos transition at gain 1 for orthogonal tanh networks). They do not show that learning time follows.
- Not about correlated inputs beyond input correlations that share the task’s singular vectors (Appendix E, stated without being worked out).
- Not an instruction. That orthogonal initialisation or pretraining should be used is the paper’s recommendation for practice. It belongs to the anthology, and is not what this account claims.
Where it sits
THEORY-186 states the small-start, one-hidden-layer case, mode by mode in order of strength. This account adds how depth and the initial scale set the length of the plateaus. THEORY-039 holds plateaus and transitions as one kind of training phase; here their length depends on the initial scale in a way that changes with depth. THEORY-182’s sigmoids are the one-hidden-layer case for a symmetric factorisation.
Note of 2026-10-13: the balanced-start condition, under gradient flow and SGD
This account is exact for decoupled, balanced initial weights. Two readings filed on 2026-10-13 bear on that condition.
- Under gradient flow, balance is kept. The difference of squared norms between adjacent layers is conserved (Neural Mechanics, LIT-1012, Theorem 1). A balanced start stays balanced, which the exact solution relies on.
- Under SGD, balance is approached. Minibatch noise drives the difference toward balance (THEORY-211). For a two-layer linear network, Ziyin et al. find uᵢ² = wᵢ² at stationarity, with probability 1.
So balance is less special under SGD than under gradient flow. But the attraction acts over the long run, at stationarity. It does not supply balance during the stage-like transitions this account describes, which still need the balanced start, or something close to it.
Note of 2026-10-19: the depth law re-derived for the decoupled start
Carried out for the promotion audit’s batch 027 (ADR-047 v2). Let m = N_l − 1 be the number of weight layers.
- Mode equations. On the decoupled set with whitened inputs, a mode of strength s has loss (1/2τ)(s − u)², u = a₁⋯a_m. Gradient flow gives τȧᵢ = (s − u)∏_{j≠i}aⱼ, so d(aᵢ²)/dt = (2/τ)(s − u)u for every i. The differences aᵢ² − aⱼ² are conserved, a balanced start stays balanced, and with aᵢ = a, u = aᵐ: τu̇ = m u^{2−2/m}(s − u).
- Small start. The time from u = ε to near s is (τ/m)∫ du/(u^{2−2/m}(s − u)). For m = 2 this is (τ/2s)ln(s/ε) + O(1). For m > 2 the integral is dominated near ε and is about (τ/m)ε^{−(m−2)/m}/((1 − 2/m)s). The exponent (m − 2)/m = (N_l − 3)/(N_l − 1) rises to 1, the 1/ε of Eq. 17, as depth grows.
- Stable step. At the optimum the Hessian is (1/τ)∇u∇uᵀ with ∂u/∂aᵢ = s^{1−1/m}, so its top eigenvalue is m s^{2−2/m}/τ and the stable step is of order τ/(m s^{2−2/m}), about 1/(N_l s²) at depth.
- Iterations. Time over that step is s^{2−2/m} times the integral in step 2. From an initial strength of order one the integral converges as m grows, so the count is bounded in depth. From ε it is about s^{1−2/m}ε^{−(m−2)/m}/(1 − 2/m), growing towards s/ε.
Each step uses the balanced, decoupled start. Off the decoupled set the modes couple and step 1 fails, which is why the general case is QUESTION-166.
Note of 2026-10-20: the wider claim filed separately
When this THEORY was narrowed to what its source showed, the wider claim it had stated was left with nowhere to go but this body and a QUESTION. Under ADR-047 v4 it is now its own Proposed THEORY, THEORY-792, which extends this one: the depth law from generic starts, such as random orthogonal weights, which the paper argues from simulation. This entry is that THEORY’s evidence and is not changed by it.