nucleation

LIT-864: Exact solutions to the nonlinear dynamics of learning in deep linear neural networks

4 min read

Properties4
StatusActive — read 2026-10-09 (NOTE-667); worth reading as the paper in which the learning dynamics of deep linear networks were solved exactly from decoupled starts at any depth (Fukumizu had solved one hidden layer for another class of starts) and the stepwise, mode-by-mode picture was set out that later work calls Saxe-style. Gradient descent on a linear network with one hidden layer and whitened inputs is a cubic system whose modes cooperate within a singular mode of the input–output correlations and repel across modes; from decoupled initial weights each mode's strength follows a sigmoid reaching s at a time of about (τ/s)ln(s/ε), with a conserved difference of squared layer strengths and, in the appendix, the unbalanced case. Beyond what the 2019 restatement keeps, it solves arbitrary depth from decoupled starts: depth slows learning through the initial end-to-end mode strength, so from strength of order one the number of iterations stays finite as depth grows once the step size is scaled down with depth, while from strength ε the delay grows like s/ε. It argues that scaled Gaussian layers preserve norms but not isometry, that random orthogonal layers give depth-independent learning times on MNIST, and that tanh networks with orthogonal weights near gain 1 keep the end-to-end Jacobian's singular values of order one. Random initial weights, the role of isometry in learning time and everything nonlinear rest on simulation, not proof.
Tagslearning-theory, representation-learning, anthology-candidate
Cited as “paper” byNOTE-667 — Exact solutions to the nonlinear dynamics of learning in deep linear neural networks
Cited as “source” byTHEORY-186 — A deep linear network trained from small weights learns the singular modes of its input–output correlations one at a time in order of strength, each in a sharp sigmoidal transition, while a shallow network learns them all together; the stages come from the product of layers and the data's spectrum, not from nonlinearity, THEORY-187 — In a deep linear network trained by gradient descent on whitened inputs from decoupled, balanced initial weights, depth slows learning through the initial end-to-end strength of each mode: counting iterations as learning time over the stable step size, from strength of order one the number of iterations stays bounded as depth grows, once the step size falls with depth as stability requires, while from small strength ε the plateau lengthens from ln(1/ε) with one hidden layer towards 1/ε at great depth, THEORY-740 — A two-layer linear network trained by gradient flow on white inputs from a decoupled, balanced start learns each singular mode of its input–output correlation in a logistic rise that waits a time of order its inverse strength times the log of the inverse initial scale, so the modes arrive one at a time in order of strength; a single-layer network on the same data learns every mode on one timescale with no plateau, so under these conditions the stages come from the product of layers and the data's spectrum, THEORY-792 — In a deep linear network trained by gradient descent from generic initial weights, not only from decoupled ones, depth slows learning through the initial end-to-end strength of each mode, so that random orthogonal weights, whose product is an isometry, learn in a number of iterations that stays bounded as depth grows, as Saxe, McClelland and Ganguli argue from simulation

Graph View

Andrew M. Saxe, James L. McClelland and Surya Ganguli (2014), International Conference on Learning Representations (ICLR 2014) — ARXIV-1312.6120 (v1 20 December 2013; read as v3, 19 February 2014)

Key takeaways

  • The general equations. For y = W³²W²¹x with squared error and a small learning rate, τ dW²¹/dt = W³²ᵀ(Σ³¹ − W³²W²¹Σ¹¹) and τ dW³²/dt = (Σ³¹ − W³²W²¹Σ¹¹)W²¹ᵀ, cubic in the weights. With Σ¹¹ = I, and rotated into the SVD of Σ³¹, each input–output mode α has a weight vector aα into the hidden layer and bα out of it. They follow gradient descent on E = (1/2τ)Σ(sα − aα·bα)² + (1/2τ)Σ_{α≠β}(aα·bβ)²: the two vectors of one mode cooperate, and vectors of different modes repel, pushing the network towards a decoupled state. All fixed points except the best rank-N₂ approximation are saddles (quoted from Baldi and Hornik).
  • The three-layer solution. From decoupled initial conditions, where W³² = U D_a Rᵀ and W²¹ = R D_b Vᵀ, the modes evolve independently. Each is a pair (a, b) with τȧ = b(s − ab), τḃ = a(s − ab). The scaling symmetry conserves a² − b², so trajectories run along hyperbolas. With a = b, u = ab obeys τu̇ = 2u(s − u), so u(t) = s e^{2st/τ}/(e^{2st/τ} − 1 + s/u₀), and the learning time is (τ/s)ln(s/ε). Appendix A solves the unbalanced case in hyperbolic coordinates and gets the same O(τ/s) timescale when a² − b² is small. Random small weights and a tanh network are compared by simulation only (Fig. 3).
  • Any depth. With N_l layers, each decoupled mode has N_l − 1 scalars descending E = (1/2τ)(s − Πaᵢ)². The differences aᵢ² − aⱼ² are conserved, and from equal starts u = Πaᵢ obeys τu̇ = (N_l − 1)u^{2−2/(N_l−1)}(s − u). In the infinite-depth limit, τu̇ = N_l u²(s − u). The continuous-time learning time then falls to zero as depth grows, but that is an artefact of a fixed step size. The stable step size falls as O(1/(N_l s²)), and with it the excess over a three-layer network tends to about cs/ε, a finite delay that grows as the initial mode strength ε shrinks. Deep linear networks on MNIST, decoupled starts at u₀ = 0.001 and step sizes tuned per depth, show learning times that saturate up to 100 layers (Fig. 4).
  • Pretraining, as a condition. Autoencoder pretraining of a linear network learns the principal components Q of the input. It leaves a decoupled start for the supervised task only if Q equals the task’s input singular vectors V, that is, if Vᵀ Σ¹¹ V is diagonal. MNIST is shown to come close (Fig. 5), and a pretrained five-layer linear network learns faster than one from small random weights.
  • Norm preservation is not isometry. Gaussian layers scaled by 1/√N preserve a typical vector’s norm on average. The singular values of a product of such layers nevertheless grow strongly kurtotic with depth, with most near zero and a long tail, which the paper shows numerically. A product of orthogonal layers is orthogonal. On MNIST, learning time grows with depth from scaled Gaussian layers but not from random orthogonal layers or pretraining (Fig. 6A). The paper names the condition dynamical isometry: as many singular values of the end-to-end Jacobian as possible near one. Its tie to learning time is argued, not derived.
  • Edge of chaos. For x^{l+1} = gWφ(x^l) with W random orthogonal and φ saturating, a mean-field recursion for the layer variance, assuming Gaussian activity, has a transition at g_c = 1 for tanh. Numerically (N = 1000, 100 layers), the end-to-end Jacobian keeps an O(1) fraction of its singular values of order one at g = 1, even with inputs deep in the nonlinear regime (Fig. 7).

Standing in the record

Filed on 2026-10-09 at the owner’s request. LIT-855 (Karkada et al., word2vec dynamics) and LIT-859 (Mainali and Teixeira, in-context learning dynamics) take their stepwise, mode-by-mode learning from it. LIT-862 (Saxe, McClelland and Ganguli 2019) restates its solution without citing it. Neither record held it. It was read on its own merits (NOTE-667).

It is the first appearance of the result that THEORY-186 states from LIT-862. It does not supply what that account’s promote_when asks for, a proof for random initial weights: it treats them by simulation, as the 2019 paper does. What it adds beyond LIT-862 is the general coupled equations, the unbalanced solution, arbitrary depth, and the account of initialisation. The reading produces THEORY-187, on how depth and the initial end-to-end mode strength set the learning time.

It carries anthology-candidate. Half of it is about initialising and pretraining deep networks, which the anthology’s training-optimization and model-stability topics hold. It stays here beside LIT-862 and THEORY-186 as a solvable model of training dynamics.

Backlinks