LIT-862: A mathematical theory of semantic development in deep neural networks

Andrew M. Saxe, James L. McClelland and Surya Ganguli (2019), Proceedings of the National Academy of Sciences 116(23):11537–11546 — ARXIV-1810.10531, DOI-10.1073/pnas.1820226116

Key takeaways

• The model. Items x (one-hot, or any orthonormal code, so Σx = I) are mapped to feature vectors y by ŷ = W₂W₁x, trained by online gradient descent on squared error. With a small learning rate the epoch-averaged dynamics are τ dW₁/dt = W₂ᵀ(Σyx − W₂W₁Σx) and τ dW₂/dt = (Σyx − W₂W₁Σx)W₁ᵀ, τ = 1/(Pλ), cubic in the weights. Everything is driven by the SVD Σyx = Σ s_α u_α v_αᵀ: v_α (the “object analyzer”) places items on a semantic dimension, u_α (the “feature synthesizer”) says which features that dimension carries, s_α its strength.
• Mode-by-mode, stage-like learning, from depth. If the weights start diagonal in the SVD basis and balanced between the layers, the network’s map is W₂W₁ = U A(t) Vᵀ with a_α(t) = s_α e^{2s_α t/τ} / (e^{2s_α t/τ} − 1 + s_α/a_α⁰) (Eq. 6), so each mode rises by a sigmoid in time (τ/s_α) ln(s_α/ε) (Eq. 10), and the transition takes a vanishing fraction of the total time as ε → 0. A shallow network, Ws, follows b_α(t) = s_α(1 − e^{−t/τ}) + b_α⁰e^{−t/τ}: every mode on the same timescale τ ln(s/ε), exponentially. Hidden representations are h_iα = √a_α(t) v_iα up to a rotation. Random small weights are covered by simulation (Fig. 3C), not by proof; the SI says so.
• Hierarchy gives a falling spectrum. For features diffusing down a tree with flip probability ε per branch, the item-similarity eigenvectors are functions constant below each node (tree wavelets, Haar for binary trees) and the eigenvalue of a level depends only on the level and falls with it, so broad distinctions are learned first. With large branching the timescale of level l grows as √(M_l/Δ_l), M_l the number of nodes at that level. The resulting trajectory reproduces the animals-and-plants MDS picture of Rogers and McClelland’s nonlinear network (Fig. 2).
• Illusory correlations. A feature’s prediction for an item is Σ a_α(t) u_mα v_iα; when the terms differ in sign it rises and then falls (or the reverse) for a time of order Δ, the gap between the two singular values. A shallow network’s predictions are provably monotone.
• Typicality, prototypes, coherence. From the SVD, v_iα = (1/(Ps_α)) Σ_m u_mα o_mi and u_mα = (1/(Ps_α)) Σ_i v_iα o_mi: typicality is projection onto the prototype and the prototype is the typicality-weighted mean, with no circularity. For one category of K_o items and K_f features in noise, recovery has a sharp threshold at C = SNR·K_oK_f/√(N_oN_f) = 1 (a BBP transition, from Benaych-Georges and Nadakuditi). Defining coherence as the singular value, between-category anticorrelation at an intermediate level can make the basic level learned first. A single new feature can regroup an item whose own features did not change (Fig. S2): categories are a global property of the item–feature matrix.
• Structural forms without a prior over forms. For features drawn from a Gaussian Markov random field on a graph, the object analyzers are the covariance’s eigenvectors: cluster indicators, tree wavelets, Fourier modes for a ring (circulant covariance, lowest frequency learned first), approximately so for grids, and extra cross-cutting dimensions.
• Generalisation and representational similarity. A new feature taught for one item is projected to others by h_jᵀh_i/‖h_i‖², so generalisation narrows as differentiation proceeds. Among all weights implementing the same map, the minimum-norm ones are W₁ = R√A Vᵀ, W₂ = U√A Rᵀ with R orthogonal (proved), so two such networks have identical hidden similarity matrices and behavioural similarity is its square; networks from large random weights do not (Fig. 11).

Standing in the record

Filed on 2026-10-09 at the owner’s request, because LIT-860 (Karkada et al. 2026) builds on it and neither record held it. LIT-860 cites it for its nearest precedent, circular hidden geometry from data on a periodic lattice. It was also filed as the source of the stepwise, mode-by-mode learning that LIT-859 (Mainali and Teixeira) and LIT-855 (Karkada et al. 2025) call the Saxe-style picture. The reading qualifies that second reason. Both papers take the dynamics from the authors’ 2014 ICLR paper, Exact solutions to the nonlinear dynamics of learning in deep linear neural networks (arXiv 1312.6120). LIT-859 cites only that paper. LIT-855 cites this one once, in a list, and proves its Lemma 3.1 by reference to 2014. Eq. 6 here is the 2014 solution restated, and this paper’s reference lists do not cite 2014. Neither record holds the 2014 paper. What this paper adds is the link from that dynamics to the structure of the data: hierarchies, rings, coherence and similarity.

It was read on its own merits (NOTE-666). The reading produces THEORY-186, the record’s first sourced statement of the deep linear result. NOTE-658 had observed that the record did not hold one. The reading also stands behind THEORY-182, whose Lemma 3.1 is this dynamics, and it is the precedent for the Fourier geometry of THEORY-183.

It carries anthology-candidate. Its mathematics is about the training dynamics of a network, and the anthology’s training-optimization and concept-geometry topics could hold it. It stays here because its question is semantic cognition: how categories, typicality and similarity structure come from the statistics of experience.