Yongyi Yang, Tomaso Poggio, Isaac Chuang and Liu Ziyin (2025), arXiv preprint — ARXIV-2510.02670
Key takeaways
- Equivariance alone forbids splitting. If the update commutes with permuting neurons, two neurons with identical weights receive identical updates, so they stay identical at every step size (Lemma 1). For gradient descent this follows from the loss being permutation-symmetric (Proposition 1); for Adam the optimizer’s moment estimates are carried as part of each neuron, and equivariance still holds (Proposition 3).
- A step moves pairs of neurons by a bounded factor. If the whole update map is K-Lipschitz, then for every pair (1 − ηK)‖xᵢ − xⱼ‖ ≤ ‖xᵢ′ − xⱼ′‖ ≤ (1 + ηK)‖xᵢ − xⱼ‖ (Lemma 2). The proof swaps the two neurons and applies the Lipschitz bound to the swap. So for ηK < 1 the map from the neuron set at step t to the set at t + 1 is a bi-Lipschitz homeomorphism (Theorem 1), and it carries the neuron distribution to the next one (Theorem 2). Under a further smoothness condition it is a C¹ diffeomorphism. For ηK ≥ 1 it is still a continuous surjection, and a quotient map when the set is compact.
- 1/K is a sufficient threshold, not an observed transition. Below it no two distinct neurons can coincide after finitely many steps. Above it the theorem permits merging and does not show it happens. For gradient descent with an L-smooth loss, 1/K is the step size that minimises the descent-lemma bound, half the classical stability limit 2/K.
- What the theorem covers in a real network is narrow. With finitely many neurons, a homeomorphism of the neuron set is just a bijection, so “topology is preserved” means “no two neurons merge”. Genus, loops and Betti numbers of the point cloud are scale-dependent, and the theorem does not protect them: the distortion bounds compound over steps. Merging in the limit of infinitely many steps is not excluded at any step size. The global Lipschitz constant K does not exist for the paper’s own sigmoid networks. For Adam the paper proves equivariance but no continuity constant.
- The experiments do not test the theorem. In two- and three-dimensional toy networks, a figure-eight or genus-2 cloud of neurons keeps its shape at a small step size and is torn apart at a larger one. For a two-layer MNIST MLP the Betti numbers of the neuron cloud, at a scale of a quarter of its diameter, stay fixed at a small step size and change at a large one. The change is fragmentation: b₀ rises from 1 to about 6 (GD) or past 50 (Adam). The theory describes merging, and in the continuum a continuous map cannot add components. In the Adam runs, the step size times the measured top Hessian eigenvalue stays below 0.05, far below 1, when the topology changes.
Standing in the record
Filed on 2026-10-09 at the owner’s request, with no stated context. It was filed right after the five papers on the theory of trained networks (LIT-855 to LIT-859, curation entry of that day) and together with arXiv 2602.15029. Read on its own merits.
Read on 2026-10-09 (NOTE-663). The reading is the source of THEORY-184, which states the proved result at its true scope. The record’s account of later training phases, THEORY-039, does not change: the paper adds a sixth two-phase picture, measured in yet another quantity. The empirical two-phase paper it cites, by an overlapping author group, is LIT-370. The anthology holds the edge-of-stability paper it leans on, ANTH-LIT-461.