Source
Yang, Poggio, Chuang and Ziyin (2025), LIT-861, Lemmas 1–4, Theorem 1 and Propositions 1–3, as read in NOTE-663.
What was actually shown
The setting is a collection of neuron vectors xᵢ ∈ ℝᴰ, updated by xᵢ′ = xᵢ + ηUᵢ(X). The update is equivariant, PU(X) = U(PX) for every finitary permutation P. For gradient descent and for Adam, the paper proves this follows from a permutation-symmetric loss. Two consequences are proved in a few lines each, and the proofs were followed here:
- No splitting, at any step size. If xᵢ = xⱼ, the swap of i and j fixes X, so Uᵢ(X) = Uⱼ(X) and the two stay equal (Lemma 1).
- Bounded relative motion. If ‖U(X) − U(Y)‖ ≤ K‖X − Y‖, apply it to Y = PX with P the swap of i and j. This gives ‖Uᵢ − Uⱼ‖ ≤ K‖xᵢ − xⱼ‖, hence (1 − ηK)‖xᵢ − xⱼ‖ ≤ ‖xᵢ′ − xⱼ′‖ ≤ (1 + ηK)‖xᵢ − xⱼ‖ (Lemma 2). For ηK < 1, distinct neurons stay distinct after any finite number of steps, and the induced map on the neuron set is a bi-Lipschitz homeomorphism (Theorem 1(iii)).
The proof uses K only through the one pair (X, PX). So the reader’s local form also holds: neurons i and j can coincide after a step only if η‖U(X) − U(PX)‖/‖X − PX‖ ≥ 1. For gradient descent this ratio is the averaged Hessian acting on the antisymmetric direction that moves i and j oppositely, and it is bounded by, but can be far below, λ_max. Equivalently, X ↦ X + ηU(X) is injective when ηK < 1. Equivariance turns injectivity on the weights into injectivity on the neurons.
What could have come out otherwise: an equivariant, K-Lipschitz rule with ηK < 1 that maps two distinct neurons to one point. The argument rules it out.
What this does not say
- That training simplifies above 1/K. Above the threshold the guarantee lapses. Nothing shows that neurons then merge, or that the network loses expressivity. The paper’s “topological breakdown” and its two-phase picture with the edge of stability are interpretation. THEORY-039, on later training phases, is not settled by it.
- That neurons never merge at small step sizes. The lower bound compounds to (1 − ηK)ᵗ. Neurons may converge to each other as t → ∞ at any step size. Only coincidence after finitely many steps is excluded.
- That the shape of the neuron cloud is preserved. For finitely many neurons a homeomorphism of the neuron set is just a bijection. Genus, loops and the Betti numbers of a point cloud at a fixed scale can all change under bi-Lipschitz steps whose distortion compounds. The paper’s experiments measure exactly such quantities. The changes they report at large step sizes are fragmentation, which this result does not predict.
- That 1/λ_max is where a network’s neurons start to merge. A global K does not exist for typical networks. The local constant that matters is the curvature across a particular swap, not the top Hessian eigenvalue. For Adam no K has been derived.
- Anything about the weights within a neuron. The result constrains distances between neurons. It does not constrain how far each one travels, or the function the network computes.
Note of 2026-10-13: the asymptotic half
“What this does not say” leaves open whether neurons merge in the limit. Ziyin (2024), LIT-1014, read in NOTE-716 and filed as THEORY-213, addresses it. Permutation symmetry implies a mirror symmetry. The coincident subspace θ_a = θ_b is a stationary condition, as here. Above a threshold weight decay every minimum lies in it, and under strong gradient noise SGD is attracted to it even where it is a saddle. So, on that account, distinct neurons approach each other asymptotically when weight decay or noise is strong, consistent with this result’s bound that they cannot merge in finitely many small steps. The SGD half rests on a second-order, diagonal-Hessian argument.
Note of 2026-10-19: Active
The independent check the promote_when asked for is the promotion audit’s (batch 005). It reproduces Lemmas 1 and 2 from equivariance and the Lipschitz bound alone, and the local form follows from the same identity. The result is elementary enough that the record could hold it as its own derivation. It stays a THEORY sourced to LIT-861, to keep the attribution. Whether QUESTION-058 is now answered for the finite-step half is left to the next review of that question.