nucleation

LIT-974: Neural Collapse Under MSE Loss: Proximity to and Dynamics on the Central Path

2 min read

Properties5
StatusActive — read 2026-10-14 (NOTE-783) in full; worth reading as the record's one derivation of neural-collapse dynamics in closed form. Under MSE loss, a trained classifier stays near the least-squares classifier for its own features (measured), and on that central path a whitened gradient flow of unconstrained features drives every class-separation singular value to infinity as (4t/N)^(1/4) and to equality, which is NC1 and NC2. It also measures collapse under MSE on five datasets and three architectures. Read it knowing that the flow is not gradient descent (features are re-whitened after every step), that the theory has no weight decay while every experiment has it, that it is MSE and not cross-entropy, and that the experiments are LIT-618's protocol with one run per cell.
Version2
Tagsrepresentation-learning, learning-theory, anthology-candidate
Cited as “paper” byNOTE-783 — Han, Papyan & Donoho — Neural Collapse Under MSE Loss
Cited as “source” byTHEORY-278 — Under MSE loss, in Han, Papyan and Donoho's VGG, ResNet and DenseNet runs on five balanced ten-class datasets, the trained classifier stays near the least-squares classifier for its own features from an early epoch; and on that central path, for unconstrained features continually renormalized to identity within-class covariance and without weight decay, gradient flow drives the class-separation singular values to infinity at equal rates, which in that model is neural collapse, THEORY-797 — Deep classifiers trained with MSE loss stay near the least-squares classifier for their own features and move along the central path by the dynamics of continually renormalized gradient flow, whose equal-rate growth of the whitened class-separation singular values is how they reach neural collapse, as Han, Papyan and Donoho propose

Graph View

X. Y. Han, Vardan Papyan, David L. Donoho (2021), ICLR 2022; arXiv:2106.02073.

Key takeaways

From the reading, NOTE-783.

  • Collapse under MSE (Appendix A). NC1–NC4 occur under MSE as under cross-entropy, on MNIST, FashionMNIST, SVHN, CIFAR10 and STL10 with VGG, ResNet and DenseNet (Figures 3–8). Activation collapse is faster under MSE than under cross-entropy (Figure 10). CIFAR100 and ImageNet are left out because MSE needs rescaling there.
  • The decomposition (Theorems 1–2). L = L_NC1 + L_NC2/3 + L⊥_LS, where L⊥_LS = ½ tr{(W̃ − W̃_LS)(Σ̃_T + μ̃_G μ̃_Gᵀ + λI)(W̃ − W̃_LS)ᵀ} ≥ 0 measures how far the classifier is from least squares.
  • The central path is measured (Figure 2). L⊥_LS becomes negligible from an early epoch, in all 15 cells. L_NC2/3 falls first, then the within-class term.
  • The derivation (Lemma 1, Proposition 2, Corollaries 1–2).
    • With λ = 0, on the path, L = ½ Σ_j 1/(ω_j² + C) in the singular values of SNR = Σ_W
    • Under the re-whitened flow, dω_j/dt = (1/N) ω_j/(C + ω_j²)².
    • So ω_j ~ (4t/N)^(1/4) for every j, with ratios tending to 1. That is NC1 and a simplex ETF, and NC3–NC4 follow.
    • The limiting orientation is set by the singular vectors at entry to the path.
  • Not shown. That ordinary training is the re-whitened flow, anything for cross-entropy, or anything with weight decay.

Standing in the record

Filed on 2026-10-12 for QUESTION-051, and read in NOTE-783. It is held as the record’s one derivation of collapse dynamics. That derivation is filed as THEORY-278, at the strength of its model: unconstrained, re-whitened features under MSE with no weight decay. It answers part of QUESTION-051’s first half, for MSE and not for the cross-entropy the question names. It bears on QUESTION-080: in this model collapse is a power-law limit whose orientation is set at the start, not a fixed point. Its MSE runs replicate the measurements of THEORY-083 (LIT-618) under a second loss, with the same protocol.

Backlinks