nucleation

THEORY-797: Deep classifiers trained with MSE loss stay near the least-squares classifier for their own features and move along the central path by the dynamics of continually renormalized gradient flow, whose equal-rate growth of the whitened class-separation singular values is how they reach neural collapse, as Han, Papyan and Donoho propose

2 min read

Properties4
StatusProposed
Tagslearning-theory, representation-learning, loss-landscapes
SourceLIT-974 — Neural Collapse Under MSE Loss: Proximity to and Dynamics on the Central Path
ExtendsTHEORY-278 — Under MSE loss, in Han, Papyan and Donoho's VGG, ResNet and DenseNet runs on five balanced ten-class datasets, the trained classifier stays near the least-squares classifier for its own features from an early epoch; and on that central path, for unconstrained features continually renormalized to identity within-class covariance and without weight decay, gradient flow drives the class-separation singular values to infinity at equal rates, which in that model is neural collapse

Graph View

Source

Han, Papyan & Donoho (2021; ICLR 2022), LIT-974, read in NOTE-783. The paper’s title and abstract present the central path and the dynamics on it as an account of how networks trained with MSE reach neural collapse.

What is asserted

That trained networks, not only the unconstrained-features model, stay near the least-squares classifier for their own features and follow the continually renormalized gradient flow along that path, so that the whitened class-separation singular values grow at a common rate and their ratio tends to 1. Neural collapse is then the limit of those dynamics.

What the record holds toward it

THEORY-278, which is Active, holds the measured half and the derived half. The measured half is that L⊥_LS is negligible from an early epoch in VGG, ResNet and DenseNet on five balanced ten-class datasets. The derived half is that, for unconstrained features renormalized to identity within-class covariance and with no weight decay, gradient flow gives ω_j/(4t/N)^(1/4) → 1 and NC1–NC4. The step between them, that networks train by the renormalized flow, is assumption A3, motivated by batch normalization and never measured.

What this does not say

  • Anything about cross-entropy, which has no closed-form classifier.
  • That collapse is reached in finite training.
  • Why collapse stalls at different levels on different datasets.

What would confirm or reject it

The promote_when, which is the narrowed entry’s former condition. QUESTION-181 asks it, and QUESTION-202 asks the wider question of where neural collapse holds at all.

Backlinks