nucleation

THEORY-788: Training a deep classifier past zero training error generally collapses its last-layer features onto their class means, drives the centred means toward a simplex equiangular tight frame aligned with the classifier, and makes the decision nearest class mean, as Papyan, Han and Donoho propose from their runs

2 min read

Properties5
StatusProposed
Tagsrepresentation-learning, learning-theory, information-theory, anthology-candidate
SourceLIT-618 — Prevalence of Neural Collapse during the terminal phase of deep learning training, LIT-613 — Traces of Class/Cross-Class Structure Pervade Deep Learning Spectra
ExtendsTHEORY-083 — In Papyan, Han and Donoho's runs of VGG, ResNet and DenseNet trained with cross-entropy on seven balanced datasets without augmentation, training past zero error shrinks last-layer within-class variability against between-class variability, and moves the centred class means and the classifier toward a common simplex equiangular tight frame and the decision toward nearest class mean, nearly to completion on the four easier datasets and only partly on ImageNet
AnswersQUESTION-202 — Does neural collapse hold outside its first protocol, under augmentation, label noise, class imbalance and several seeds per setting, and does gradient descent on cross-entropy provably reach it?

Graph View

Source

Papyan, Han and Donoho (2020, PNAS), LIT-618, read in NOTE-476. Their title and discussion present neural collapse (NC1 to NC4) as prevalent in the terminal phase of canonical deep-learning training, not as a property of their particular runs. LIT-613 is the companion reading the narrowed entry also rests on.

What is asserted

That standard training of a deep classifier well past zero training error generally drives within-class variability of last-layer features to zero (NC1), the centred class means to a simplex equiangular tight frame (NC2), the classifier into alignment with them (NC3), and the decision to nearest class mean (NC4), across architectures, datasets and training details.

What the record holds toward it

THEORY-083, Active, is the measured part: in 480 VGG, ResNet and DenseNet runs with cross-entropy on seven balanced datasets without augmentation, the four phenomena moved in the predicted direction, nearly to completion on the four easier datasets and only partly on ImageNet. QUESTION-202 asks whether collapse holds outside that protocol and whether gradient descent on cross-entropy provably reaches it. Under class imbalance the record’s expected geometry is THEORY-286’s, not the simplex ETF, so NC2 as stated here is a claim for balanced classes.

What this does not say

  • Not why collapse happens. The held theorems assume the collapsed state.
  • Not that the terminal phase improves generalization; the measured test gains are small and without seeds (THEORY-083).

What would confirm or reject it

Confirmed by the promote_when: collapse measured outside the original protocol, under augmentation, label noise and several seeds, or a gradient-descent derivation for cross-entropy. Rejected if standard training run well past zero error leaves within-class variability flat or, with balanced classes, the means far from equiangular.

Backlinks