nucleation

QUESTION-202: Does neural collapse hold outside its first protocol, under augmentation, label noise, class imbalance and several seeds per setting, and does gradient descent on cross-entropy provably reach it?

2 min read

Properties5
StatusOpen
Tagsrepresentation-learning, learning-theory
Linelearned-structure
RefinesQUESTION-051 — What structure does training for one classification task leave in a network's last layers, and what does that structure cost for decisions and distributions the network was not trained on?
Answered byTHEORY-788 — Training a deep classifier past zero training error generally collapses its last-layer features onto their class means, drives the centred means toward a simplex equiangular tight frame aligned with the classifier, and makes the decision nearest class mean, as Papyan, Han and Donoho propose from their runs

Graph View

Why it is a question

Papyan, Han & Donoho (LIT-618, read in NOTE-476, whose figures were read from the plots on 2026-10-20) measured the four collapses through the terminal phase of training for VGG, ResNet and DenseNet on seven balanced datasets, with no augmentation, dropout removed and one plotted run per dataset and network. Every measure moved toward collapse, close to completion on the easier datasets and partially on ImageNet (THEORY-083). Until 2026-10-20 the rest was THEORY-083’s promote_when; it moved here when the title was narrowed to those runs.

What would answer it

  • Measurements of NC1 and NC2 through the terminal phase under data augmentation, label noise and several seeds per setting, and of NC1 under class imbalance. Under imbalance the geometry to test is that of THEORY-286, not equiangularity: NC2 is predicted for balanced classes only.
  • A held paper that derives collapse as the limit of gradient descent on cross-entropy. LIT-974 derives it for MSE under a re-whitened flow (THEORY-278), and LIT-994 gives an endpoint, not a gradient-descent derivation (THEORY-286).

It would be answered in the negative if standard training run well past zero error leaves within-class variability of last-layer features flat, or, with balanced classes, leaves the centred class means far from equiangular. Further runs under the original protocol cannot settle it, nor can the theorems, which assume the collapsed end state.

Backlinks