nucleation

QUESTION-051: What structure does training for one classification task leave in a network's last layers, and what does that structure cost for decisions and distributions the network was not trained on?

4 min read

Properties6
StatusOpen
Version2
Tagsrepresentation-learning, learning-theory, information-geometry
Linelearned-structure
Refined byQUESTION-080 — When a trained network's parameters keep moving after its loss has converged, what about the network stays fixed, and is the last-layer geometry of neural collapse among the things that do?, QUESTION-181 — Do trained networks follow the continually renormalized gradient flow toward neural collapse under MSE loss, and does the classifier stay near the least-squares classifier of its features outside Han, Papyan and Donoho's settings?, QUESTION-192 — Does lazy-regime training leave the within-class variability of a network's features at its initial value beyond one-dimensional, zero-bias, two-class data, where the tangent and initialization kernels are not proportional?, QUESTION-202 — Does neural collapse hold outside its first protocol, under augmentation, label noise, class imbalance and several seeds per setting, and does gradient descent on cross-entropy provably reach it?, QUESTION-236 — Does fixed-feature transfer to new classes fall as penultimate class separation rises outside ImageNet-trained ResNet-50s, and does it move with within-class spread when the loss is held fixed?
BearingTHEORY-286 — Under class imbalance the last-layer geometry that unregularized cross-entropy prefers is not the simplex ETF but simplex-encoded-label interpolation: the logits equal the simplex-encoded labels, the classifier and embedding Gram matrices come from that matrix's SVD, any finite weight decay changes the geometry, and training approaches it more slowly as imbalance grows, THEORY-085 — A classifier's class-driven Fisher outliers pair each class's feature mean with that class's own error mean, so K-FAC's single Kronecker product of class-averaged factors adds every cross-class pairing and misplaces them, THEORY-104 — Under distribution shift the best layer for a linear readout lies below the top: across the last layers of a trained network, in-distribution accuracy rises while out-of-distribution accuracy falls, THEORY-246 — Among ResNet-50s trained on ImageNet with nine objectives and stopped early, seven of which beat softmax cross-entropy's top-1 by up to 0.9 points, fixed penultimate features with more class separation transfer worse by linear probe to datasets whose classes are mostly not ImageNet's (Spearman ρ = −0.93 across the nine; mean accuracy 75.9% for softmax to 48.3% for squared error), and within cosine softmax a higher temperature raises separation and lowers that accuracy; the objectives' representations differ mainly in the last third of the network, and full fine-tuning brings mean transfer to within 1.3 points across them, THEORY-294 — For a wide two-layer ReLU network with zero bias on one-dimensional data whose two classes lie on opposite sides of the origin, the tangent kernel is a constant multiple of the initialization kernel, so lazy training leaves the within-class variability of its features, NC1, where initialization put it, THEORY-278 — Under MSE loss, in Han, Papyan and Donoho's VGG, ResNet and DenseNet runs on five balanced ten-class datasets, the trained classifier stays near the least-squares classifier for its own features from an early epoch; and on that central path, for unconstrained features continually renormalized to identity within-class covariance and without weight decay, gradient flow drives the class-separation singular values to infinity at equal rates, which in that model is neural collapse, THEORY-083 — In Papyan, Han and Donoho's runs of VGG, ResNet and DenseNet trained with cross-entropy on seven balanced datasets without augmentation, training past zero error shrinks last-layer within-class variability against between-class variability, and moves the centred class means and the classifier toward a common simplex equiangular tight frame and the decision toward nearest class mean, nearly to completion on the four easier datasets and only partly on ImageNet

Graph View

The question

Training a classifier past zero error drives its last layers toward a specific geometry, in which features collapse onto class means arranged as a simplex. The class means then dominate the curvature. The question is why training reaches this structure, and what the structure removes. The second part means: what can no longer be read from the top layers, and where in depth it can still be read.

Why the record needs it

QUESTION-010 asks whether a chain can keep what one task family needs while destroying what another needs. A trained network’s layers are a chain of maps, so information about any target can only fall along them. What training adds is the choice of which information is spent. That makes these results a measured case for QUESTION-010, though not an answer to it for reconstruction chains. CLAIM-164 is touched: the class-mean span is the cluster that a spectral cut keeps.

What the record holds

  • THEORY-083: features collapse “onto their class means”, toward a simplex equiangular tight frame, and the decision becomes “nearest class mean”. Limit: it does not say why; balanced classes, last layer.
  • THEORY-104: “in-distribution accuracy rises while out-of-distribution accuracy falls” across the last layers, so the best layer for a readout under shift “lies below the top”. Limit: the layer is chosen on out-of-distribution data; the effect is weak for ViTs.
  • THEORY-085: class-driven Fisher outliers “pair each class’s feature mean with that class’s own error mean”. The class is a shared cause, and a class-blind factorization misplaces it. Limit: spectra are judged by eye on one MLP.
  • THEORY-278: under MSE a trained classifier stays near the least-squares classifier for its features, and on that path a re-whitened gradient flow on unconstrained features drives the class-separation singular values to infinity at equal rates, which is collapse. Limit: MSE, unconstrained features and a modified flow, not gradient descent on cross-entropy.
  • THEORY-286: under class imbalance unregularized cross-entropy prefers simplex-encoded-label interpolation, not the simplex ETF, which it is only for balanced or binary data. Limit: an endpoint for unconstrained features; it does not show that gradient descent reaches it.
  • THEORY-294: lazy-regime training of a wide shallow network does not collapse its features, so collapse needs feature learning. Limit: proved for one-dimensional, two-class data; higher dimensions are measured.
  • THEORY-246: among ImageNet-trained ResNet-50s, objectives that raise penultimate class separation (R² 0.349 for softmax to 0.845 for squared error) give fixed features that transfer worse to datasets with other classes (Spearman ρ = −0.93 over nine losses; mean linear transfer 75.9% to 48.3%), and the differences lie in the last few blocks, which fine-tuning rebuilds. Limit: networks early-stopped near 92% training accuracy, not in the terminal phase; one dataset and a correlation across losses, with one temperature sweep; transfer read only at the penultimate layer; for the same classes under input shift the more separated losses do slightly better, so the cost is for new classes.

The derivation for cross-entropy asked for below is still open: the held derivations are for MSE or for an endpoint, and all remove the data. For the second half the record now holds a direct measurement: Kornblith et al. (2021), LIT-1044, read in NOTE-762 and filed as THEORY-246. It measures the cost at the penultimate layer for new class partitions only, short of the terminal phase, and does not read the earlier layers, so the layer-by-layer half of what would answer it is still open.

What would answer it

A derivation showing why gradient descent on cross-entropy reaches collapse. Alongside it, a layer-by-layer measurement relating the degree of collapse to the loss of readability for held-out tasks, with the layer not chosen on test data.

Backlinks