nucleation

QUESTION-080: When a trained network's parameters keep moving after its loss has converged, what about the network stays fixed, and is the last-layer geometry of neural collapse among the things that do?

3 min read

Properties5
StatusOpen
Tagslearning-theory, loss-landscapes, representation-learning
Linelearned-structure
RefinesQUESTION-051 — What structure does training for one classification task leave in a network's last layers, and what does that structure cost for decisions and distributions the network was not trained on?
BearingTHEORY-211 — In the stochastic-differential-equation model of SGD, with a gradient-noise covariance that depends on the parameters, minibatch noise drives the difference of squared norms along a rescaling symmetry, at a rate proportional to learning rate over batch size, to the point on the symmetry orbit where the gradient-noise traces of the two parameter groups are equal, whenever both groups carry noise; so at stationarity SGD neither keeps the difference fixed, as gradient flow does, nor lets it diffuse, as isotropic noise does, and differs from gradient descent by a finite amount however small the noise

Graph View

The question

THEORY-083 (Papyan, Han & Donoho) says that through the terminal phase of training a classifier’s last-layer features collapse onto their class means, which converge to a simplex equiangular tight frame. Its own summary says “the dynamics are unexplained”. THEORY-209 (Kunin et al.) says that a converged network is not at a point. It keeps moving at a constant speed, sampling a stationary distribution whose shape the optimizer sets.

Both are accounts of what training leaves once it has converged. The question is what is fixed while the parameters move: the function, the features, the last-layer geometry, or only the loss.

Why the record needs it

QUESTION-051 asks what structure training for one task leaves in the last layers. If neural collapse is a fixed point of the features while the parameters keep diffusing, the structure QUESTION-051 asks about is a property of the stationary distribution, not of a single trained network. If collapse erodes or fluctuates during the limiting dynamics, then measurements of it depend on when training was stopped.

What the record holds

  • THEORY-083: collapse through the terminal phase, measured under one protocol with cross-entropy, weight decay and training past zero error.
  • THEORY-209: limiting dynamics under a quadratic minimum. The ResNet-18 tests are resumed training at the last hyperparameters, where the loss has plateaued.
  • The regimes may differ. With cross-entropy on separable data there is no finite minimum without weight decay, so the quadratic assumption fails there. With weight decay there is one, but THEORY-083’s protocol is not the resumed training THEORY-209 measures.
  • THEORY-085: the Hessian’s top eigenspace is made of class outliers tied to class means. That is where THEORY-209’s isotropy was tested, and also where collapse lives.
  • THEORY-210: along symmetry directions, SGD drifts conserved quantities toward balance. That is motion that need not change the features at all.
  • THEORY-278: under MSE with no weight decay, on the central path, collapse is a limit approached at rate t^(1/4), not a fixed point, and the orientation it approaches is set by the singular vectors at the start of the path. The degree of collapse then depends on when training stops, and the shape on where it started.
  • THEORY-286: under class imbalance and weight decay, the last-layer geometry depends on the decay’s strength, and the approach to it is slow and slows with imbalance. So the geometry is not a property of the task alone, and measurements of it depend on when training stops.

What would answer it

Neural-collapse metrics (within-class variability, the simplex ETF geometry, classifier alignment) tracked through limiting dynamics at a plateau, as in THEORY-209’s protocol, with the parameter speed and diffusion measured at the same time. Or a derivation of collapse as a property of SGD’s stationary distribution rather than of a limit point.

Backlinks