nucleation

LIT-994: Imbalance Trouble: Revisiting Neural-Collapse Geometry

2 min read

Properties5
StatusActive — read 2026-10-14 (NOTE-764) in full; worth reading as the statement of what neural collapse becomes under class imbalance. In the unconstrained-features max-margin problem, the logits equal the simplex-encoded labels at any imbalance. The classifier and embedding geometries come from that matrix's SVD, and they are the simplex ETF only for balanced or binary data. Majority classifiers come out longer and minority classifiers closer together, and the minority-collapse threshold is given in closed form. Read it knowing that gradient descent is only conjectured to reach this geometry, that under any finite weight decay the geometry differs and depends on its strength, that convergence is slow and slower with imbalance, and that the deep-net runs are one seed per cell with weight decay on.
Version2
Tagsrepresentation-learning, learning-theory, anthology-candidate
Cited as “paper” byNOTE-764 — Thrampoulidis et al. — Imbalance Trouble
Cited as “source” byTHEORY-286 — Under class imbalance the last-layer geometry that unregularized cross-entropy prefers is not the simplex ETF but simplex-encoded-label interpolation: the logits equal the simplex-encoded labels, the classifier and embedding Gram matrices come from that matrix's SVD, any finite weight decay changes the geometry, and training approaches it more slowly as imbalance grows, THEORY-738 — In the unconstrained-features max-margin problem with k classes under step imbalance and feature dimension at least k − 1, the unique optimal logits are the simplex-encoded labels, every margin equals one, and the classifier and embedding Gram matrices are the two square-root factors of that matrix's singular value decomposition; the geometry is the simplex equiangular tight frame exactly when the classes are balanced or there are two of them

Graph View

Christos Thrampoulidis, Ganesh R. Kini, Vala Vakilian and Tina Behnia (2022), Advances in Neural Information Processing Systems 35 (NeurIPS 2022).

Key takeaways

From the reading, NOTE-764.

  • The invariant is the logits (Theorem 1). For the unconstrained-features max-margin problem at any (R, ρ)-STEP imbalance, ŴᵀĤ = Ẑ, with Ẑ[c, i] = 1 − 1/k for the true class and −1/k otherwise. Then ĤᵀĤ = UΛUᵀ and ŴᵀŴ = VΛVᵀ from Ẑᵀ = UΛVᵀ, and embeddings collapse to class means.
  • The ETF is the balanced special case (Corollary 1.3). Otherwise, majority classifiers are longer than minority ones (Lemma 3.1). The minority classifiers’ cosine, (R − 7)/(R − 7 + 2k(2 + √((R+1)/2))), rises with R and reaches 1 only as R → ∞ (Eq. 4).
  • Regularization matters under imbalance (Propositions 1–2). No finite ridge gives SELI; the ridge-regularized geometry depends on λ, and tends to SELI as λ → 0. With balanced data it is the ETF at every λ.
  • Minority collapse (§H). It does not occur for small regularization at finite imbalance. It needs R above a threshold that rises with the minority fraction and with 1/λ.
  • Measured (Figures 2, 4–7, 13; Table 3).
    • In the unconstrained-features model and in ResNet-18 and VGG-13 on imbalanced MNIST, Fashion-MNIST and CIFAR10, NC1 holds.
    • The geometry ends closer to SELI than to the ETF.
    • The approach slows as R grows, and is worse for embeddings than for classifiers.
    • Minority test error on CIFAR10 rises from 13.8% at R = 1 to 98.1% at R = 100.
  • Not shown. That gradient descent reaches SELI, which is conjecture C1. Nothing is run with more than one seed.

Standing in the record

Filed on 2026-10-12 for QUESTION-051, and read in NOTE-764. Its geometry is filed as THEORY-286. It answers part of QUESTION-051’s first half: under cross-entropy, the preferred last-layer geometry is fixed by the labels’ simplex encoding, and the simplex ETF of THEORY-083 is its balanced case. It bears on QUESTION-080: under weight decay and imbalance the end geometry depends on the regularizer, and the slow approach makes measured geometry depend on when training stops. It shows that THEORY-083’s promote_when, which counts imbalanced replications as tests of equiangularity, needs restricting to balanced classes.

Backlinks