nucleation

LIT-985: Can Kernel Methods Explain How the Data Affects Neural Collapse?

2 min read

Properties5
StatusActive — read 2026-10-14 (NOTE-774) in full; worth reading as the record's one account of neural collapse in which the data enter, and for its negative result. In the case it can solve, lazy-regime training of a wide two-layer net leaves the within-class variability of the features exactly where initialization put it, because there the NTK is a within-class rescaling of the NNGP kernel. It also gives NC1 as a closed function of any kernel matrix. Read it knowing that the theorem covers one-dimensional data with two classes on opposite sides of the origin and zero bias, that the higher-dimensional claim is simulation (and for ReLU the NTK is less collapsed, not equal), that its NC1 is tr Σ_W / tr Σ_B on shallow MSE-trained binary nets, and that the data-aware kernel it proposes fails under class overlap and imbalance. Its printed Eq. 16 appears to keep a cross-class term the kernel sets to zero.
Version2
Tagsrepresentation-learning, learning-theory, anthology-candidate
Cited as “paper” byNOTE-774 — Kothapalli & Tirer — Can Kernel Methods Explain How the Data Affects Neural Collapse?
Cited as “source” byTHEORY-294 — For a wide two-layer ReLU network with zero bias on one-dimensional data whose two classes lie on opposite sides of the origin, the tangent kernel is a constant multiple of the initialization kernel, so lazy training leaves the within-class variability of its features, NC1, where initialization put it, THEORY-813 — Training a wide network in the lazy, kernel regime does not make its features more collapsed than at initialization, so neural collapse, where it occurs, is something only feature learning produces

Graph View

Vignesh Kothapalli, Tom Tirer (2024), Transactions on Machine Learning Research (2025); arXiv:2406.02105.

Key takeaways

From the reading, NOTE-774.

  • NC1 from a kernel (Theorem 4.1). tr Σ_W and tr Σ_B of kernel-defined features are sums over the class blocks of the kernel matrix. So NC1 = tr Σ_W / tr Σ_B can be computed for the NNGP, the NTK or any other kernel, with the data in.
  • The lazy regime does not collapse (Theorem 5.1). The setting is one-dimensional data, two classes N(μ_c, σ_c²) on opposite sides of 0, ReLU, and σ_b → 0. There the NNGP kernel is (σ_w²/2)xx′ within a class and 0 across, and the NTK is (σ_w⁴/2 + σ_w²/2)xx′ within a class and 0 across. So their expected NC1 is equal.
  • Beyond the theorem (Figures 2–3). On Gaussian mixtures up to d₀ = 128 the NTK tracks the NNGP with Erf, and is less collapsed with ReLU. Erf gives about ten times lower NC1 than ReLU at d₀ = 1.
  • A data-aware kernel (§6, Appendix G). Seroussi et al.’s equations of state lower NC1 below the NNGP at width 500, toward finite networks, for separable data. With overlapping classes in d₀ ≥ 64, finite networks’ NC1 is about twice the kernel’s, and under imbalance the two move in opposite directions.
  • Raw NC1 can mislead (Appendix F). Relative to the inputs’ own NC1, the ReLU kernel features are about half as collapsed as the data when the classes are symmetric. With well-separated inputs the data are more collapsed than the features.

Standing in the record

Filed on 2026-10-12 for QUESTION-051, and read in NOTE-774. Its lazy-regime result is filed as THEORY-294, at the strength of its narrow theorem. It is the record’s one account of collapse in which the data enter. It bears on QUESTION-051’s first half negatively: whatever produces collapse, the lazy regime does not, in the case that can be computed. It sits beside THEORY-212’s account of static-kernel learning. It also marks the shared limit of the unconstrained-features accounts filed from LIT-974 and LIT-994 (THEORY-278, THEORY-286), which cannot see the data.

Backlinks