Source
Kothapalli & Tirer (2024, TMLR 2025), LIT-985, read in NOTE-774: Theorem 4.1, Theorem 5.1 with its proof (Appendix D, Eqs. 49–57), Figures 2–3, and Appendices E–G.
What was actually shown
The formula. For features with inner products Q(x, x′) = hᵀh′, the traces of the within-class and between-class covariances are explicit sums over the class blocks of the kernel matrix (Theorem 4.1). So NC1 = tr Σ_W / tr Σ_B can be computed for the features of any kernel without the features themselves.
The theorem. Take a two-layer net in the infinite-width limit with ReLU activation and zero bias variance. Take one-dimensional data, two classes N(μ₁, σ₁²) and N(μ₂, σ₂²) with μ₁ < 0 < μ₂ and |μ_c| ≫ σ_c, so every cross-class product x x′ is negative. Then:
- the NNGP post-activation kernel is (σ_w²/2)xx′ within a class and 0 across (Eq. 54);
- the NTK is (σ_w⁴/2 + σ_w²/2)xx′ within a class and 0 across (Eq. 56);
- so E[NC1(H_NTK)] = E[NC1(H_GP)] = Σ_c[(n_cμ_c² + n_cσ_c²)/N − μ_c²/2] / Σ_c[μ_c²/2 − n_c²μ_c²/N²] + Δ_h.o.t. (Theorem 5.1, Eq. 57).
The NTK describes the features a wide net keeps throughout gradient descent with a small learning rate. The NNGP describes them at initialization. So in this setting lazy training leaves within-class variability exactly where initialization put it. For Erf the same comparison holds approximately, by expansion (Appendix E).
The measurements. On two-class Gaussian mixtures in d₀ up to 128, the NTK’s NC1 tracks the NNGP’s with Erf and exceeds it with ReLU (Figures 2–3). So the NTK is not more collapsed anywhere tested. A data-aware kernel (Seroussi et al.’s equations of state) does fall below the NNGP at width 500, where finite networks also fall (Figure 4). The measurement could have gone the other way: the NTK could have shown lower NC1 with dimension, as the finite nets sometimes do.
What this does not say
- It does not prove the result beyond its setting. The theorem has one input dimension, two classes split by the origin, and zero bias. There the NTK is a within-class rescaling of the NNGP, so the equality is nearly structural. Everything else is simulation on Gaussian mixtures.
- It does not show that feature learning produces collapse. The paper shows the lazy regime does not. Its candidate for the alternative, the data-aware kernel, tracks finite networks on separable data. It fails under class overlap in high dimension and under imbalance, where the two move in opposite directions (Figures 5, 12, 13).
- It does not use LIT-618’s metric or regime. NC1 here is tr Σ_W / tr Σ_B, and the networks are two-layer, binary and MSE-trained for 1000 steps. That is not the terminal phase of deep cross-entropy training that THEORY-083 describes.
- It does not say features become more collapsed than the data. In Appendix F the kernel features are about half as collapsed as the raw inputs, by the relative metric, when the classes are symmetric.
Connections
- THEORY-212 says that with strongly unbalanced layers a two-layer linear net learns with a static kernel and leaves the larger layer task-agnostic. This account is the collapse-side counterpart: a static kernel leaves within-class variability unchanged.
- THEORY-086 describes what kernel-regime gradient descent does fit: the target, eigenspace by eigenspace. Fitting the target in the lazy regime does not reorganize the features. This account adds that in particular it does not collapse them.
- THEORY-278 and THEORY-286 derive collapse in unconstrained-features models, where nothing but the labels shapes the features. This account is the record’s only one where the data enter, and what it finds is a negative.
- THEORY-083. If this account holds generally, the terminal-phase collapse THEORY-083 measures is a signature of training outside the lazy regime.
Note of 2026-10-19: Theorem 5.1, re-derived
Re-derived for the promotion audit (batch 045, ADR-047 v2), in the theorem’s setting. The network is two-layer ReLU in the infinite-width limit with zero bias. The inputs are one-dimensional, in two classes, and every product x·x′ of inputs from different classes is negative.
- Both kernels are homogeneous. For scalar inputs and w ~ N(0, σ_w²), relu(wx)·relu(wx′) is zero whenever x and x′ have opposite signs. When they share a sign it is w²·xx′·1{wx > 0}. So the NNGP post-activation kernel is c₁·xx′ within a class and 0 across. The weight-gradient term of the tangent kernel is E[a²]·P(wx > 0, wx′ > 0)·xx′, which is likewise c₂·xx′ within a class and 0 across. So the tangent kernel is κ times the NNGP kernel, entry by entry, for a constant κ > 0. In the paper’s parametrisation κ = σ_w² + 1 (Eqs. 54 and 56).
- NC1 does not see the constant. By Theorem 4.1, tr Σ_W and tr Σ_B of kernel features are linear functionals of the kernel matrix. Multiplying the kernel by κ multiplies both by κ and leaves their ratio unchanged.
So NC1 of the tangent-kernel features equals NC1 of the initialization-kernel features for every sample that meets the sign condition, not only in expectation. Eq. 57’s closed form follows, with its correction for samples where some cross-class product is not negative. Nothing in the argument survives once the kernels stop being proportional, as they do beyond one dimension. That wider case, which the promote_when asked for, is QUESTION-192.
Note of 2026-10-20: the wider claim, filed Proposed
Under ADR-047 v4 the narrowing-loss sweep (batch 071) filed THEORY-813 as Proposed, extending this entry. It carries the general reading this entry’s title carried before the promotion audit narrowed it: that lazy-regime training does not collapse a wide network’s features in general, so collapse needs feature learning, as Kothapalli and Tirer conclude. This entry is its evidence and is not changed by the filing.