nucleation

THEORY-813: Training a wide network in the lazy, kernel regime does not make its features more collapsed than at initialization, so neural collapse, where it occurs, is something only feature learning produces

2 min read

Properties5
StatusProposed
Tagslearning-theory, representation-learning
SourceLIT-985 — Can Kernel Methods Explain How the Data Affects Neural Collapse?
ExtendsTHEORY-294 — For a wide two-layer ReLU network with zero bias on one-dimensional data whose two classes lie on opposite sides of the origin, the tangent kernel is a constant multiple of the initialization kernel, so lazy training leaves the within-class variability of its features, NC1, where initialization put it
AnswersQUESTION-192 — Does lazy-regime training leave the within-class variability of a network's features at its initial value beyond one-dimensional, zero-bias, two-class data, where the tangent and initialization kernels are not proportional?

Graph View

Source

Kothapalli and Tirer (2024), LIT-985, read in NOTE-774: Theorem 5.1 (the solved case, claim C2) and Figures 2–3 (claim C3, “the NTK is not more collapsed than the NNGP in higher dimensions”, moderate: Gaussian data, two classes, 10 draws).

What is asserted, and by whom

The thesis is that lazy training never lowers within-class variability below its value at initialization, for wide networks in general and not only in the case where the tangent kernel is a within-class rescaling of the initialization kernel. Collapse is then a signature of feature learning. Kothapalli and Tirer state it as the moral of their analysis; the abstract states the higher-dimensional part as an empirical finding.

What the record holds toward it

  • THEORY-294 (Active): the exact result for one-dimensional data with the two classes on opposite sides of the origin and zero bias, where the equality is close to built in.
  • The paper’s Gaussian simulations in higher dimension, which find the NTK less collapsed than the NNGP, consistent with the thesis but not a test of lazy training of finite networks.

What would confirm or reject it

A verified-lazy network on real data whose NC1 stays at its initial value while the same architecture, trained out of the lazy regime, collapses would confirm it; one whose NC1 falls well below its initial value with the kernel verified static would reject it. QUESTION-192 asks the same thing as a question; this entry is the answer Kothapalli and Tirer propose.

Backlinks