Refinetti et al. (2021) — ARXIV-2102.11742
Key takeaways
A two-layer network with a few hidden units beats kernel methods on a Gaussian-mixture classification task in the high-dimensional limit, which separates feature learning from lazy training.
From the paper’s own abstract:
A recent series of theoretical works showed that the dynamics of neural networks with a certain initialisation are well-captured by kernel methods. Concurrent empirical work demonstrated that kernel methods can come close to the performance of neural networks on some image classification tasks. These results raise the question of whether neural networks only learn successfully if kernels also learn successfully, despite neural networks being more expressive.
Standing in the anthology
Arrived in the imported batch, which brought in the teacher-student line, which solves the learning dynamics exactly in a solvable model.
A reading arrived under this identifier and is not filed. Its title and one
of its authors belong to no paper, and its own opening paragraph says the
work it means is three others. There is no note here because nothing was
read closely enough to place, which is what Deferred says.