LIT-557: Tensor Programs II: Neural Tangent Kernel for Any Architecture

Yang, Microsoft Research (2020) — ARXIV-2006.14548

Key takeaways

• NTK at initialization, for every standard architecture: the kernel ⟨∇θ f(x), ∇θ f(x̄)⟩ converges almost surely to a computable deterministic limit. This is Jacot et al.’s NTK INIT step (LIT-360), generalized beyond MLPs. The NTK TRAIN step, that the kernel stays fixed during training, is deferred
• NETSOR⊤ adds matrix transposes to TP-I’s language, so backpropagation can be expressed
• The Gradient Independence Assumption, treating Wᵀ in backprop as independent of W, is justified under a Simple GIA Check that most architectures meet. When the check fails, GIA-based calculations can be wrong (§6.3)
• The empirical NTK deviates from the limit roughly as width^(−1/2) for batchnorm-ReLU, transformer and RNN, over 100 seeds and widths 2⁶–2¹³

Standing in the anthology

Filed from #163, as the second step of the Tensor Programs line under TP-IV (LIT-548). It extends TP-I (LIT-558) and the NTK paper (LIT-360). No practice.

The kernel regime this paper generalizes is the one TP-IV shows to be feature-free (THEORY-076). That is the series’ own arc: characterize the kernel limit for every architecture, then show why real networks should not be parametrized into it.

Skimmed, not read: abstract, introduction and contribution statements. No NOTE.