Yang, Microsoft Research (2019) — ARXIV-1910.12478
Key takeaways
- Every standard architecture is a GP at initialization, in the infinite-width limit. This covers RNNs (with a notion of GP over variable-length outputs), batchnorm, layernorm and attention, which were new here
- NETSOR, a language of matrix multiplications and coordinatewise nonlinearities that can express these forward passes. The GP result holds for anything it expresses
- No weight transposes in the forward pass. Architectures that use
WandWᵀtogether are deferred to later papers - Reference kernels for a simple RNN, GRU, transformer and batchnorm-ReLU network (GP4A)
Standing in the anthology
Filed from #163 to complete the Tensor Programs series under TP-IV
(LIT-548), where µP comes from. This paper holds no practice, and none is
filed. Its role is as the first step in the line TP-I → TP-II
(LIT-557) → TP-III (LIT-556) → TP-IV → TP-V (LIT-148).
Skimmed, not read: abstract, introduction and contribution statements. No NOTE is filed. The theorems are summarized here from the paper’s own statements of them, not from checking the proofs.