Yang, Microsoft Research (2020) — ARXIV-2009.10685
Key takeaways
- Free Independence Principle: weights are asymptotically free from the diagonal matrices built from (pre-)activations under any bounded coordinatewise function. This holds for any NETSOR⊤-expressible architecture, at random initialization only. The paper makes no claim about trained weights
- What it justifies: the free-independence assumption used to compute a network’s input-output Jacobian singular-value distribution, the quantity behind dynamical isometry and the training of extremely deep networks. That calculation had been checked empirically but not proved
- A new route to random matrix laws: it re-derives the semicircle and Marchenko–Pastur laws from its Master Theorem, as a benchmark for the method
- The Master Theorem for general tensor programs, which TP-IV then applies to the whole of training
Standing in the anthology
Filed from #163, as the third step of the Tensor Programs line. It
extends TP-II (LIT-557), and TP-IV (LIT-548) extends it. No
practice.
Skimmed, not read: abstract and introduction. No NOTE.