Li et al. (2017) — ARXIV-1712.09913
Key takeaways
- Filter normalization is the contribution: scale each random plotting direction filter-wise to the norm of the trained filter. Without it, loss-surface plots ignore batch normalization and the network’s rescaling symmetries, and their sharpness comparisons “may be misleading”
- Under that normalization, sharpness correlates with generalization error — “even when making comparisons across disparate network architectures and training methods”. The conditional is the result
- The convex-to-chaotic transition: “when networks become sufficiently deep, neural loss landscapes quickly transition from being nearly convex to being highly chaotic”, coinciding with a drop in generalization and then a loss of trainability
- Skip connections prevent that transition and “promote flat minimizers”, which is offered as the explanation for why they are necessary at extreme depth
- Non-convexity is quantified as the ratio
|λ_min/λ_max|of the Hessian across the filter-normalized surface (Figure 7). For ResNet-56 the negative eigenvalues stay under 1% of the positive curvatures over a large region - It is affordable: eigenvalues via an implicitly restarted Lanczos method needing only Hessian-vector products, with no explicit Hessian
- SGD optimization trajectories lie in an extremely low-dimensional space
Standing in the anthology
The source of THEORY-011 and of SOTA-011 and SOTA-012.
Read in full for #114; the reading is NOTE-008. All three practices are confirmed.
The paper is also the record’s cleanest example of a methodological correction becoming a result: earlier work had shown sharpness measures are not invariant to network symmetries, and filter normalization is what makes the sharpness/generalization claim sayable at all.