You et al. (2017) — ARXIV-1708.03888
Key takeaways
- The negative result is the opening move: “the current recipe for large batch training (linear learning rate scaling with warm-up) is not general enough and training may diverge”. Warm-up is prior art here — Goyal et al. (LIT-007) — cited, used as baseline, and argued past
- Layer-wise Adaptive Rate Scaling: a per-layer learning rate scaled by
the trust ratio
‖w‖/‖∇w‖, on the observation that the ratio differs by orders of magnitude between layers - The reframing: warm-up manages the problem in time, going slowly until things settle, where the variation actually lives across layers
- AlexNet to batch 8K and ResNet-50 to batch 32K, both without loss in accuracy
- CNNs on ImageNet; no Transformer evidence
Standing in the anthology
Read in full for #114; the reading is NOTE-012.
SOTA-008 — linear warmup for large batch — moved off this note to LIT-007, where warmup comes from. This paper is retained beside it as the one that found the recipe’s ceiling, which a reader of that practice should know.
The trust ratio is the empirical ancestor of the per-layer rate rules this record holds later, µP among them.