Dey et al., Cerebras with Princeton, Harvard and ETH Zürich (2025) — ARXIV-2505.01618
Key takeaways
- Parametrizations are rules for how model and optimizer hyperparameters change with size. Studying compute efficiency under different ones, the paper finds that some — including µP as commonly used — do not transfer the optimal base learning rate across changes in depth, so a value tuned shallow is wrong deep.
- A second failure is lazy learning: with the wrong depth scaling, deep layers barely move from initialisation and the network behaves like a shallower one.
- CompleteP — the parametrization with depth exponent α = 1 — achieves both depth-wise transfer and non-lazy learning in every layer. Against µP it saves 11.8% of FLOPs for optimally shaped models and 34.4% for a 179-layer model.
- A wider range of width-to-depth ratios stays compute-efficient under it, which frees model shape to follow hardware constraints.
Standing in the anthology
The source of SOTA-144, the depth step of the chain that starts at LIT-148. Proposed until a second group reports transfer across depth with it; the
depth question is the one LIT-119 answered by ablation (SOTA-125), which is what this would let a team skip.