Caballero et al. (2022) — ARXIV-2210.14891
Key takeaways
Identifies and formalizes the empirical observation that neural network learning curves often exhibit two distinct scaling regimes separated by a kink or inflection point, causing single power laws (Kaplan et al.) to fail. Proposes the Broken Neural Scaling Law (BNSL) functional form, a smooth composite that interpolates between two power-law regimes and reduces to the standard power law + floor when the break is absent. Demonstrates improved fits across vision, language, and RL benchmarks. Directly relevant to training runs where warm-start conditions (e.g. gossip-initialized weights) produce kinks in the MSE trajectory.
- BNSL functional form. ε(t) = C_∞ + b · t^{-c₀} · (1 + (t/d₁)^{1/f₁})^{-c₁·f₁} For t << d₁: ε(t) ≈ C_∞ + b·t^{-c₀} (early exponent c₀) For t >> d₁: ε(t) ≈ C_∞ + b·d₁^{c₁}·t^{-(c₀+c₁)} (late exponent c₀+c₁) At t = d₁: smooth crossover; f₁ controls how sharp the kink is. Six free parameters: {C_∞, b, c₀, d₁, f₁, c₁}. Holds when: Six parameters; reduces to 3 when d₁ → ∞ (single power law)
- Reduction to Kaplan power-law + floor. When d₁ → ∞ (break point beyond observed range), the BNSL term (1 + (t/d₁)^{1/f₁})^{-c₁·f₁} → 1, and: ε(t) = C_∞ + b·t^{-c₀} which is exactly Kaplan et al.’s L(S) = L_∞ + (S_c/S)^α with c₀ = α and b = S_c^α. Holds when: Single-regime limit
- Chained BNSL for multiple breaks. For K kinks, the BNSL can be chained: the output of one BNSL segment feeds as the input resource to the next. This handles staircase-like learning curves (e.g., emergent abilities in large LMs that appear as discontinuous jumps). Holds when: K-break generalization
- Better fit than single power law on benchmark tasks. BNSL achieves significantly lower residuals than Kaplan power law + floor across vision transformers (ImageNet), language models, and RL benchmarks, particularly for runs that exhibit early rapid improvement followed by slower scaling. Holds when: Empirical; no theoretical guarantee
What the evidence does not cover
- Purely empirical functional form; no mechanistic derivation.
- Six free parameters can overfit short curves; need sufficient range of t.
- Break sharpness f₁ is often poorly identifiable from noisy data.
- Chained BNSL adds parameters rapidly; model selection becomes critical.
- Does not address the Saad-Solla S-curve (exponential) regime — BNSL assumes power-law dynamics throughout.
Standing in the anthology
Read — the reading is NOTE-158. Arrived in the imported batch, which brought in the scaling-law line.