anthology-of-the-sota

LIT-145: Scaling Laws and Compute-Optimal Training Beyond Fixed Training Durations

2 min read

Properties6
StatusActive
Tagstraining-optimization
Extended byLIT-684 — Model Merging in Pre-training of Large Language Models
Compared againstLIT-682 — ScheduleFree+: Scaling Learning-Rate-Free & Schedule-Free Learning to Large Language Models
Cited as “introduced by” bySOTA-415 — During a warmup-stable-decay run, estimate the annealed score from a uniform average of recent stable-phase checkpoints instead of launching a decay branch
Cited as “source” bySOTA-140 — Use a warmup-stable-decay schedule: hold the learning rate, then decay it sharply over the final 10–20% of tokens, SOTA-141 — Decay the learning rate linearly all the way to zero, SOTA-415 — During a warmup-stable-decay run, estimate the annealed score from a uniform average of recent stable-phase checkpoints instead of launching a decay branch

Graph View

Hägele et al., EPFL and Hugging Face (2024) — ARXIV-2405.18392

Key takeaways

  • Cosine schedules tie the result to a training length chosen in advance, so every point on a scaling law is its own run. A constant learning rate followed by a cooldown scales as reliably and predictably as cosine and reaches the same or better loss.
  • Because the cooldown can be launched from any point, all intermediate checkpoints of one long run become valid data points, which cuts the compute of a scaling-law study by a large factor.
  • The cooldown’s length and shape matter; a cooldown of a modest fraction of the run, decaying to zero with a 1-sqrt shape, is what the paper recommends.
  • Stochastic weight averaging along the trajectory yields improved performance at every scale tested without extra training compute.

Standing in the anthology

The controlled evidence behind SOTA-140: LIT-144 shipped the schedule, this paper measured it against cosine across scales and durations. Also the record’s first note on weight averaging as a training-time tool.

ScheduleFree+ (LIT-682) ran the warmup-stable-decay schedule this paper validated as a baseline, from 120M to 2B parameters at 20 and 100 tokens per parameter, and WSD lost to it at every size. The same paper found WSD “consistently outperformed by the linear decay schedule” when both are tuned to the horizon. Two caveats attach. That paper attributes WSD to Zhai et al. and Hu et al. and describes a linear anneal over about 10% of the run, not this paper’s 1-sqrt cooldown. It cites this paper only for checkpoint averaging and for the Schedule-Free limitations its modifications address.

Backlinks