anthology-of-the-sota

SOTA-415: During a warmup-stable-decay run, estimate the annealed score from a uniform average of recent stable-phase checkpoints instead of launching a decay branch

3 min read

Properties6
StatusProposed
Version3
Consensusemerging
Tagstraining-optimization, analysis-and-evaluation
SourceLIT-684 — Model Merging in Pre-training of Large Language Models, LIT-145 — Scaling Laws and Compute-Optimal Training Beyond Fixed Training Durations
Introduced byLIT-145 — Scaling Laws and Compute-Optimal Training Beyond Fixed Training Durations

Graph View

Source

Li et al. (2025), LIT-684. The practice’s idea is earlier, in Hägele et al. (2024), LIT-145: averaging along a constant-LR trajectory improves the model at no training cost.

What to do

In the stable phase of a warmup-stable-decay run (SOTA-140), take a uniform average (SMA) of the last ~10 checkpoints and evaluate it. Treat the result as an estimate of where an anneal from here would land, for decisions such as when to stop, whether to change data, or how to fit a scaling curve. Launch the actual decay for the model you release.

  • Use a uniform average. Weighted and exponential schemes converge with it late in training. It is also the cheapest thing to get right, which is not the same as having nothing to get right — see below.
  • Keep the window short early. At 204B tokens on a 1.3B/13B MoE, a 32B-token window was 4.6 points below the raw checkpoint. By about 450B tokens the window hardly matters.
  • The window is a hyperparameter, and it does not have to be guessed. The bullet above is a heuristic for tuning one, so “uniform averaging has no hyperparameter” — this practice’s v1 wording — was describing the scheme rather than the choice. SOTA-428 is the alternative: keep two power-function averages during the run and reconstruct any window afterwards by least squares, which works retroactively from stored snapshots at reduced accuracy. That makes the window measurable on runs already finished instead of guessed, and it is how the “by about 450B tokens the window hardly matters” claim could be checked at other scales rather than carried.

Why

Averaging checkpoints along a noisy constant-LR trajectory cancels much of the noise that a decay would anneal away. That is the effect both sources measure. Why it works is not established in the record. The source’s Taylor argument is algebra that holds for any average near a minimum and measures nothing.

Conditions

  • One fork against a real anneal. The “matches annealing” evidence is one 1.3B/13B MoE, four plotted benchmarks, no seeds. At the end the anneal led by about 1.5 MMLU, tied on BBH and GSM8K, and trailed on HumanEval.
  • Private data and architecture, and a composite metric with unstated weights.
  • The learning rate is not swept. The average’s value depends on the stable-phase LR, which the source held at its scaling-law setting.
  • Not a replacement for the decay. Nothing here supports skipping it for a model you will use. SOTA-156 is the practice that argues for no decay at all, by a different mechanism.

Known implementations

  • None recorded.

Backlinks