Source
Karras, Aittala, Lehtinen, Hellsten, Aila and Laine (2023), LIT-720 — ARXIV-2312.02696.
What to do
During training. Maintain two running averages of the weights using a power-function profile rather than an exponential one:
θ̂_γ(t) = β_γ(t)·θ̂_γ(t−1) + (1 − β_γ(t))·θ(t), β_γ(t) = (1 − 1/t)^(γ+1)
at σ_rel = 0.05 and σ_rel = 0.10 (γ = 16.97 and 6.94). Store both
vectors in every snapshot you were already writing. The paper snapshots every
~8M training images.
After training. For whatever averaging length you want, solve the least-squares fit between the stored profiles and the target profile, and take that linear combination of the stored vectors. Then sweep the length like any other evaluation-time parameter.
Two averages, not one, is the whole trick. With two per snapshot,
reconstruction error falls as O(1/n⁴) in the snapshot count and a few dozen
snapshots is “virtually perfect”. With one it still works, “albeit with much
lower accuracy” — which is what makes the retroactive case possible.
LIT-720 (EDM2) introduced both the power-function profile and the
least-squares reconstruction, and the error rate above is its measurement:
reconstruction MSE “in the order of O(1/n⁴)” in the snapshot count. Its sweep
of the learning-rate decay t_ref on ImageNet-512, below, is what the cost
argument rests on.
Why the pre-training guess costs more than it looks
The averaging length has no obvious price: you pick a decay constant, the run proceeds, and the number you report is whatever that choice produced. What the paper measures is that freezing it inflates the apparent sensitivity of other hyperparameters:
learning-rate decay t_ref | EMA length fixed at 13% | EMA length swept post hoc |
|---|---|---|
[30k, 160k] | FID up to 72% worse than optimum | all within 10% |
So a run that reports “the learning-rate decay matters a lot” may be reporting a
fact about the pair, not about the decay. THEORY-111 is that account, and
it is Proposed because this is one measurement.
Three further reasons the length cannot be set once and reused:
- it “differs considerably between the configurations” of the same model family;
- the optimum narrows as the architecture gets better, so a value tuned on an earlier config is both wrong and wrong by more;
- it “slowly shifts towards relatively longer EMA as the training progresses”, which is striking because the definition is already relative to run length.
Why a power function rather than an exponential
Two reasons, both stated: a long exponential average “puts non-negligible weight
on initial stages of training where network parameters are mostly random”; and
longer runs want longer averages, so the profile should stretch with training
time rather than needing to be re-tuned. The power profile gives the
initialization weight exactly zero and is scale-independent. Report the length as
σ_rel — the peak’s width as a fraction of training time — not as γ.
Conditions and what it does not cover
Measured in one setting. ImageNet-512, EDM2’s architecture, FID. The reconstruction arithmetic is setting-independent; the claim that it changes which other knobs matter is not.
Storage, not compute. The cost is two parameter vectors per snapshot instead of one, and no extra training steps. This is what distinguishes it from SOTA-408, whose ImageNet result buys +0.8 top-1 with ten further epochs of averaging — a real compute charge that this approach does not incur, because the averages are accumulated inside the run that was happening anyway. The two are not alternatives measured against each other; they are different cost structures for related ends, and nobody has compared them.
It does not tell you which length to pick. It makes the sweep possible, and the sweep needs an evaluation metric — with FID that reopens everything SOTA-307, SOTA-383 and SOTA-425 say about reading FID differences, and a sweep over averaging lengths scored on a noisy metric is a selection problem, not a measurement. Expect to need the error bar.
Per-tensor lengths are open. Sweeping one tensor’s length against the global optimum improved Config B’s FID from 7.24 to ~6.5 — 10% — with the gain coming from a very short length in one case and a very long one in another, which the authors read as “any global choice is an uneasy compromise”. They did not pursue it. Worth noting that in their final configuration the effect vanished, so this may be a diagnostic for an unfinished architecture rather than a knob to turn.
Related
- SOTA-415 — uniform-average recent checkpoints to estimate an annealed score. It recommends uniform averaging partly because “it has no hyperparameter”; this practice is the argument that the hyperparameter is worth having.
- SOTA-408 — average the tail under a cyclical or high constant learning rate. Same family, different cost structure, and its extra-epochs objection does not apply here.
- SOTA-409 — average across a hyperparameter sweep rather than along one trajectory. Orthogonal: that averages models, this averages time.
- SOTA-217 — align permutations before averaging separately trained networks. Not needed here: every snapshot comes from one trajectory.
Worth naming: this would be the only Active practice in that list. The other
five averaging practices are all Proposed, one of them (SOTA-408) with
converged consensus. The asymmetry is deliberate and narrow — what is Active
here is that you can reconstruct any averaging length from two stored averages,
which is arithmetic with a measured error rate, not that post-hoc sweeping is
known to be the right thing to do everywhere. A reader who takes the status as a
ranking against its neighbours has read it wrong.