Source
Karras, Aittala, Lehtinen, Hellsten, Aila and Laine (2023), LIT-720, Figure 12 — reported as an aside about learning rates, not as a claim about methodology.
The account
A sensitivity sweep varies one hyperparameter and holds the rest fixed. That is the right experiment when the rest are genuinely independent, and it silently attributes any interaction to the knob being moved. The averaging length is an unusually bad thing to hold fixed, for a structural reason: it is chosen before the run, it is rarely reported, and its own optimum depends on the very quantities being swept — so the fixed value is right for at most one cell of the sweep.
The measurement. EDM2-S on ImageNet-512, learning-rate decay t_ref against
FID:
| averaging length | FID across t_ref ∈ [30k, 160k] |
|---|---|
| fixed at 13% | up to 72% worse than the optimum |
| swept post hoc | all within 10% of the optimum |
A sweet spot still exists — t_ref = 70k — so the decay is not nothing. But a
reader of the first row would conclude the learning-rate decay is a
high-sensitivity hyperparameter worth careful tuning, and a reader of the second
would conclude it is worth getting inside a factor of five. Most of what the first
row measures is the mismatch between a frozen average and the run it was frozen
for.
Why the averaging length in particular. LIT-720 measures three things that make a single fixed value indefensible: the optimum “differs considerably between the configurations” of one model family; it narrows as the architecture improves, so a stale value is wrong by more; and it “slowly shifts towards relatively longer EMA as the training progresses”, even though the length is already defined relative to run length. A knob whose optimum moves with architecture, training duration and the other hyperparameters is not a constant anybody can set once.
What follows if it is right
Published sensitivity curves are a lower bound on how forgiving a hyperparameter is, not an estimate of it. Every such curve that was produced with an averaging scheme fixed by convention — which is to say, most of them — overstates the swept knob’s importance by whatever the interaction contributes.
The direction is usually but not always knowable, and the difference is worth
stating. The swept curve is a pointwise lower bound on the frozen one, since at
each setting it takes the best available averaging length. That alone does not
bound the range: a lower envelope can in principle dip further at one point and
look more sensitive. What makes the bias one-directional in practice is how the
frozen value gets chosen — tuned at the configuration the authors settled on,
which is near the swept curve’s own minimum. When that holds, the two curves share
a minimum and the frozen one has the larger spread. In this paper it holds: 13% is
the optimum at t_ref = 70k, the best setting.
It is cheap to check retroactively. Because the reconstruction in
SOTA-432 works from stored snapshots, the 2×2 can be run on training runs
that already finished, at the cost of storage and evaluation rather than compute.
That is unusual for a methodological claim and is what the promote_when asks
for.
What this does not say
Not that the learning-rate decay does not matter. The sweet spot is real and the paper reports it.
Not a general claim about all frozen hyperparameters. The argument turns on specific properties of the averaging length — chosen pre-run, unreported, with an optimum that tracks everything else. A knob without those properties may be perfectly safe to hold fixed, and this account says nothing about which ones those are.
Not the same as the seed-variance argument. SOTA-307 is about the noise in a reported FID: retraining moves it more than resampling does, so a small gap is inconclusive. This is about bias in a reported sensitivity: the curve is the wrong shape, not noisy. A perfectly estimated FID at every point would leave this intact. THEORY-095 is a third and separate thing — the estimator’s own bias.
Not evidence that uniform averaging is wrong. SOTA-415 recommends a uniform average partly because it has no hyperparameter to tune, and that is a real benefit under a real constraint. What this account says is that the constraint is avoidable, not that the choice made under it was mistaken.