Source
Beyer, Zhai and Kolesnikov (2022), LIT-628, §2.
When this applies
You are choosing anything — an architecture detail, a hyperparameter, an epoch count, an augmentation level — on a benchmark whose “validation” split is what results are reported on. ImageNet-1k is the case in front of us; the pattern covers most long-lived public benchmarks.
Do this
Split the training set, not the evaluation set. Beyer et al. train on the first 99% of ImageNet-1k and select on the remaining 1%. The reported numbers then come from a split nothing was chosen on.
The convention is LIT-628’s, stated in its setup rather than tested: Beyer, Zhai and Kolesnikov keep the 1% minival “to encourage the community to stop selecting design choices on the validation (de-facto test) set.” That sentence is both where the recommendation comes from and the whole of the paper’s case for it.
The cost is 1% of training data. The thing bought is that the reported number means what it is usually taken to mean.
Why this is filed at all
Because the alternative is invisible and universal. A public benchmark’s validation split becomes a de-facto test set the moment results are reported on it, and then every subsequent paper’s design choices are selected against it — not by anyone cheating, but because that is the number available. The selection is distributed across the literature rather than happening inside one study, which is exactly why no single paper’s methods section looks wrong.
SOTA-197 asks readers to account for test-set proximity to the training
data. This is the adjacent failure and the opposite direction: proximity of
the selection procedure to the reported split.
Why it is only Proposed
Nobody has measured the effect, including this paper. Beyer et al. assert the practice and its motive in a single sentence and move on; there is no comparison of a search run against a minival versus against the validation split. So what is established is that a credible group thinks the convention is wrong and adopted the fix — which is a reason to consider it and not evidence about its size.
The promote_when asks for the measurement, and the cheap version already
exists as data: any hyperparameter sweep that logged validation and test
numbers separately can be re-analysed for the gap.
Limitations
- 1% of ImageNet-1k is about 12,800 images, which is small enough that the minival is noisy for fine distinctions. The paper does not discuss sizing it, and a slice large enough to separate close candidates costs correspondingly more training data.
- It does not help with a benchmark that has no separate test split at all, which is most of them — the practice makes the reported number honest about selection, and says nothing about the many-papers-one-split problem underneath.
- The record has no measurement of what this is worth, which is stated rather than implied: this document exists because the convention had no representation here, not because its effect is known.