Source
Beyer, Zhai and Kolesnikov (2022), LIT-628.
When this applies
You are training a vision transformer on ImageNet-1k or a dataset of that order, from scratch, and you want a baseline rather than a result — the number another change will be measured against.
Do this
Five changes to the original ViT recipe, in descending order of what the ablation attributes to each at 300 epochs:
| change | worth |
|---|---|
| RandAugment (level 10) + Mixup (p = 0.2) | 6.3 points |
global average pooling instead of a [cls] token | 1.8 |
| batch size 1024 instead of 4096 | 1.4 |
| fixed 2D sin-cos position embeddings instead of learned | 0.4 |
| — linear versus MLP classification head | nothing |
Keep the architecture. Inception-style crop at 224², random horizontal flip.
The recipe and every number in the table are LIT-628’s. Beyer, Zhai and Kolesnikov proposed it as the plain-ViT baseline for ImageNet-1k, and ablated it on ViT-S/16 by reverting one change at a time: at 300 epochs the full recipe scores 80.0%, reverting each change gives 73.7, 78.2, 78.6 and 79.6, and the original ViT recipe scores 67.1. With 90 epochs it reaches 76.5% in 6h30 on a TPUv3-8.
Do not add the things that look like the next step. The paper names them: dropout, stochastic depth, SAM, CutMix, repeated augmentation, blurring, high-resolution fine-tuning, checkpoint averaging, distillation. None is in the baseline and it reaches 80%.
The part worth understanding rather than copying
Augmentation is not buying accuracy, it is buying the right to train longer. Without RandAugment and Mixup the improved recipe goes 73.6 → 73.7 → 73.7 across 90 / 150 / 300 epochs — flat. With them, 76.5 → 78.5 → 80.0. The original recipe behaves the same way: 66.8 → 67.2 → 67.1.
So the four structural changes set the level and the augmentation sets whether the schedule does anything. A practitioner who adopts the structural changes and skips the augmentation because it “only adds a little” gets a model that is done at 90 epochs and will conclude that longer training does not help.
Limitations
-
ViT-S/16 at 224², ImageNet-1k. The authors recommend ViT-B/32 or ViT-B/16 when more compute and data are available, and note that increasing patch size is nearly equivalent to reducing resolution — but the ablation numbers are for S/16 and are not claimed to transfer.
-
The batch-size result is a batch-size-reduction result, which is the opposite of the usual direction and is not explained. 1024 beats 4096 by 1.4 points at fixed epochs. Whether that is an optimisation effect, an interaction with the augmentation, or a learning-rate artefact is untested here.
It is at least consonant with what the record already holds. SOTA-258 says to scale batch size with the token budget rather than with compute or model size, and at a fixed 90 epochs of ImageNet-1k the token budget is fixed, so a larger batch buys nothing it can use. THEORY-026 is the account — critical batch size is set by how much data has been seen — and SOTA-260 recommends the smallest batch that saturates the device. None of that is evidence for this number, and Beyer et al. offer no analysis; it is a reason the result is unsurprising rather than a reason to believe 1024 specifically.
-
convergedrefers to the reachable level, not this recipe. Two concurrent efforts reach comparable accuracy by other routes; that is what makes 80% credible and what makes “these five changes specifically” one group’s finding. -
Every number here is ImageNet-1k top-1, and one of the five changes is known to invert on other tasks. LIT-723 ablates an absolute position embedding in a vision backbone against three benchmarks and gets +0.4 top-1, −0.2 COCO box AP and −0.6 ADE20K mIoU — the classification gain is the same 0.4 points the sin-cos item is worth here, and the dense-task scores move the other way. That is a different architecture and a different encoding, so it does not refute this row. What it does is mark this row as the one to re-measure if the baseline is being used as a backbone for detection or segmentation rather than as the classification baseline it is scoped to be. SOTA-434 is the recommendation on that side.