Source
Balestriero, Bottou & LeCun (2022), LIT-234 (NeurIPS 2022), read in full in NOTE-845. NOTE-845 filed no THEORY, because what the paper supports for practice (tune regularisation per class) belongs in the anthology, and its general point is the bias–variance trade-off illustrated. This account files the measurement the reading rates strong (C1) with the one it rates moderate (C2), which are findings about classes and augmentations, not instructions, and leaves the practice where the NOTE put it.
What was actually shown
Label preservation is class-dependent (§2.2, Figs. 2–3, 10–11). Pretrained models are evaluated on cropped test images. Some classes stay recognisable down to an 8% crop. For others accuracy starts falling at about 50%, while the average turns at about 30%. Six architectures agree on which classes are fragile. CutOut and colour jitter behave the same.
The average hides per-class losses (§2.3, Fig. 4). ResNet-50 with the random-crop lower bound τ from 100% to 8%, averaged over 20 runs. Mean test accuracy rises from about 70% to about 76%. “Academic gown” falls from about 62% to 40%, “barn spider” from about 68% to 46%.
It survives transfer (§2.5, Fig. 6). With a frozen ImageNet backbone and a linear probe on iNaturalist, pretraining crop strength moves selected classes by 30–60 points while the average stays near 37–40%.
What could have come out otherwise. The classes could have lost recognisability at the same crop, and every class could have gained with the mean. Neither happened for the classes shown.
What this does not say
- How many classes lose. Figs. 4–6 show four pairs of classes chosen to illustrate divergence. A one-sided Welch test per class rejects for 4.4% of classes at the 5% level and 2.1% at the 1% level, uncorrected for 1,000 tests; 4.4% at 5% is not, alone, more than chance, and 2.1% at 1% is about twice it. With 50 validation images per class one image moves a class by 2 points, a noise the test over training runs does not count. The printed null hypothesis is reversed (NOTE-845).
- That regularisation’s cost must fall unevenly. The paper asserts it from these figures. Theorem 1 says only that a zero-error fit to augmentations that leave the target’s level sets is not the target, and it needs a group of transformations, which random crop is not.
- That weight decay does the same. Shown on one figure with no numbers or test (C3, weak).
- That the class losses are caused by label-destroying crops. The two measurements are made separately; correlating each class’s crop fragility with its loss under training is the open test.
- What to do about it. Tuning or reporting per class is practice and belongs to the anthology.
Connections
- THEORY-228 bounds linear-probe error by the probability that augmentation changes the label, divided by a spectral gap. This account shows that probability is far from uniform across classes, which bears on how that bound’s error is distributed. It is a remark, not a result the paper draws (NOTE-845).
- The worst-group literature (group DRO, spurious correlations) measures the same kind of per-group regression; the paper does not engage it, and the record holds none of it.