nucleation

LIT-234: The Effects of Regularization and Data Augmentation are Class Dependent

3 min read

Properties5
StatusActive — read 2026-10-14 (NOTE-845) in full; worth reading as a clear demonstration, on standard ImageNet ResNet-50 training, that the regularization strength best on average is not best for every class: stronger random crop and changed weight decay raise mean accuracy while some named classes lose 20 points or more, and the shift survives a frozen-backbone transfer to iNaturalist. Read it knowing that the evidence is per-class curves for hand-picked classes over 10–20 runs, that its significance test is uncorrected for 1,000 classes (4.4% rejections at the 5% level), that the weight-decay claim rests on one figure with no numbers, that Theorem 1 does not reach the class-level claim, and that the abstract's iNaturalist example (class #8889, 70% to 30%) is a drop from moderate to strong cropping, not from none.
Version3
Tagslearning-theory, anthology-candidate
Cited as “paper” byNOTE-205 — Balestriero, Bottou & LeCun 2022 — regularization is class-dependent, NOTE-845 — Balestriero, Bottou & LeCun — regularization effects are class dependent
Cited as “source” byTHEORY-510 — How much of an ImageNet image a crop must keep for its label to survive varies widely by class, with architectures agreeing on which classes are fragile, and strengthening random crop raises mean accuracy while some classes lose 20 points or more; how many classes lose is not established

Graph View

Randall Balestriero, Leon Bottou, Yann LeCun (2022), Advances in Neural Information Processing Systems 35 (NeurIPS 2022), pp. 37878–37891 — ARXIV-2204.03632

Key takeaways

From the reading, NOTE-845.

  • Label preservation is class-dependent. Six pretrained ImageNet models, tested on cropped images, keep some classes recognisable at an 8% crop and lose others below about 50%. They agree on which classes are fragile (Figs. 2–3). CutOut and colour jitter behave the same.
  • The average hides per-class losses. Over 20 ResNet-50 runs, moving the random-crop lower bound from 100% to 8% raises mean accuracy from about 70% to 76%. “Academic gown” falls from about 62% to 40% and “barn spider” from about 68% to 46% (Fig. 4).
  • The statistics are thin. A one-sided Welch test per class, uncorrected for 1,000 classes, rejects for 4.4% at the 5% level and 2.1% at the 1% level. The first is not above chance alone. ImageNet’s 50 validation images per class are not accounted for.
  • Weight decay too, on one figure. Varying weight decay over 10 runs sends selected classes’ accuracies in opposite directions (Figs. 5, 12). No numbers are given.
  • It survives transfer. With a frozen backbone and a linear probe on iNaturalist, pretraining crop strength moves selected classes by 30–60 points while the average stays flat near 40% (Fig. 6). The abstract’s class #8889 rises then falls; its 70%→30% is from moderate to strong cropping.
  • Theorem 1 is narrow. Zero training error on augmentations that leave the target’s level sets yields a model that is not f*. It says nothing about which classes pay.

Standing in the record

Filed on 2026-09-26 at the owner’s request, from a list they grouped under the heading Representation learning as a spectral approximation, and read in full on 2026-10-14 in NOTE-845. The reading finds no spectral or self-supervised content. Its only contact with that heading is that the label-change probability under augmentation, which THEORY-228’s bound uses, varies widely across classes.

The reading filed no THEORY from it. What it supports is an instruction for machine-learning practice: evaluate and tune regularization per class or worst class, not only on average. That belongs in the Anthology of the SOTA. Its general point, that per-class risk curves under one complexity knob need not share a minimum, is the bias–variance trade-off illustrated, not a new account. The measurements themselves are findings, not instructions: THEORY-510 (Proposed, 2026-10-16) states the class-dependence of label preservation under cropping and the per-class losses under a rising mean, with the statistics’ limits.

Tagged anthology-candidate (ADR-005): the reading confirms the seed’s judgement that it is chiefly about machine-learning practice. It is kept here by the owner’s decision of 2026-09-26 that new work stays in nucleation until a transfer is judged appropriate (ADR-010).

Backlinks