nucleation

LIT-678: Intermediate Layer Classifiers for OOD generalization

3 min read

Properties5
StatusActive — read 2026-10-03 (NOTE-541); worth reading as the measurement that, under distribution shift, a trained classifier's best linear probe is usually not on the penultimate layer: Intermediate Layer Classifiers (linear probes on frozen layer-l features, l ≤ L − 2) beat last-layer retraining in the few-shot setting on most of nine shifted datasets, and, trained only on in-distribution data, raise worst-group accuracy from 79.4% to 87.1% on Waterbirds and 56.0% to 82.0% on CelebA. The proposed reason is that penultimate features move further under shift, especially for minority groups, while intermediate ones stay put. Selection of the layer uses OOD validation data, and the gains for ViTs are small.
Tagsrepresentation-learning, learning-theory, anthology-candidate
Extended byLIT-657 — The Generalization Ridge: Information Flow in Natural Language Generation
Cited as “paper” byNOTE-541 — Intermediate Layer Classifiers for OOD generalization
Cited as “source” byTHEORY-104 — Under distribution shift the best layer for a linear readout lies below the top: across the last layers of a trained network, in-distribution accuracy rises while out-of-distribution accuracy falls

Graph View

Arnas Uselis and Seong Joon Oh (2025), ICLR 2025 — ARXIV-2504.05461

Key takeaways

  • The penultimate layer is not where OOD-useful information is richest. With ample OOD data for the probe, the best intermediate ResNet layer beats last-layer retraining by +16.8, +6.3, +1.6 and +6.3 points on CMNIST, CIFAR-10 (so written; the shifted set is CIFAR-10C), CIFAR-100C and MultiCelebA. For ViTs the differences are +1.1, +1.3 and −0.2 on CMNIST, CIFAR-10C and CIFAR-100C (Section 4.2.1, Figure 3).
  • The gap widens when OOD data are scarce. At under 3% of the OOD probe data, ILCs beat last-layer retraining by +5.7, +3.4 and +27.0 points on Waterbirds, CelebA and MultiCelebA (Section 4.2.2, Figure 4).
  • Zero-shot, it holds too. Probes trained on in-distribution data only give worst-group accuracy Base < Last layer < Best layer: Waterbirds 79.4% → 87.1%, CelebA 56.0% → 82.0%, MultiCelebA 20.8% → 46.0% (Section 4.3.1, Figure 5). Gains on CIFAR-10C are +2 to +5 points in mean accuracy; on ImageNet variants they average +0.34 (ImageNet-R), +0.91 (ImageNet-A) and about +2 (cue-conflict, silhouette) (Figures 6–7).
  • Depth. Layer 7 of 8 (the “pen-penultimate”) beats layer 8 in both settings on every dataset in Figure 8, and the best layer varies by dataset (Section 5.1, C.4).
  • Why. A sensitivity score, the mean pairwise distance from test points to probe points within a group normalized by the probe points’ own mean pairwise distance, is lower at intermediate layers than at the penultimate one for minority groups (Section 5.2, Figure 9).

Standing in the record

Filed on 2026-10-03 with the owner’s batch (ADR-027), among its papers on where in a network generalization is carried. The Generalization Ridge (LIT-657) extends it to language generation and to training dynamics; the visual compositional-generalization study by the same group (LIT-667) borrows its probe protocol.

Against neural collapse. It names neural collapse (LIT-618, stated as THEORY-083 in the record) as a reason last-layer features transfer badly: classes collapsed to their means keep little else. Appendix C.1 measures a layer-wise collapse proxy (Galanti et al.’s class-distance normalized variance) on CIFAR-10C and CIFAR-100C, and finds the network most collapsed at the penultimate layer under shift as well. That is consistent with collapse being strongest where OOD probes do worst, but the paper does not test whether collapse causes the loss. It notes that in Li et al.’s pretrained-model study the most collapsed layer depends on the downstream task, and attributes the difference to task-specific training here.

Against the spectra line. Papyan’s class/cross-class structure (LIT-613) shows feature class means separating from the bulk with depth. This paper’s minority groups, mapped far from training points at the penultimate layer, are the within-class variation that collapse removes. That reading is mine. The paper’s own explanation is feature sensitivity.

Anthology. It carries a direct instruction for practice (probe an intermediate layer, not the penultimate, under distribution shift), so it is plainly anthology-candidate. The anthology’s layer-wise holdings are the closest: that middle transformer layers tolerate deletion and reordering while the first and last do not (ANTH-THEORY-020), and the logit lens (ANTH-LIT-570) for reading intermediate predictions. Neither is cited by the paper.

Backlinks