Arnas Uselis, Andrea Dittadi and Seong Joon Oh (2025), ICML 2025 — ARXIV-2507.07102
Key takeaways
- Diversity, not volume. With n = 3 values per concept and k = 1 or 2 combinations per value, ResNet-50s trained from scratch fit seen pairs almost perfectly and lose a great deal on unseen ones (MNIST digit accuracy drops about 78%). Quadrupling the in-distribution data still leaves drops of 60–80% (Figure 4). Raising n with k = n − 1, or raising k at fixed n, improves generalization (Figure 3).
- Three phases of feature learning (Section 4.2, Figure 5). With few combinations seen, features are spurious and not even decodable. With moderate coverage they become decodable (linear probes on balanced data reach 100%) but not linearly structured. Only with high coverage do they become linearly factored (R² > 0.8) and near-orthogonal across concepts, and zero-shot accuracy on unseen pairs exceeds 90% on most datasets.
- Why factorization matters (Proposition 4.1). If a pair’s embedding is u_{c1} + u_{c2} and the per-concept vectors jointly span 2n − 1 dimensions, then observing k = 2 combinations per value (a diagonal and an off-diagonal cycle) identifies the factors, and a linear classifier then gets all (n − k)·n unseen pairs right.
- Pretrained models are partly there (Section 5). Using factors recovered from k = 2 combinations, DINOv2 and CLIP exceed 90% on some concepts (CLIP on colour, DINOv2 on shape, scale and orientation), and no model reaches the ideal. Probes on their features beat a from-scratch ResNet-50 but still improve with more combinations (Figure 8).
Standing in the record
Filed on 2026-10-03 with the owner’s batch (ADR-027), among its papers on where generalization is carried in a network. Here the carrier is a geometric property of the representation: additivity of concept vectors. It shares its probing protocol with the same group’s intermediate-layer paper (LIT-678), cited for the decodability metric, and declares no relation to it.
Against the record’s representation theories. Linear factorization is a claim about the representation’s geometry up to an additive structure. The record holds that a representation is fixed by its kernel only up to an orthogonal transformation (THEORY-004), and that convergence of representations is convergence of kernels (THEORY-002, from the Platonic Representation Hypothesis, LIT-302). Linear factorization and orthogonality of concept subspaces are invariant under that orthogonal freedom, so they are properties a kernel-level comparison can see. The paper does not make this connection; it is mine. The record’s reading of Park et al.’s linear representation hypothesis (LIT-304) is the language-model counterpart. The paper cites a different Park et al. paper (on categorical and hierarchical concepts), which neither record holds.
Against the learning-theory line. The result sits with data diversity rather than model or data size, the opposite emphasis to the scaling literature it argues with. Its Proposition is a counting argument about when unseen combinations are determined by seen ones under an exact linear model, not a generalization bound.
Anthology. It ends with an instruction for practice (build datasets for
combinatorial diversity, not volume), so anthology-candidate.