Source
Assran, Balestriero, Duval, Bordes, Misra, Bojanowski, Vincent, Rabbat and Ballas (2022; ICLR 2023), LIT-231, read in full in NOTE-850: §3 with Propositions 1–4 and their proofs (Appendix C), §4 with Tables 1 and 5–11, §5 with Tables 2–4.
What was actually shown
The prior, method by method, at different strengths.
- MSN. The regularizer is the entropy of the batch-mean assignment p̄, and −H(p̄) is D_KL(p̄ ‖ uniform) up to a constant (footnote 2). In the zero-temperature limit with sharpening, σ·L_MSN becomes K-means plus λ Σ_k (N_k/N) log((N_k/N)/(1/K)), a penalty on unequal cluster sizes (Proposition 3). The prior is in the loss.
- SwAV. The Sinkhorn–Knopp projection enforces P1_N = (N/K)1_K, equal cluster sizes, as a constraint (Eq. 8; Proposition 4, a one-paragraph pointer to Wang et al. 2010).
- VICReg. For a simplified loss, α‖Cov(Z) − I‖²_F plus the invariance term, and γ ≫ α, the loss becomes K-means on the embeddings plus an orthogonality penalty on the centroids (Proposition 2). The last step of the proof is argued in words. The uniform prior then follows only from K-means’s tendency toward “roughly the same number of samples” per cluster, which is cited, not shown.
- SimCLR. Nothing is proved. The paper relies on Garrido et al.’s equivalence with VICReg “under mild conditions”.
The effect. Each method was pretrained on ImageNet-1K twice: with 960 classes per batch, and with the fewest classes that fill a batch (8 for SimCLR, 2 for VICReg and MSN), keeping each image’s marginal sampling probability unchanged. Linear ImageNet accuracy fell by 11.1 (SimCLR), 17.7 (MSN) and 17.7 (VICReg) points, and by 0.8 (data2vec) and 0.1 (MAE). Places205 and Clevr/Count fell for the first three, while depth tasks moved by about two points or less for all. This could have come out otherwise: the instance-based methods, which compute no batch statistics, were run under the same sampling and did not degrade.
Changing the prior. Replacing MSN’s uniform target with the true class distribution of iNaturalist-18 raised iNat18 linear accuracy from 29.1 to 32.1, and a hand-set power law (τ = 0.25) to 30.1. The same power law on ImageNet cost 15–20 points on the semantic tasks. So the target over cluster sizes affects what is learned, in the direction the account predicts.
What this does not say
- It does not say every volume-maximizing objective carries the prior. It is exact for MSN and SwAV. For VICReg it rests on a reduction of a simplified loss plus a cited property of K-means, and for SimCLR on a cited equivalence.
- It does not say that natural class imbalance causes the drops reported on long-tailed data. The experiment uses two to eight classes per batch from a balanced dataset. Its size at natural imbalance is not measured.
- It does not isolate the volume term. The methods compared differ in architecture (ResNet-50, ViT-B, ViT-L) and in objective, and data2vec avoids collapse by other means. The contrast is within each method, balanced against imbalanced.
- It is not an instruction. That a matched prior helps on long-tailed data is advice for practice, and belongs in the anthology. The gains are single runs of 1–3 points, and the best one uses labels.
- It does not say what the representation is. The account is about the regularizer’s preference over partitions of a batch. What the learned eigenfunctions or spectral embedding are is the subject of the accounts below.
Connections
- THEORY-305. Reads SimCLR’s loss as a graph-Laplacian form, spectral clustering only in a sense that admits any regularizer. This account says what a particular regularizer prefers. K-means and its spectral relaxation are linked by the result Proposition 1 cites (Zha et al. 2001).
- THEORY-007, THEORY-009 and THEORY-228. The spectral account of the positive-pair graph, its eigenfunctions and its probe guarantees. Those concern which subspace the invariance term selects. This one concerns a bias of the anti-collapse term over partitions, and is compatible with them.
Note of 2026-10-19: the prior in MSN and SwAV, checked
MSN’s regularizer is the entropy of the batch-mean assignment p̄ over K prototypes. Then
D_KL(p̄ ‖ U_K) = Σ_k p̄_k log(p̄_k K) = log K − H(p̄),
so −H(p̄) = D_KL(p̄ ‖ U_K) − log K. Maximizing H(p̄) is minimizing the KL from the soft cluster sizes p̄ to uniform, which is LIT-231’s footnote 2. Proposition 3 sharpens this to hard cluster sizes at zero temperature, and the title needs only the identity. SwAV’s Sinkhorn–Knopp step projects the assignment matrix onto matrices whose prototype sums are all N/K (Eq. 8), so equal soft cluster sizes are a constraint of the method. Neither half needs Propositions 2 or 4, which are the sketches NOTE-850 flags.
Note of 2026-10-20: the wider claim, filed Proposed
Under ADR-047 v4 the narrowing-loss sweep (batch 070) filed THEORY-805 as Proposed, extending this entry. It carries the general reading this entry’s title carried before batch 033 narrowed it: that volume-maximizing joint-embedding objectives as a class, VICReg and SimCLR included, carry a prior toward equal-sized clusters that costs them semantic transfer on class-imbalanced data, natural long tails included, as Assran et al. argue. This entry is its evidence and is not changed by the filing.