nucleation

LIT-231: The Hidden Uniform Cluster Prior in Self-Supervised Learning

3 min read

Properties5
StatusActive — read 2026-10-14 (NOTE-850) in full; worth reading for the clearest statement that a joint-embedding objective's anti-collapse term is not neutral: MSN's mean-entropy regularizer is a KL penalty pulling cluster sizes toward uniform, and SwAV enforces equal sizes outright. Read it knowing that the same prior is reached for VICReg only through an informal reduction of a simplified loss to K-means, and for SimCLR only through a cited equivalence; that the imbalance experiment uses two to eight ImageNet classes per batch, not a naturally long-tailed dataset; and that the iNat18 gains are single runs of 1–3 points, the largest using the true class labels. The account is THEORY-306; the practical advice belongs in the anthology.
Version2
Tagsrepresentation-learning, probabilistic-modeling, anthology-candidate
Cited as “paper” byNOTE-201 — Assran et al. 2022 — hidden uniform cluster prior in SSL, NOTE-850 — Assran et al. — The Hidden Uniform Cluster Prior in Self-Supervised Learning
Cited as “source” byTHEORY-306 — MSN's objective penalizes unequal cluster sizes explicitly and SwAV's enforces equal sizes, and in one study with mini-batches of two to eight ImageNet classes, SimCLR, MSN and VICReg lost 11 to 18 points of ImageNet linear accuracy while MAE and data2vec lost under one, THEORY-805 — Volume-maximizing joint-embedding objectives as a class, VICReg and SimCLR as well as MSN and SwAV, carry a prior toward equal-sized clusters, and that prior costs them semantic transfer when the pretraining data are class-imbalanced, including on naturally long-tailed data

Graph View

Mahmoud Assran, Randall Balestriero, Quentin Duval, Florian Bordes, Ishan Misra, Piotr Bojanowski, et al. (2022), International Conference on Learning Representations (ICLR 2023, poster) — ARXIV-2210.07277

Key takeaways

From the reading, NOTE-850.

  • The anti-collapse term prefers equal clusters. MSN’s regularizer, the entropy of the mean assignment, is a KL divergence to the uniform distribution plus a constant; in the zero-temperature limit the loss is K-means plus a penalty on unequal cluster sizes (Proposition 3). SwAV’s Sinkhorn step enforces equal sizes outright (Proposition 4).
  • For VICReg and SimCLR the link is weaker. VICReg, simplified and with γ ≫ α, reduces to K-means plus a centroid-orthogonality term, by an argument whose last step is informal (Proposition 2). The uniform prior then rests on K-means’s cited preference for equal clusters. SimCLR is covered only through Garrido et al.’s equivalence.
  • Batches of few classes hurt these methods. With two to eight ImageNet classes per batch (same marginal sampling), ImageNet linear accuracy falls by 11.1 (SimCLR), 17.7 (MSN) and 17.7 (VICReg) points, and by 0.8 and 0.1 for data2vec and MAE, which compute no batch statistics. Depth tasks barely move. Architectures differ across methods.
  • PMSN. MSN with a KL to any target over cluster sizes. On iNat18, the true class distribution gives +3.0 and a hand-set power law +1.0 on iNat18 linear accuracy; the same power law on ImageNet costs 15–20 points. Single runs.
  • Table 1 mislabels a column. Its CIFAR100 figures are the CIFAR-100 1% results of the appendix tables, and a few cells differ from those tables.

Standing in the record

Filed on 2026-09-26 at the owner’s request, from a list they grouped under the heading Representation learning as a spectral approximation, with a skim (NOTE-201). Read in full on 2026-10-14 in NOTE-850, which supersedes the skim. The account filed from it is THEORY-306: volume-maximizing objectives carry an equal-cluster prior, stated at the strength the paper shows it for each method. It sits beside the record’s spectral accounts of self-supervised objectives (THEORY-007, THEORY-228, THEORY-305), which concern what the invariance term selects; this one concerns what the anti-collapse term prefers.

Tagged anthology-candidate (ADR-005): the paper’s practical advice, to match the prior to the class distribution when pretraining on long-tailed data, is an instruction for machine-learning practice and belongs in the anthology. It is kept here by the owner’s decision of 2026-09-26 that new work stays in nucleation until a transfer is judged appropriate (ADR-010). The anthology was not checked for this paper in the reading.

Backlinks