nucleation

THEORY-805: Volume-maximizing joint-embedding objectives as a class, VICReg and SimCLR as well as MSN and SwAV, carry a prior toward equal-sized clusters, and that prior costs them semantic transfer when the pretraining data are class-imbalanced, including on naturally long-tailed data

2 min read

Properties5
StatusProposed
Tagsrepresentation-learning, probabilistic-modeling
SourceLIT-231 — The Hidden Uniform Cluster Prior in Self-Supervised Learning
ExtendsTHEORY-306 — MSN's objective penalizes unequal cluster sizes explicitly and SwAV's enforces equal sizes, and in one study with mini-batches of two to eight ImageNet classes, SimCLR, MSN and VICReg lost 11 to 18 points of ImageNet linear accuracy while MAE and data2vec lost under one
AnswersQUESTION-171 — Do volume-maximizing joint-embedding objectives prefer equal-sized clusters as they are actually trained, and does the preference cost semantic transfer on naturally long-tailed data?

Graph View

Source

Assran et al. (2022), LIT-231, §§3–5 and Appendix C, read in NOTE-850. The class-wide prior and its explanation of the degradation joint-embedding methods show on long-tailed data are the paper’s thesis; NOTE-850 rates both halves weak beyond MSN and SwAV and beyond batches of two to eight classes.

What is asserted, and by whom

The thesis has two halves. The prior: every volume-maximizing joint-embedding objective, not only those that state it, prefers equal-sized clusters (NOTE-850, C3 and C4, both weak: VICReg by Proposition 2 for a simplified loss with an informal last step, SimCLR by a cited equivalence). The effect: that prior is why these methods lose semantic transfer on class-imbalanced data, natural long tails included (C7, weak, inferred from batches of two to eight classes). Assran et al. assert both.

What the record holds toward it

  • THEORY-306 (Active): the explicit prior in MSN’s loss and SwAV’s equipartition, and 11 to 18 points of ImageNet linear accuracy lost by SimCLR, MSN and VICReg with two to eight classes per batch, against under one for MAE and data2vec, one run per cell.
  • Matching the prior to the class distribution helps on long-tailed pretraining (C6, moderate), which fits the thesis without testing it, since the gain was obtained with the true or a guessed class prior.

What would confirm or reject it

A derivation or measurement of the prior for VICReg and SimCLR as trained, and the imbalance effect on naturally long-tailed data with seeds, confirm it; learned partitions that track unequal classes as well as an instance-based method’s reject it for that method. QUESTION-171 asks the same thing as a question; this entry is the answer Assran et al. propose.

Backlinks