anthology-of-the-sota

LIT-595: VICReg: Variance-Invariance-Covariance Regularization for Self-Supervised Learning

3 min read

Properties4
StatusActive
Tagsrepresentation-and-encoding, model-stability, vision-and-graphics
Compared againstLIT-596 — Barlow Twins: Self-Supervised Learning via Redundancy Reduction
Cited as “source” bySOTA-367 — Or put the anti-collapse constraint in the loss, as a variance floor and a decorrelation term, THEORY-087 — Why a negative-free siamese network avoids collapse is unsettled, and the three accounts contradict each other

Graph View

Bardes et al. (2021) — ARXIV-2105.04906

Key takeaways

  • Collapse is prevented by naming it and penalising it. Three terms: variance, a hinge mean(max(0, γ − std(z_j))) over embedding dimensions with γ = 1, which simply forbids a dimension from going constant; invariance, the mean-squared error between the two views’ embeddings; covariance, driving off-diagonal covariance entries to zero so the dimensions do not encode the same thing.
  • The difference from Barlow Twins is where the regularisation is applied, and the paper is explicit: VICReg regularises each branch separately, while Barlow Twins works on the cross-correlation between the branches. That independence is what lets VICReg drop weight sharing between branches — the two encoders need not even be the same network.
  • The list of things it does not need is the contribution. “Weight sharing between the branches, batch normalization, feature-wise normalization, output quantization, stop gradient, memory banks”. Every item on that list is a mechanism some other method treats as essential.
  • The variance term transplants. The paper reports it “stabilizes the training of other methods and leads to performance improvements” — which makes it the one component of this cluster with evidence of working outside its own method, and the reason SOTA-367 is stated as a component rather than as a method.
  • 73.2% ImageNet top-1, the same figure Barlow Twins reports. Four mechanisms — predictor-plus-EMA, stop-gradient alone, cross-correlation identity, explicit variance floor — all land within a couple of points of each other, which is itself the evidence THEORY-087 rests on.

Standing in the anthology

Unit C of #304. compared_against: LIT-596 because VICReg runs the comparison and names the architectural difference; the converse could not have been run, as Barlow Twins is the earlier paper.

What the comparison with Barlow Twins found. VICReg borrows the decorrelation term from Barlow Twins (LIT-596) and adds the per-branch variance hinge. With shared-weight ResNet-50 branches the two are level (73.2 each on ImageNet linear, the Barlow Twins figure as that paper reports it; 68.6 against 68.7 in the paper’s shorter branch study). They separate when the branches differ: with unshared weights VICReg loses 2.1 points and Barlow Twins 4.5, and with different architectures VICReg leads by 2.8 (ResNet-50/ResNet-101) and 2.3 (ResNet-50/ViT-S). On image-caption pretraining on MS-COCO it also leads on retrieval, image-to-text R@1 33.6 against 31.4.

Worth noting for the record’s own vocabulary: VICReg’s framing splits the field into “contrastive methods and information maximization methods”, which

is a third way of carving up what THEORY-087 treats as two families. The carve-ups do not agree either.

Backlinks