Source
Grill et al. (2020), LIT-594 — ARXIV-2006.07733, and Chen & He (2020), LIT-593 — ARXIV-2011.10566.
The claim
You can drop the negatives entirely. Match two augmented views of the same input and nothing else — provided the two branches are not the same function.
The minimal recipe, which is SimSiam’s:
- one encoder, shared weights, both views through it;
- a small prediction MLP on one branch only;
- a stop-gradient on the other branch.
BYOL adds a third asymmetry — the unpredicted branch is an EMA copy rather than the same weights — and reaches 74.3% ImageNet linear against SimSiam’s 67.7% at 100 epochs. The EMA helps. SimSiam’s result is that it is not required.
The recommendation starts with BYOL, LIT-594: an online network with a predictor regresses onto an EMA copy of itself under a different augmentation, with no negatives, and reaches 74.3% with a ResNet-50; its ablations supply the predictor and EMA findings below. SimSiam, LIT-593, removed the EMA from that design, showed that a shared encoder, a predictor and a stop-gradient still learn, and supplies the stop-gradient finding.
What is load-bearing, as far as anyone has shown
- Stop-gradient is not optional. Removing it, with everything else held fixed, sends the loss to its minimum of −1 immediately — the optimiser finds the constant solution in a few steps.
- The predictor is not optional either. Removing it from BYOL yields an unsupervised Mean Teacher, which collapses.
- The EMA target is optional, but only just. BYOL can drop it if the predictor is kept near-optimal — solved in closed form per batch (52.5%) or merely given a higher learning rate (66.5%). Raise the projector’s rate as well and it falls to ≈25%. What matters is the predictor being ahead of what it predicts.
So the shared content of the two papers is: the branch being predicted must not be chasing the branch predicting it. Every mechanism in this family is a way of arranging that.
Why bother, when negatives work
Negatives cost a batch or a queue (SOTA-363) and they impose a distribution you must design — the negative sampling is an assumption that two different images are dissimilar, which is wrong for near-duplicates and for fine-grained classes. Removing them removes that assumption and the memory that carried it.
Conditions
- Nothing here explains why it works, and the two papers’ explanations are incompatible. THEORY-087 is the dispute; treat the recipe as empirical and do not reason from the mechanism.
- The collapse failure is silent unless you look for it. Loss goes down. Run SOTA-366.
- Weight decay is part of the method. BYOL reports that removing it makes both BYOL and SimCLR diverge.
- The augmentation dependence does not go away (SOTA-361); these methods still rely on the view construction to define what is invariant.
- The alternative family exists and is competitive — SOTA-367 puts the constraint in the loss instead and lands within a couple of points.