Chen & He (2020) — ARXIV-2011.10566
Key takeaways
- The paper is a subtraction, and the subtraction is the result. SimSiam is “BYOL without the momentum encoder”: one shared-weight encoder, a prediction MLP on one branch, a stop-gradient on the other, maximise similarity. It uses none of (i) negative pairs, (ii) large batches, (iii) momentum encoders — the three mechanisms the field had offered as the reason collapse does not happen — and it still learns.
- Stop-gradient is the piece that cannot be removed, and the failure is instant. With architecture and every hyperparameter held fixed, removing it means “the optimizer quickly finds a degenerated solution and reaches the minimum possible loss of −1”.
- The collapse detector is three lines and is the most portable thing here. Take the ℓ2-normalised output and measure its per-channel standard deviation across samples. Collapsed to a constant → 0. Scattered on the unit hypersphere as a zero-mean isotropic Gaussian would be → 1/√d. The paper’s own runs sit at 1/√d with stop-gradient and at 0 without it. Filed as SOTA-366.
- It names the confound that hid this. A momentum encoder “is always accompanied with stop-gradient (as it is not updated by its parameters’ gradients)”, so any experiment attributing the effect to momentum was attributing it to a bundle. Removing the bundle and keeping only the stop-gradient is the experiment nobody had run.
- The hypothesis is an alternating optimisation, and it is offered as a
hypothesis. SimSiam is proposed as EM-like over two variable sets,
“analogous to k-means clustering” —
θplays the cluster centres, the per-image variable plays the assignments — with the alternation interval set to one SGD step. Proof-of-concept experiments follow; the paper does not claim to have proved it. - The direct empirical disagreement with BYOL. BYOL reports that removing its momentum encoder gives 0.3% accuracy. SimSiam is that removal and reports 67.7% at 100 epochs (68.1% with BN on the projector output, its default). Both papers are from careful groups and the discrepancy is not resolved in either; THEORY-087 is where the record keeps it.
Standing in the anthology
Unit C of #304. Declares extends: LIT-594 because it is
structurally BYOL minus one component; the disagreement about what that
component does is prose here and a Proposed theory document there, rather
than a corrects edge, because neither paper’s measurements are in
question — only their accounts.
What it takes from BYOL and what it removes. From BYOL (LIT-594) it keeps the whole design: two augmented views, an encoder and projector, a predictor on one branch, a negative-free similarity loss, and a target branch that receives no gradient. The one change is that the target is the online network itself rather than an exponential moving average of it, so the stop-gradient is left standing alone. BYOL’s own ablations already showed its target network could go if the predictor was kept near-optimal (66.5% with a raised predictor learning rate). What SimSiam adds is the ablation that separates the two: with the moving average gone, removing the stop-gradient as well is what collapses it.