Robinson et al. (2020) — ARXIV-2010.04592
Key takeaways
- Samples hard negatives without labels by reweighting the contrastive objective rather than changing the sampling procedure — “only two extra lines of code” and no computational overhead, via importance sampling plus a positive-unlabeled correction for false negatives
- The hardness knob is forced by a real tension, stated precisely: the hardest negatives are the points closest to the anchor, which are exactly the ones most likely to share its label — so the residual error of an approximate false-negative correction grows with hardness. Hardness trades learning signal against contamination
- The family’s extremes are legible: hardness fully down recovers the earlier debiased-contrastive method; pushed up it tightly clusters each class and maximally separates classes
- Margins are small — MoCo-v2 on CIFAR-10 linear readout 88.08% → 88.47%
Standing in the anthology
Read — NOTE-066. The document is Rejected and the reading does not argue
with that; the CIFAR-10 margin here is 0.39%.
Two things transfer, neither about contrastive learning. Reweighting and resampling are interchangeable when the objective is an expectation, and reweighting is free — which applies to data mixing, curriculum ordering, and anywhere the record recommends changing what a model sees. And where “harder examples” is proposed, ask what makes them hard and whether that also makes them wrong; the record has several practices recommending more of something difficult and no statement of why such advice needs a ceiling.
Of the takeaways replaced, “mining strategies comparison” is wrong — this paper proposes one method and derives a family around it; it compares nothing.