Tian et al. (2020) — ARXIV-2005.10243
Key takeaways
- Proposes the InfoMin criterion for choosing contrastive views: reduce the mutual information between views while keeping task-relevant information intact. Too much shared information makes the task trivially solvable, too little destroys the signal — a sweet spot, not a direction
- Turns augmentation design from a bag of tricks into a constrained optimisation with a stated objective, and derives an augmentation set from it rather than collecting one
- InfoMin Aug with PIRL’s JigSaw reaches 73.0% ImageNet linear readout with ResNet-50, nearly 4% over SimCLR; transfers to PASCAL VOC detection and COCO instance segmentation consistently beat supervised ImageNet pretraining
- The unresolved tension: the criterion requires knowing which information is task-relevant, in a setting defined by not having labels
Standing in the anthology
Read — NOTE-057. The document is Rejected and the reading does not argue
with that.
The reason to have read it is the pairing with LIT-092, also Rejected,
also in this batch, and a direct methodological objection to this one: it argues
that reasoning about augmentations and the contrastive objective cannot by itself
predict downstream performance, because the function class and optimiser matter
and are absent from such accounts. InfoMin is exactly an
augmentation-and-objective account. Recording that they disagree is worth more
than either alone.
“Information bottleneck perspective” was also loose: InfoMin concerns mutual information between views, which is not the information-bottleneck objective.