Assran et al. (2023) — ARXIV-2301.08243
Key takeaways
The prediction target is a representation, and that is the whole idea. Generative self-supervision predicts pixels and spends capacity on detail that carries no semantics; invariance-based self-supervision predicts sameness across hand-crafted augmentations and inherits whatever biases the augmentation set encodes. Predicting the embedding of a masked region from the embedding of a visible one is neither — it is non-generative and needs no augmentations.
The masking strategy is the load-bearing hyperparameter. Two conditions are called out as crucial: target blocks large enough to be semantic, and a context block informative enough — spatially distributed — to make the prediction possible without being trivial. This is the part that later JEPA work carries forward unchanged in spirit.
Efficiency is a reported result, not a footnote. ViT-Huge/14 on ImageNet, 16 A100s, under 72 hours, with strong downstream performance from linear classification through object counting and depth prediction. The breadth of that list is the claim that the representations are semantic rather than task-tuned.
The comparison against MAE, which predicts pixels. MAE (LIT-601) is this paper’s main baseline among methods that use no view augmentations. It is the pixel-reconstruction design I-JEPA’s encoder/predictor most resembles. On ImageNet-1K linear probing, I-JEPA ViT-H/14 reaches 79.3% after 300 epochs against MAE ViT-H/14’s 77.2% after 1600. With 1% of the labels the figures are 73.3% against 71.5%. The paper puts I-JEPA’s ViT-H/14 pretraining at over 10× more efficient than MAE’s, despite costing about 7% more per iteration to compute targets in representation space. The win does not cover full fine-tuning. There, MAE’s ViT-H at 448 resolution keeps the lead, 87.8% against I-JEPA’s 87.1%. So the comparison favours predicting representations for off-the-shelf features and on compute, and not on the best fine-tuned score.
Standing in the anthology
The architectural idea, filed for its successor’s sake. The record holds no vision self-supervision material — the vision cluster it inherited is generative (diffusion, radiance fields, detection) and none of it is about learning representations by prediction in latent space. I-JEPA is here because LIT-215 is here, and a line whose origin is missing is a line the reader cannot check.
Not a practice, and not close to one. Nothing in this record trains a vision encoder. The reason to hold it is that the JEPA argument — predict in representation space, not in observation space — is a claim about objectives that is stated generally and has been tried outside vision. If it ever arrives in language-model training, this is where the line starts.
Audited by #121; unsourced and unread. No reading note. See LIT-215: this is one of a three-paper joint-embedding-predictive cluster in the same state.