Source
Oquab et al. (2023), LIT-599 — ARXIV-2304.07193, §3.
The claim
Self-supervised pretraining is usually scaled by pointing it at more uncurated data. Do the opposite: use the curated datasets you already have as queries, and retrieve their nearest neighbours out of the uncurated pool.
The pipeline, as run:
- embed the uncurated pool and several curated datasets;
- for each curated query image, retrieve its nearest neighbours —
typically 4, because larger
N“leads to more collisions”; - deduplicate;
- train on the result.
It is an index job, not a training job: Faiss, 20 nodes of 8×V100-32GB, under two days to produce a 142M-image set.
The comparison that makes it a practice rather than a preference
Both the pipeline and the comparison are LIT-599’s. Oquab et al. built LVD-142M this way to train DINOv2, and proposed curation by retrieval as the alternative to scaling the uncurated pool, on the thesis that existing self-supervised objectives already give general-purpose features “if trained on enough curated data from diverse sources”.
Against 142M images randomly sampled from the same source, the curated set wins. That controls for volume, for the source distribution and for the crawl — the three confounds that make most data claims unfalsifiable — and leaves selection as the only difference.
Against ImageNet-22k, the curated set matches on ImageNet-1k while significantly outperforming it on the other benchmarks. That is the shape worth noticing: curation aimed at generality does not buy in-distribution accuracy, it buys the tail. A pipeline evaluated only on its seed distribution would have reported no gain.
Why retrieval rather than filtering
A filter needs a quality predicate — a classifier, a heuristic, a threshold — and every such predicate is a hypothesis about what good data looks like. Retrieval-from-seeds needs no predicate: it inherits the judgement already embedded in the curated datasets somebody built. The paper names its inspiration as text-curation pipelines where a model trained on Wikipedia scores crawled documents, which is the same trick.
Conditions
- It inherits the seeds’ biases, all of them. Whatever the curated datasets over- or under-represent, the retrieved set will amplify, because proximity to a seed is the only criterion. This is the cost of having no predicate, and it is not a small one.
Nis a real hyperparameter with a stated failure. More neighbours gives more collisions — images retrieved for several queries — which degrades diversity while appearing to add volume.- Retrieval quality is a function of the embedding you retrieve with, which was itself trained on something. The pipeline is not assumption-free, it moves the assumption into the index.
- One group, one pipeline, one domain. The controlled comparison is clean and singular; nothing here has been replicated independently.
- The distillation result is separable and is not this practice. DINOv2 trains a 1B-parameter ViT-g and distills it into the smaller models, which beat the same architectures trained from scratch on all 12 benchmarks. Useful, and a different claim.