Oquab et al. (2023) — ARXIV-2304.07193
Key takeaways
- The claim is about data, and it is stated as such. “Existing pretraining methods, especially self-supervised methods, can produce such features if trained on enough curated data from diverse sources.” The objective is assembled from DINO and iBOT rather than invented; what is new is the pipeline that feeds it.
- Curation by retrieval, not by filtering. Take a large uncurated pool and several curated datasets as queries; retrieve each query’s nearest neighbours in embedding space — typically 4, because “much larger than 4 … leads to more collisions”; deduplicate. The result is LVD-142M. Faiss for the index; 20 nodes of 8×V100-32GB, under two days. The inspiration is named as text-curation pipelines, where a model trained on Wikipedia scores documents from a crawl.
- The controlled comparison is the reason to believe it. Against 142M images randomly sampled from the same source, curated wins — so this is not a data-volume result. Against ImageNet-22k, a ViT-g on LVD-142M matches on ImageNet-1k “while significantly outperforming it on the other benchmarks”, which is the general-purpose claim rather than a leaderboard one.
- Train the big one, distill the small ones. A 1B-parameter ViT-g is trained, then distilled into the smaller models. The distilled model beats the same architecture trained from scratch on all 12 benchmarks.
- Two regularisers worth naming. A KoLeo term, from the Kozachenko-Leonenko differential entropy estimator, encouraging features to spread uniformly within a batch; and Sinkhorn-Knopp centering borrowed from SwAV (LIT-598) in place of the teacher’s softmax centering. Both appear as separate ablation rows. This is the same anti-collapse question THEORY-087 tracks, arriving as a stack of regularisers rather than as a mechanism anyone claims to understand.
- The deliverable is frozen features. The whole argument is features that “work across image distributions and tasks without finetuning”, competitive with the best openly available weakly-supervised models. That is what makes it usable as a measurement instrument rather than only as a backbone.
What its features do wrong, measured afterwards
This note recommends DINOv2 as a frozen feature extractor, which is what the
record reaches for it for. Two things about those features were characterized
later, by the same group, in LIT-662.
2.37% of patch tokens are high-norm outliers. They sit on background
patches redundant with their neighbours, they have discarded their own local
content — position probe 22.8 top-1 against 41.7 for normal tokens, pixel
reconstruction L2 25.23 against 18.38 — and they carry global information
instead. The account is THEORY-101: the model is using redundant patches
as scratch space, and DINO ViT-B/16 does not do it because it is too small to.
And it broke a task family the previous generation had advanced. In their words, “we observed that DINOv2 is surprisingly incompatible with LOST”. Unsupervised object discovery was built on DINO and had “significantly surpassed the previous state of the art” with it; DINOv2 is the stronger model on dense prediction and scores 35.3 corloc on VOC 2007 against DINO’s 61.9.
Registers recover most of it — 55.4, SOTA-400 — and not all of it,
so something else about this model also costs object discovery and nobody has
named it.
Worth carrying because of what it does to the phrase this note’s thesis rests on. Curation-plus-scale produces general-purpose features on the evaluations run; a task family nobody in the comparison was running regressed by half.
Where it sits in its own line
The record now holds all four: DINO (LIT-664), this, the registers paper
(LIT-662) and DINOv3 (LIT-663). Read together they say something
this note could not say alone.
Two dense-feature defects in this recipe, both characterized later, both by the same group, on different axes.
| defect | axis | measured | remedy |
|---|---|---|---|
| high-norm token artifacts | model size — ViT-L and above | 2.37% of tokens; VOC corloc 35.3 against DINO’s 61.9 | register tokens (SOTA-400) |
| loss of patch locality | training duration — after ~200k iterations | VOC mIoU declines while ImageNet top-1 rises | Gram anchoring (SOTA-401) |
DINOv3 reports the second as present “to a lesser extent, during the training of DINOv2”, and is explicit that it “differ[s] from the high-norm patch outliers” — with registers, patch norms stay stable and locality still rots.
So this note’s thesis — curation plus scale gives general-purpose features — holds on the evaluations run, and both things that went wrong were invisible to them. One needed a larger model to appear and the other a longer schedule.
The three comparisons, from this model’s side. Against DINO (LIT-664), whose objective is half of this one, DINOv2’s own Table 4 re-evaluates DINO on frozen ImageNet-1k features: DINO ViT-S/8 gets 78.6 k-NN and 79.2 linear, against 83.5 and 86.5 for DINOv2 ViT-g/14 and 79.0 and 81.1 for the distilled ViT-S/14. On object discovery the order reverses, 35.3 LOST corloc on VOC 2007 against DINO’s 61.9. The registers paper (LIT-662) retrains this recipe with and without registers: frozen linear probes improve slightly (ImageNet 84.3 → 84.8, ADE20k 46.6 → 47.9 mIoU, NYUd RMSE 0.378 → 0.366) and LOST corloc rises from 35.3 to 55.4. DINOv3 (LIT-663) uses DINOv2 ViT-g/14 with registers as its main self-supervised baseline, and its 7B model leads on frozen dense probes (ADE20k 55.9 against 49.5 mIoU, VOC 86.6 against 83.1, NYUv2 depth RMSE 0.309 against 0.372) and on TokenCut object discovery (66.1 corloc against 55.6).
Standing in the anthology
Unit D of #304, filed with LIT-598 as the pair of instruments the
record was already reaching for: SOTA-338’s promote_when requires a
Fréchet distance “in a non-ImageNet feature space (CLIP, SwAV or DINOv2)”,
and SOTA-307 reports DINOv2 FID, precision, density and coverage. The
record could not promote its own practice without a paper it did not hold.
Sources SOTA-369, and joins THEORY-087 as a source for the way it stacks anti-collapse regularisers rather than choosing among them.