Source
Siméoni et al. (2025), LIT-663.
The problem this exists for, which is the part to read first
On a long self-supervised schedule, the two things you measure come apart:
- ImageNet linear-probe accuracy improves monotonically, all the way.
- Pascal VOC segmentation mIoU declines after about 200k iterations, and on a 7B model falls below its early levels.
If you are watching the global number, nothing is wrong. What is degrading is locality: the cosine-similarity map from a reference patch to all others goes from “smooth and well-localized” at 200k to “substantially” degraded by 600k, and CLS-to-patch similarity climbs throughout — patches drifting toward the global summary and away from their own content.
This is not the register defect. With registers, patch norms stay stable
through training (SOTA-400 is still right and still worth doing). This
is a different failure on a different axis — duration rather than size — and it
needs its own remedy.
Both the decline and the fix come from DINOv3 (LIT-663). It set out to train a 7B model for an extended duration, measured ImageNet linear-probe accuracy and VOC segmentation diverging in that run and in ViT-g, and introduced Gram anchoring to recover the dense features.
What to do
Keep a Gram teacher: an early checkpoint of the EMA teacher, from before
the dense metrics turned over. Add a loss pulling the student’s patch
similarity structure towards the teacher’s, on L2-normalized patch features:
L_Gram = ‖ X_S · X_Sᵀ − X_G · X_Gᵀ ‖²_F
- Compute it on global crops only.
- The point of using the Gram matrix rather than the features is that “the local features are free to move, provided the structure of similarities remains the same” — it constrains geometry, not position.
- Refresh the Gram teacher to the current EMA teacher periodically; they use every 10k iterations.
It can be started late. They begin after 1M iterations for efficiency and report that the late application “still manages to ‘repair’ very degraded local features” — dense gains land within the first 10k iterations of turning it on.
Conditions
You only need it if you are training long. The decline sets in after ~200k iterations in their runs. A short schedule never reaches the regime, and the cost is a loss term plus a stored teacher for no benefit.
The trigger is a dense metric, not a global one. The practice is unusual in that its own necessity is invisible to the evaluation most people run. If you are not tracking a patch-level benchmark through training, you cannot tell whether this applies to you — which is the more portable lesson and holds whatever you do about the loss.
One group, one model line. The unreplicated reading is the honest one:
nobody outside the authors has reported the decline, so the record cannot say
whether other self-supervised recipes have it. The authors say it appeared “to
a lesser extent” in DINOv2 and is “unresolved” elsewhere.
The teacher choice is unsearched. Which early iteration, and how often to refresh, are stated as choices. There is no ablation here saying they matter or do not.
It repairs without explaining. Nothing here says why a global objective costs locality as training continues. The record holds no account of it, and this practice does not declare one.
Known implementations
- DINOv3 — the source’s own, so adoption rather than independent replication
(
DP-005).