Source
Minderer et al. (2022), LIT-072 — OWL-ViT, fine-tuning a contrastively pretrained image-text model for open-vocabulary detection.
The claim
A model assembled from separately pretrained parts should not be fine-tuned at one learning rate. Each part carries different knowledge, is being asked for a different amount of change, and has a different amount to lose.
The measurement: using the same rate for the image and text encoders is “clearly sub-optimal”, and the text encoder needs a rate roughly 100× lower. The authors’ explanation is that this
may help to prevent catastrophic forgetting of the wide knowledge the model acquired during the contrastive pre-training stage
The recommendation comes from LIT-072’s Table 3, a sweep of the text encoder’s learning rate while fine-tuning a contrastively pretrained image-text ViT into the OWL-ViT detector, with zero-shot quality measured on LVIS v1.0 val’s rare categories. The paper measured the asymmetry and both failing corners; the forgetting explanation is the authors’ suggestion and the sweep does not isolate it.
The optimum is interior
Freezing the text encoder entirely — learning rate 0 — also does not work well. So this is not “fine-tune the new part and freeze the old one”, which is the usual binary. The response is non-monotone with a maximum somewhere in the middle, and both corners are wrong.
That is what makes it a decision rather than a default: a single shared rate is one corner, freezing is the other, and the answer is neither.
The metric conflict comes with it
The same sweep moves AP^OI — in-distribution performance — in the opposite
direction: the shared-rate setting that collapses zero-shot transfer improves
it. So the component learning rate is a dial between preservation and fitting,
and picking it requires knowing which you want. See SOTA-196, which is
that finding on its own.
Relation to the other ways of protecting a pretrained part
The record now carries three instruments for the same goal, at different price points:
- This — turn the learning rate down on what you do not want to move. Costs one hyperparameter per component.
- SOTA-199 — regularize against the pre-fine-tuning model’s own samples. Costs a loss term and a sampling pass.
- SOTA-184 — do not touch the weights at all; train a low-rank update.
None has been compared against the others.
Conditions
One measurement, vision-language, 2022. The 100× factor is specific to this model pair and is not derived — nothing here says how to choose the ratio, only that it is large and that both corners fail.
The explanation (catastrophic forgetting) is offered by the authors and not isolated.
Known implementations
- OWL-ViT, Table 3