anthology-of-the-sota

LIT-072: Simple Open-Vocabulary Object Detection with Vision Transformers

2 min read

Properties6
StatusActive
Version2
Tagsvision-and-graphics, multimodal-learning, adaptation-and-tuning, analysis-and-evaluation
Cited as “introduced by” bySOTA-196 — Report zero-shot and in-distribution performance separately; one hyperparameter can move them in opposite directions, SOTA-201 — Give each pretrained component its own learning rate when fine-tuning a composite model
Cited as “paper” byNOTE-027 — Simple Open-Vocabulary Object Detection with Vision Transformers
Cited as “source” bySOTA-196 — Report zero-shot and in-distribution performance separately; one hyperparameter can move them in opposite directions, SOTA-201 — Give each pretrained component its own learning rate when fine-tuning a composite model

Graph View

Minderer et al. (2022) — ARXIV-2205.06230

Key takeaways

  • Open-vocabulary detection from a contrastively pretrained image-text ViT with minimal architectural addition, measured on LVIS v1.0 val for its long tail of rare categories
  • The transferable finding is about fine-tuning two pretrained encoders. A shared learning rate is “clearly sub-optimal”: the text encoder needs one about 100× lower, which the authors suggest prevents catastrophic forgetting of what contrastive pretraining gave it. Freezing it completely also fails — the optimum is interior, not at a corner
  • The two metrics have opposite optima. “The optimal recipe for zero-shot transfer does not necessarily maximize in-distribution performance” — the same-learning-rate setting that collapses AP_rare^LVIS improves AP^OI. One hyperparameter, two measures, opposite gradients

Standing in the anthology

Read — NOTE-027. Nothing is sourced here; open-vocabulary detection is not a line the anthology tracks.

The reach is in the fine-tuning result. The record’s adaptation practices treat a fine-tune as one operation with one learning rate; this measures that the real decision is component-wise, with a factor of 100 between components and a non-monotone response. Paired with LIT-079’s prior-preservation loss it makes two different instruments for protecting a pretrained component, neither of which the record states.

Of the takeaways replaced, “zero-shot capabilities” was the third bullet in this batch to use that phrase in place of saying what the paper does — and here the paper’s finding is that zero-shot capability is in tension with in-distribution accuracy, which is the more interesting claim.

Backlinks