Topics — The twenty-eight kinds of claim this anthology files against — the primary axis of both the practice registry and the reading list, and the same twenty-eight for each (ADR-026). The filing rule is the point: take a domain topic when the claim is ABOUT the domain; a claim merely discovered in one still takes its kind. The axis organizes the record, it does not bound it: content the list cannot place is evidence the list is short (ADR-059).
Multimodal learning — what changes when one model has to take in more than one kind of signal — joint architectures and fusion, contrastive image-text training, cross-modal transfer, and what a second modality buys the first.
10 of 449 SOTA documents. Back to the full index.
| # | Title | Summary | Status |
|---|---|---|---|
| SOTA-262 v2 | Give each modality its own weights and let the streams attend jointly | Esser et al. (2024), LIT-449 — MMDiT gives the text and image streams separate projections and MLPs, then runs attention over the concatenated sequence so information flows both ways. It beats UViT and DiT at matched budget. Liang et al. (2024), LIT-483 — the same shape reached independently for autoregressive multi-modal LLMs, matching a dense baseline at 55.8% of the FLOPs, and ablated component by component: the feed-forward carries most of the benefit, the attention projections less, the layer norms none. | Active |
| SOTA-271 | Train on a second modality even when the target is single-modality | Huh et al. (2024), LIT-458 — if representations converge on a modality-agnostic model of what generated the data, then data from any modality helps find it. The vision direction is already common practice; the language direction is not, and the paper cites a case where adding image data improved text performance. | Proposed |
| SOTA-357 | Build a zero-shot classifier from prompt templates and ensemble them in embedding space | Active | |
| SOTA-359 v2 | Supervise vision from the caption, and match image to caption rather than predicting its words | Active | |
| SOTA-376 | Score each image-text pair independently with a sigmoid, so the loss needs no global normalization | Active | |
| SOTA-377 | Stop scaling the contrastive batch past about 32k, because the benefit saturates there | Active | |
| SOTA-389 v3 | Caption training video with a model that watches the video, not with captions of its frames | Proposed | |
| SOTA-407 v2 | Interpolate the zero-shot and fine-tuned weights at about half way rather than shipping the fine-tuned model | Wortsman et al. (2021), LIT-674. Fine-tuning a zero-shot model spends the robustness that made it worth starting from. Ship (1 − α)·θ_zero-shot + α·θ_fine-tuned at α = 0.5 instead: +3.5 to +23.2 pp under six distribution shifts against the fine-tuned model, with reference accuracy falling by at most 0.3 pp and usually rising, at no cost in training or inference. Requires that the fine-tuned weights were obtained from the zero-shot weights. | Active |
| SOTA-429 | Drop half the image patches when training a CLIP-style model, spend the saving on more pairs and a larger batch, and unmask only for a short final tune | Li et al. (2022), LIT-706 — FLIP. Remove 50% of image patches and run the ViT on the rest, with CLIP’s loss and nothing else. The saving buys a 2× larger batch at the same memory and 2× the samples per hour; ViT-L/16 on LAION-400M reaches its unmasked reproduction’s accuracy more than 3× faster. The accuracy comes from what the saving buys: at equal batch, masking is parity. | Proposed |
| SOTA-436 | Route a very large or non-grid input through cross-attention into a small learned latent array, so that depth no longer scales with input size | Jaegle et al. (2021), LIT-747 — the Perceiver. A learned array of 512 latents cross-attends to all 50,176 ImageNet pixels, at a cost of O(MN) rather than O(M²), and a 48-block latent transformer then runs at a cost independent of input size. It matched ResNet-50 and ViT-B/16 at 78.0% top-1 without convolutions, did not beat them, and ran unchanged on audio, video and point clouds. The case for doing it is cost and modality-independence, not accuracy, and the cost advantage is argued from complexity, not measured against a baseline. | Proposed |