Topics — The twenty-eight kinds of claim this anthology files against — the primary axis of both the practice registry and the reading list, and the same twenty-eight for each (ADR-026). The filing rule is the point: take a domain topic when the claim is ABOUT the domain; a claim merely discovered in one still takes its kind. The axis organizes the record, it does not bound it: content the list cannot place is evidence the list is short (ADR-059).
Multimodal learning — what changes when one model has to take in more than one kind of signal — joint architectures and fusion, contrastive image-text training, cross-modal transfer, and what a second modality buys the first.
23 of 737 LIT documents. Back to the full index.
| # | Title | Summary | Status |
|---|---|---|---|
| LIT-049 v3 | Learning to Prompt for Vision-Language Models | Zhou et al. (2021), ARXIV-2109.01134. Learnable prompt optimization. | Active |
| LIT-071 v3 | CoCa: Contrastive Captioners are Image-Text Foundation Models | Yu et al. (2022), ARXIV-2205.01917. Unified vision-language architecture. | Active |
| LIT-072 v2 | Simple Open-Vocabulary Object Detection with Vision Transformers | Minderer et al. (2022), ARXIV-2205.06230. Open vocabulary detection. | Active |
| LIT-080 v2 | PaLI: A Jointly-Scaled Multilingual Language-Image Model | Chen et al. (2022), ARXIV-2209.06794. Joint vision-language scaling. | Active |
| LIT-135 | Qwen3.8-27B | Qwen Team (2026), model card and release blog; no technical report. A 27B dense vision-language model with 48 Gated DeltaNet layers and 16 gated full-attention layers, an MTP draft head and a 262k native context; architecturally identical to Qwen3.6-27B four months earlier, with the generation’s gains coming from post-training on agentic and coding work. | Active |
| LIT-449 v2 | Scaling Rectified Flow Transformers for High-Resolution Image Synthesis | Esser et al. (2024), ARXIV-2403.03206. A 61-way comparison of diffusion and rectified-flow formulations settles on a straight-line path with timesteps drawn from a logit-normal, then scales it to 8B for text-to-image. Adds a resolution-dependent shift of the timestep schedule, and MMDiT, which gives each modality its own weights while letting the two token streams attend jointly. | Active |
| LIT-458 v2 | The Platonic Representation Hypothesis | Huh et al. (2024), ARXIV-2405.07987. Representations are converging: across architectures, objectives and modalities, and more so as models get larger. The conjecture is that they are converging on a statistical model of whatever generated the data. The mathematical argument holds only for bijective observations, which the authors say plainly and test at the boundary. | Active |
| LIT-483 | Mixture-of-Transformers: A Sparse and Scalable Architecture for Multi-Modal Foundation Models | Liang et al. (2024), ARXIV-2411.04996. Untie every non-embedding parameter by modality — feed-forward, attention projections, layer norms — and keep global self-attention over the whole sequence. Matches a dense baseline at 55.8% of the FLOPs in the Chameleon setting and 37.2% with speech added. The component ablation is the useful part: the feed-forward carries most of it, attention adds less, layer norms add nothing. | Active |
| LIT-510 | Gemini: A Family of Highly Capable Multimodal Models | Gemini Team, Google (2023), ARXIV-2312.11805 — held for its evaluation methodology rather than its models. Its appendices publish three things the record wanted: an MMLU sweep in which the ordering against GPT-4 flips depending on the inference procedure; a measurement of how cheaply a benchmark can be inflated (100 fine-tuning steps on HellaSwag-adjacent web extracts take Ultra to 96.0% at 1-shot); and a FLEURS ablation retraining without the benchmark’s own training set. Read as NOTE-255. | Active |
| LIT-588 v2 | Learning Transferable Visual Models From Natural Language Supervision | Radford et al. (2021), ARXIV-2103.00020. Train an image encoder and a text encoder to match 400M (image, text) pairs contrastively, then build a classifier for any label set by embedding its class names. The most durable result is not the zero-shot accuracy — it is that supervised adaptation which raises ImageNet by 9.2% buys no average robustness. | Active |
| LIT-605 v2 | Sigmoid Loss for Language Image Pre-Training | Zhai et al. (2023), ARXIV-2303.15343. Replace CLIP’s softmax with a pairwise sigmoid, so the loss never needs a global view of the similarity matrix — no all-gathers, only a per-device block in memory, and better results below 16k batch. Then trains at batch size one million and finds the benefit saturated at 32k. | Active |
| LIT-619 v2 | Wan: Open and Advanced Large-Scale Video Generative Models | Wan Team, Alibaba (2025), ARXIV-2503.20314. The open Wan2.1 video models, 1.3B and 14B, and the full recipe behind them. The recipe covers data curation, a 127M causal 3D VAE compressing 4×8×8, a cross-attention DiT trained by rectified flow on an image-then-video resolution curriculum, the 2D context parallelism it needs, and a mask-conditioned image-to-video variant. The abstract claims video scaling laws in data and model size, but the report plots no scaling curve. | Active |
| LIT-620 v2 | HunyuanVideo: A Systematic Framework For Large Video Generative Models | Kong et al., Tencent Hunyuan (2024), ARXIV-2412.03603. A 13B open video model: FLUX-style dual-then-single-stream transformer, a multimodal LLM as text encoder, a from-scratch causal 3D VAE. It fits the line’s only explicit video compute-optimal scaling law, on a proxy model that differs from the shipped one in objective, text encoder and conditioning. It reports no controlled ablations. | Active |
| LIT-632 v2 | Make-A-Video: Text-to-Video Generation without Text-Video Data | Singer et al., Meta AI (2022), ARXIV-2209.14792. An unCLIP text-to-image model is extended into video with identity-initialized temporal layers trained on video whose captions are discarded. Text alignment comes entirely from images. That limits it: it “can not learn associations between text and phenomenon that can only be inferred in videos”. The paper has no ablations, and its title describes the training signal, not the data. | Active |
| LIT-648 | Improving Image Generation with Better Captions | Betker et al., OpenAI and Microsoft (2023), no arXiv id. Identical T5-conditioned latent diffusion models are trained on the same images with alt-text, short synthetic or descriptive synthetic captions. The descriptive ones give higher CLIP score, and more synthetic in the blend is better up to the 95% tested. The controlled evidence is CLIP score alone, at 256px, and on ground-truth prompts the gain is inside the checkpoint noise. The paper does not describe the DALL-E 3 model, and it says its DALL-E 3 comparisons are not a measurement of recaptioning. | Active |
| LIT-674 | Robust fine-tuning of zero-shot models | Wortsman, Ilharco, Kim, Li, Kornblith, Roelofs, Gontijo-Lopes, Hajishirzi, Farhadi, Namkoong and Schmidt (2021), ARXIV-2109.01903. Fine-tuning a zero-shot model buys target accuracy and costs robustness. Linearly interpolating the zero-shot and fine-tuned weights recovers both: at α = 0.5, +3.5 to +23.2 pp under six distribution shifts against the fine-tuned model while reference accuracy falls by at most 0.3 pp and usually rises. No extra cost at training or inference. And it names the precondition: averaging all the layers of two unrelated networks “typically fails, achieving no better accuracy than a randomly initialized neural network” — this works because the trajectory is shared. | Active |
| LIT-706 | Scaling Language-Image Pre-training via Masking | Li et al. (2022), ARXIV-2212.00794 — FLIP. Train CLIP with 50–75% of image patches removed and the image encoder run on the rest: the saved compute buys more pairs per hour and a 2–4× larger batch at the same memory, which outweighs the information lost per image. ViT-L/16 on LAION-400M reaches its CLIP reproduction’s accuracy more than 3× faster. The accuracy gain at a fixed schedule is 1–2 points at L and nothing at B/16, and at equal batch size masking 50% buys nothing — the batch is doing the work. | Active |
| LIT-732 | Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization | Selvaraju, Cogswell, Das, Vedantam, Parikh and Batra (2016; ICCV 2017, IJCV 2019), ARXIV-1610.02391. The referent behind both halves of SOTA-430’s verdict table — GradCAM, which passes the randomization test, and Guided GradCAM, which fails it, are both defined here, and the second is defined as the pointwise product of the first with guided backpropagation. Grad-CAM reads one layer: gradients of the class logit into the last conv feature maps, global-average-pooled into per-channel weights. Also carried: a 2016 human study in which guided backprop, the best-looking method, comes last of four — 44.44% against Deconvolution’s 53.33% — a number its authors did not gloss until v4 in December 2019, fourteen months after LIT-713 made the same point another way. | Active |
| LIT-745 | SigLIP 2: Multilingual Vision-Language Encoders with Improved Semantic Understanding, Localization, and Dense Features | Tschannen et al. (2025), ARXIV-2502.14786. Keep SigLIP’s architecture and sigmoid loss, and add a captioning-and-grounding decoder loss (LocCa) throughout, self-distillation and masked prediction for the last 20% of training, a 90/10 English/multilingual mix with de-biasing filters, and distillation through active data selection for the smallest models. It beats SigLIP at every size and resolution compared on zero-shot classification, retrieval, VLM transfer, dense probing and localization (RefCOCO val, L/256: 67.3 → 86.0). None of the ingredients is ablated, so the gains cannot be credited to any one of them. | Active |
| LIT-747 | Perceiver: General Perception with Iterative Attention | Jaegle et al. (2021), ARXIV-2103.03206 — the Perceiver. A learned latent array of N=512 queries cross-attends to the raw input (50,176 ImageNet pixels), so attention costs O(MN) rather than O(M²), and a deep latent transformer then runs at O(N²) per layer, independent of input size. With 2D Fourier-feature positions and no convolution it reaches 78.0% top-1 on ImageNet without pretraining, unchanged when the pixels are permuted, while ViT-B-16 and ResNet-50 given the same features fall to 61.7 and 39.4. The same architecture runs on audio, video and point clouds. | Active |
| LIT-770 v2 | DMAD: Distribution Matching as Adversarial Distillation for Fast Visual Generation | Yu et al., Texas A&M and ByteDance (2026), ARXIV-2610.02188. DMAD replaces DMD’s fake-score critic with a discriminator and trains the student on a linear loss on its logit. At the unrestricted discriminator optimum that gradient equals DMD’s idealized one (App. A, checked); nothing bounds it off the optimum, where training runs with one discriminator step per student step. One-step ImageNet-64 FID is 1.24 against DMD2’s 1.28, at 570K iterations, and 1.04 with an ImageNet-pretrained discriminator scored only by FID. Four-step SDXL reaches 14.47 FID at lower CLIP. Wan2.1 VBench margins are 0.1–0.3 points. No diversity metric and no seeds are reported. | Active |
| LIT-783 | Large Scale Diffusion Distillation via Score-Regularized Continuous-Time Consistency | Zheng et al., Tsinghua and NVIDIA (2025), ARXIV-2510.08431. sCM made to run on 14B text-to-image and text-to-video models, with a FlashAttention-2 JVP kernel under FSDP and context parallelism. Pure sCM loses fine detail and temporal coherence there. Adding a DMD loss at weight 0.01 (rCM) fixes it. Four-step Wan2.1 VBench is 84.43 at 1.3B against a re-implemented DMD2’s 84.56, and 84.92 at 14B; GenEval tracks the teacher at 1–4 steps. The diversity advantage over DMD2 is shown in samples only. One run per row. | Active |
| LIT-796 | InstaFlow: One Step is Enough for High-Quality Diffusion-Based Text-to-Image Generation | Liu, Zhang, Ma, Peng and Liu, UT Austin and Helixon (2023), ARXIV-2309.06380. Reflow Stable Diffusion once, then distil the reflowed model to one step with an LPIPS regression. Against distilling SD 1.4 directly, at the same 100K steps and 3.2M generated pairs, reflow first gives one-step FID 31.0 against 40.9 on COCO-2017-5k and 20.0 against 34.6 on COCO-2014-30k. The scaled model, InstaFlow-0.9B, scores 23.4 and 13.10. The teacher is a diffusion model, not a rectified flow, and the distiller is a regression, not distribution matching. One run per row. | Active |