multimodal-learning.
12 of 378 NOTE documents. Back to the full index.
| # | Title | Summary | Status |
|---|---|---|---|
| NOTE-027 | Simple Open-Vocabulary Object Detection with Vision Transformers | Transfers a contrastively pretrained image-text model to open-vocabulary detection with minimal architectural change. Its transferable finding is about fine-tuning: the text encoder needs a learning rate about 100× lower than the image encoder, freezing it entirely is also wrong, and the recipe that maximises zero-shot transfer is not the one that maximises in-distribution performance. | Read |
| NOTE-049 | Learning to Prompt for Vision-Language Models | Replaces hand-written CLIP prompts with continuous context vectors learned end-to-end while every pretrained parameter stays frozen. One or two shots beat prompt engineering; sixteen shots average about 15% gain, over 45% at the best. The paper also reports where it overfits and names the cause as noisy labels. | Read |
| NOTE-055 | CoCa: Contrastive Captioners are Image-Text Foundation Models | Trains one encoder-decoder with both a contrastive and a captioning loss by omitting cross-attention in the first half of the decoder — so those layers produce unimodal text embeddings for the contrastive loss, and the cascaded remainder cross-attends for captioning. Both objectives share one computational graph, at minimal extra cost. | Read |
| NOTE-069 | PaLI: A Jointly-Scaled Multilingual Language-Image Model | Scales language and vision components jointly across 100+ languages and shows a properly scaled model handles many languages while keeping English-only state of the art. Its most useful passage is an ablation honest enough to explain a regression by the language ratio of the data that caused it. | Read |
| NOTE-196 | Scaling Rectified Flow Transformers | A 61-way sweep over diffusion and rectified-flow formulations settles on a straight-line path with logit-normal timestep sampling — rectified flow with uniform timesteps does not win. Adds a resolution-dependent timestep shift, MMDiT’s per-modality weights with joint attention, and QK-RMSNorm for high-resolution stability. 8B, weights released. | Read |
| NOTE-206 | The Platonic Representation Hypothesis | Vision and language models measure distance between datapoints more alike as they get larger, across architectures and objectives. The conjecture is that they converge on a model of what generated the data. The proof holds for bijective observations only, and the caption-density experiment tests that boundary rather than asserting past it. | Read |
| NOTE-232 | Mixture-of-Transformers | Read to test whether it satisfies SOTA-262’s promotion condition. It does, as that condition was written — second group, per-modality weights against a shared-weight backbone, FLOPs held identical. The component ablation is a bonus the condition did not ask for and the practice needed. | Read |
| NOTE-255 | Gemini 1.0: complete disclosure, and an abstract that picks the flattering row | Read from the #180 worklist for the model and filed for the appendices. The headline — first model to exceed human-expert MMLU — holds under one inference procedure out of three the report itself publishes, and under the other two GPT-4 is ahead. Nothing is concealed. The disambiguating numbers are in the same document, which is what makes this the corpus’s cleanest instance of DP-010. | Read |
| NOTE-333 v2 | Wan | An open 14B text-to-video recipe whose evidence is mostly systems measurement: 2D context parallelism cuts communication overhead from over 10% to under 1%, and caching and 8-bit kernels give 1.62× and 1.27×. Its three modelling ablations run on 1.3B text-to-image. Two are read from training loss, and on FID one of them favours the rejected option. The applications in §5 are demonstrated almost entirely by figures. | Read |
| NOTE-344 | HunyuanVideo | A 13B open text-to-video model whose one quantitative design argument is a pair of compute-optimal fits on a proxy family: the image fit puts more of each added FLOP into parameters (0.5634 against 0.4325), and the video fit, initialized from the image envelope, reverses it (0.3618 against 0.6289). No architectural or data choice is ablated, and the proxy differs from the shipped model in objective, text encoder and conditioning. | Read |
| NOTE-349 | Make-A-Video | Make-A-Video extends an unCLIP text-to-image model to video with identity-initialized pseudo-3D conv and attention layers trained on video with no captions. Text reaches the video decoder only through one CLIP image embedding, so text-only-in-video associations cannot be learned, and the discussion concedes this. System-level results are strong (zero-shot UCF-101 FVD 367.23, MSR-VTT CLIPSIM 0.3049), and there is no ablation of any design choice, image initialization included. | Read |
| NOTE-371 | FLIP | Removing 50% of image patches in CLIP training halves the image encoder’s cost and lets the contrastive batch double at the same memory; ViT-L/16 on LAION-400M then matches its unmasked reproduction more than 3× faster and beats it by about a point at equal epochs. At equal batch the masking gains nothing, and at ViT-B/16 the full-schedule model is 0.2 behind — the improvement is the batch and the extra samples that masking pays for. | Read |