vision-and-graphics.
52 of 378 NOTE documents. Back to the full index.
| # | Title | Summary | Status |
|---|---|---|---|
| NOTE-018 | Segment Anything | The data engine is the transferable part. Three stages — assisted-manual, semi-automatic, fully automatic — each stage’s labels training the model that produces the next stage’s, ending in 1.1B masks. The model and the dataset are built together. | Read |
| NOTE-027 | Simple Open-Vocabulary Object Detection with Vision Transformers | Transfers a contrastively pretrained image-text model to open-vocabulary detection with minimal architectural change. Its transferable finding is about fine-tuning: the text encoder needs a learning rate about 100× lower than the image encoder, freezing it entirely is also wrong, and the recipe that maximises zero-shot transfer is not the one that maximises in-distribution performance. | Read |
| NOTE-032 | Gaussian Shell Maps for Efficient 3D Human Generation | Puts 3D Gaussians on inflated and deflated shells around a template human body, with a CNN generating the texture stack whose features become the Gaussians’ attributes. Rendering fast enough removes the need for the 2D upsamplers that made prior 3D GANs multi-view inconsistent — a quality problem solved by a speed fix. | Read |
| NOTE-035 | Neural Scene Graphs for Dynamic Scenes | The first neural rendering method to decompose a dynamic scene into a scene graph — objects with learned transformations and radiance, described by a single implicit function plus a per-object latent — so novel arrangements can be rendered, not just novel views. Learned from a video alone. | Read |
| NOTE-037 | K-Planes: Explicit Radiance Fields in Space, Time, and Appearance | Represents a d-dimensional scene with “d choose 2” planes, so static (d=3) and dynamic (d=4) scenes are the same model at different d. The factorisation makes dimension-specific priors easy to attach and separates static from dynamic components naturally; a linear decoder with a learned colour basis matches a nonlinear MLP. | Read |
| NOTE-042 | Instant Neural Graphics Primitives with a Multiresolution Hash Encoding | Replaces a large coordinate MLP with a small one fed by trainable feature vectors stored in a multiresolution hash table. Hash collisions are not resolved — the multiresolution structure lets the network disambiguate them, and gradient descent averages the colliding entries by importance. Seconds to train, milliseconds to render. | Read |
| NOTE-044 | OnePose: One-Shot Object Pose Estimation without CAD Models | Estimates object pose from a single scanned video, with no CAD model and — the part the record’s own summary got backwards — no instance- or category-specific training. Builds an SfM point cloud of the object once, then matches 2D query points to it with a generic graph attention network that was never trained on that object or its category. | Read |
| NOTE-045 | 3D Gaussian Splatting for Real-Time Radiance Field Rendering | Represents a scene as anisotropic 3D Gaussians initialised from structure-from-motion points, optimised with interleaved density control, and rendered by projecting to 2D and α-blending. Differentiable like a volumetric field, rasterizable like geometry — the first real-time (≥30 fps) radiance-field rendering at 1080p. Static scenes. | Read |
| NOTE-046 | NeuS2: Fast Learning of Neural Implicit Surfaces for Multi-view Reconstruction | Cuts NeuS surface reconstruction from about 8 hours to minutes for a static object, and to as little as 20 seconds per frame for a moving sequence. The point of the speedup is what it unlocks: at 8 hours per object, dynamic scenes with thousands of frames are simply not attemptable. | Read |
| NOTE-047 | MiDaS v3.1 - A Model Zoo for Robust Monocular Relative Depth Estimation | Swaps the image encoder in a fixed depth-estimation architecture across BEiT, Swin, SwinV2, Next-ViT, LeViT and recent convolutional backbones, and reports the resulting performance-runtime frontier. The best backbone improves depth quality by 28%; the paper’s other deliverable is the procedure for integrating a new one. | Read |
| NOTE-170 | VGGT: Visual Geometry Grounded Transformer | One feed-forward pass produces camera parameters, point maps, depth maps and 3D point tracks together, in under a second, beating methods that post-process with geometry optimisation. Removes the global alignment step DUSt3R still needed, and works as a pretrained backbone — which is when a subject stops being a field and becomes a component. | Read |
| NOTE-171 | DUSt3R: Geometric 3D Vision Made Easy | Regress pointmaps from uncalibrated, unposed images instead of solving for cameras and triangulating. The result worth carrying is not the accuracy but the direction of dependency: matches and camera parameters fall out of the 3D output, where the classical pipeline needs them before it can begin. | Read |
| NOTE-172 | SuperGlue: Learning Feature Matching with Graph Neural Networks | Learn the matcher instead of hand-designing it: a GNN predicts assignment costs, differentiable optimal transport solves them, and correspondence and rejection happen jointly. Filed for the line rather than for a practice — its surviving assumption, that a detector ran first, is what the next paper identifies as the binding constraint. | Read |
| NOTE-173 | LoFTR: Detector-Free Local Feature Matching with Transformers | Remove the keypoint detector and match densely, coarse-to-fine, with descriptors conditioned on both images by cross attention. The argument is a failure mode rather than an efficiency: detectors cannot emit repeatable points in low-texture regions, so no downstream matcher can recover what was never proposed. | Read |
| NOTE-185 | Self-Supervised Learning from Images with a Joint-Embedding Predictive Architecture | Predict representations, not pixels, and drop hand-crafted augmentations entirely. From one context block, predict what a learned target-encoder outputs for several masked target blocks in the same image. The masking strategy is the whole design: targets large enough to be semantic, context spatially distributed, and the mask applied to the target-encoder’s OUTPUT rather than its input. A ViT-H/14 on ImageNet in under 1200 GPU-hours. | Read |
| NOTE-186 | V-JEPA 2: Self-Supervised Video Models Enable Understanding, Prediction and Planning | I-JEPA’s construction at video scale — 1M+ hours, up to 1B parameters — plus a second act: post-train a latent action-conditioned world model on under 62 hours of unlabelled robot video and plan with it zero-shot on real arms. The scaling section attributes its gains one ingredient at a time, and the one worth stealing is unrelated to video: raise resolution only in the decay phase and pretraining costs up to 8x less. | Read |
| NOTE-187 | Video models are zero-shot learners and reasoners | The argument by analogy: LLMs became generalist from three primitives — large, generative, web-scale — and video models now have all three. The evidence is a catalogue of tasks Veo 3 was never trained for and can do anyway, from segmentation and edge detection through affordances and tool use to maze and symmetry solving. It is a position paper with demonstrations, and the demonstrations are of one proprietary model. | Read |
| NOTE-188 | NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis | Store the scene in the weights of an MLP that maps a 5D coordinate to density and view-dependent colour, and render it with classical volume rendering, which is differentiable — so posed images are the only supervision. Two additions do the work everybody remembers: positional encoding, without which the result is oversmoothed, and hierarchical sampling, without which it is unaffordable. | Read |
| NOTE-224 | Dropout Reduces Underfitting | Reading it: the persuasive number is the training loss, not the accuracy. Early dropout lowers ViT-T’s training loss from 3.443 to 3.394 while standard dropout raises it to 3.885 — so the same operator is doing two opposite things depending only on when it runs, and one of them is not regularization. | Read |
| NOTE-237 | gen2seg | Read as the counter-route to SAM’s data engine. The headline (narrow finetuning, broad generalization) is well evidenced but the argument lives in the controls, which are unusually good: the same backbone under a conventional head scores ~1.5 mIoU, and swapping only the decoder recovers part of the gap. | Read |
| NOTE-243 | GaussianToken and the cost of the continuous half | Read from the #180 worklist and filed without a practice. The idea — let the quantization units move and size themselves instead of sitting on a grid — is a good one, and the evidence for it cannot be separated from the five continuous parameters each token carries alongside its index. The comparison is at equal token count, and nobody computed bits. | Read |
| NOTE-256 | Image-GS: the rate accounting GaussianToken skipped, and an ablation that inverts its own headline | Read from the #180 worklist for its adjacency to the tokenizer trunk. Two things worth keeping: a comparison run the way this record keeps asking for — six baselines at matched model size, bitrate on the axis — and an ablation in which the two unglamorous optimizer choices outweigh the content-adaptive allocation the paper is named for. | Read |
| NOTE-295 | SIREN | Sine-activated MLPs whose derivatives are well behaved, trainable because weights U(±√(6/n)) keep pre-activations standard normal and outputs arcsine at every depth, with ω₀ = 30 on the first layer. They fit signals and derivatives that ReLU, tanh and positional encoding do not. Read §1–4.2 and supplement §1; the later experiments were skimmed. | Read |
| NOTE-296 | Fourier features | A coordinate MLP’s NTK has fast spectral falloff, so it fits high frequencies too slowly to matter. A sinusoidal input mapping makes the composed kernel stationary and sets its bandwidth, and the NTK linear model predicts the trained networks’ loss curves. Gaussian random frequencies are best on seven tasks, and only their scale matters. Read §1–7; the appendices were not read. | Read |
| NOTE-298 | Diffusion Forcing | Independent per-token noise levels during training give one causal model both autoregressive sampling and horizon-wide guidance, with sampling schedules chosen at inference. Stable long video rollouts are shown qualitatively. Planning and robot results are measured. Read §1–5; the appendix proofs and time-series results were not read. | Read |
| NOTE-300 | StyleGAN3 | Aliasing in the generator glues fine detail to pixel coordinates. Continuous-signal design, with 2×-upsampled filtered nonlinearities, Fourier input and no noise, gives translation- and rotation-equivariant generators at StyleGAN2’s FID. Read §1–3.2 and Figure 3; the internal representation analysis and the appendices were skimmed. | Read |
| NOTE-301 | StyleGAN2 | Diagnoses two StyleGAN artifacts and fixes each at its cause: weight demodulation instead of instance normalization, and a fixed skip/residual architecture instead of progressive growing. Adds lazy regularization and path-length regularization. Read §1–4.1 and Tables 1–2; §4.2 onward and the appendices were skimmed. | Read |
| NOTE-302 | CycleGAN | Unpaired translation with two adversarial losses and a bidirectional cycle-consistency loss. It works for appearance changes and fails for geometric ones. The ablation supports the cycle term against GAN-only, but not strictly the bidirectional form over one direction. Read §1–6; the appendix was skimmed. | Read |
| NOTE-303 | Projected GAN | A GAN discriminator on frozen, randomly mixed, multi-scale pretrained features reaches prior-best FIDs up to 40× faster and sets new FIDs on 22 datasets. Everything is measured in FID or ImageNet-derived metrics, and the discriminator is ImageNet-pretrained, which LIT-563 later shows distorts FID. Read §1–5; the appendix was not read. | Read |
| NOTE-306 | SDXL | A larger latent diffusion UNet with two text encoders, micro-conditioning on original size and crop, multi-aspect training and a refiner. Size conditioning is ablated on ImageNet: most of the gain is from keeping the data, and conditioning adds about 3 FID. Read §1–2.5 and Appendix B; the rest was skimmed. | Read |
| NOTE-310 | FLUX.1 Kontext | The FLUX.1 architecture, and an editing model built by appending context latents to the token sequence. It is a system report: human preference against other systems, latency, one reconstruction table, and no ablations. Main text and appendix A read. | Read |
| NOTE-333 v2 | Wan | An open 14B text-to-video recipe whose evidence is mostly systems measurement: 2D context parallelism cuts communication overhead from over 10% to under 1%, and caching and 8-bit kernels give 1.62× and 1.27×. Its three modelling ablations run on 1.3B text-to-image. Two are read from training loss, and on FID one of them favours the rejected option. The applications in §5 are demonstrated almost entirely by figures. | Read |
| NOTE-334 | Movie Gen | At 5B, flow matching beats v-prediction diffusion on quality by 3.3 annotation σ, but on text alignment by only 1.9σ. Video-native captions beat captions stitched from three frames on alignment by 2.9σ. Every σ in the paper is rater variance, and each ablation arm was trained once. The Llama 3 block beats a DiT block that differs only in norm, activation and bias. The scaling claim rests on image-only runs covering less than a quarter of a decade of compute. | Read |
| NOTE-335 | Self Forcing | Training a few-step autoregressive video model on its own rollouts, with a distribution-matching loss on the whole clip, beats teacher- and diffusion-forced training in matched few-step comparisons (VBench 84.31 against 82.76 and 82.32 chunk-wise), and does not drop when moved to frame-wise generation, where the baselines do. It runs at 17 fps with 0.69s latency on one H100, and its parity with the slower 1.3B base rests on a 14B teacher: the SiD variant, with a 1.3B teacher, scores 84.07 against the base’s 84.26. | Read |
| NOTE-336 | Stable Video Diffusion | At fixed architecture and a fixed step count, a video model pretrained on a curated quarter of a 9.8M-clip pool is preferred by human raters to one trained on the whole pool, and the lead survives high-quality fine-tuning (Elo +127 against +89 over an image-only start at 10k steps). Almost every result is a preference bar from 64 prompts with no numbers in the text, and two of the curation choices the paper shipped lost its own ablation. | Read |
| NOTE-338 v2 | Step-Video-T2V | A 30B text-to-video DiT on a 16×16×8 VAE. By the authors’ own benchmark the VAE reconstructs about as well as HunyuanVideo’s 8×8×4, and an independent measurement in Open-Sora 2.0 disagrees. Video DPO raises a tie-weighted preference score from 45% to 55% on 300 prompts. Every architecture comparison is reported without numbers, and the “state of the art” in the abstract is contradicted by the paper’s own Table 9. | Read |
| NOTE-339 | AnimateDiff | Temporal-attention modules trained on video over a frozen Stable Diffusion 1.5 plug into community checkpoints fine-tuned from the same base without further tuning. Against two baselines not built for the task it wins clearly only on smoothness: users ranked the training-free Text2Video-Zero higher on preserving the personalized domain (2.620 against 2.280), which is the property the design exists for. Version 1’s reported failure on stylized domains was dropped in version 2, and its “most checkpoints” became “any”. | Read |
| NOTE-341 | Video LDM (Align your Latents) | Video LDM makes a latent image diffusion model into a video model by inserting temporal layers and fine-tuning the decoder and upsampler on video. On driving scenes, an image-pretrained LDM with frozen spatial layers reaches FVD 534.17 against 1155.10 for the same design trained from scratch. The “images” are 1 fps frames of the same driving videos, and the pretrained arm gets 73K image steps the other does not. Decoder video fine-tuning cuts reconstruction FVD 390.88 → 32.94. | Read |
| NOTE-342 | MAGVIT-v2 | A causal 3D-CNN video tokenizer with lookup-free quantization (sign-binary latents plus an entropy penalty) improves a fixed ~307M masked token model on Kinetics-600 frame prediction from FVD 9.9 to 5.2 at equal decoding steps. The claim that LFQ, unlike VQ, keeps improving generation as the vocabulary grows rests on a figure with no numbers. “Beats diffusion” holds only with guidance (1.78 against MDT’s 1.79 at 256, 1.91 against VDM++‘s 2.65 at 512), and the strongest diffusion baselines at 512 are about 6× its size. | Read |
| NOTE-343 v2 | CausVid | Distilling a bidirectional video DiT into a 4-step block-causal student with distribution matching gives a streaming generator: 1.3s to the first frame and 9.4 fps, against 219.2s and 0.6 fps for the teacher. At equal initialization a bidirectional teacher makes a better causal student than a causal one (VBench 94.7 / 64.4 / 30.1 against 91.9 / 61.7 / 28.2). The causal teacher is also the weaker model, so the paper’s error-accumulation explanation is not isolated. | Read |
| NOTE-344 | HunyuanVideo | A 13B open text-to-video model whose one quantitative design argument is a pair of compute-optimal fits on a proxy family: the image fit puts more of each added FLOP into parameters (0.5634 against 0.4325), and the video fit, initialized from the image envelope, reverses it (0.3618 against 0.6289). No architectural or data choice is ablated, and the proxy differs from the shipped model in objective, text encoder and conditioning. | Read |
| NOTE-345 | Open-Sora 2.0 | An 11B video model initialized from FLUX, trained mostly at 256px, and adapted to 768px as text/image-to-video. The “$200k” is 4,160 H200 GPU-days for one final run on HunyuanVideo’s VAE. The 4×32×32 Video DC-AE is a separate, unconverged experiment whose cost is outside that figure. Every training-strategy choice that makes the run cheap is asserted, not ablated. | Read |
| NOTE-346 | CogVideoX | An open text-to-video DiT whose design choices are backed by curves rather than numbers. On full 3D against factorized attention, the text says only that factorized FVD is “much higher … in early steps” and that factorized training is “unstable and prone to collapse”. Model size, steps and resolution are unstated. The one measured number is the cost: full attention’s forward pass is 1.08×, 1.67× and 2.30× slower at three sizes. | Read |
| NOTE-347 | Video Diffusion Models | A standard Gaussian diffusion model with a space-time factorized 3D U-Net generates video. Appending 0, 4 or 8 independent frames to each training video takes text-to-video FVD from 202.28 to 68.11 to 57.84 at fixed batch size and steps, but the added frames are extra per-step compute and the paper’s own gloss is “more independent examples in a batch”. Reconstruction guidance beats the replacement method for autoregressive extension, 136 against 451 FVD. | Read |
| NOTE-348 | Imagen Video | A seven-model pixel-space cascade whose one well-measured result is that guided progressive distillation to 8 steps per stage cuts sampling from 618s to 35s. The CLIP metrics cannot tell the two apart: they fall slightly under constant guidance, rise under oscillating guidance, and rate every sample above the real videos. Joint image-video training, the choice later reports copied, is asserted without a number. | Read |
| NOTE-349 | Make-A-Video | Make-A-Video extends an unCLIP text-to-image model to video with identity-initialized pseudo-3D conv and attention layers trained on video with no captions. Text reaches the video decoder only through one CLIP image embedding, so text-only-in-video associations cannot be learned, and the discussion concedes this. System-level results are strong (zero-shot UCF-101 FVD 367.23, MSR-VTT CLIPSIM 0.3049), and there is no ablation of any design choice, image initialization included. | Read |
| NOTE-350 v2 | LTX-Video | A 1.9B video DiT on a 32×32×8, 128-channel VAE with no patchifier (1:8192 pixels to tokens), whose decoder also does the last denoising step. It is fast and preferred over same-size 2024 open models in a 20-rater survey. The VAE has no reconstruction metric at all, and the denoising decoder, the paper’s central idea, rests on one unquantified internal study. | Read |
| NOTE-351 | Emu Video | Generating a first frame and conditioning the video on it beats direct text-to-video at matched data, steps and trainable parameters (70.5% / 63.3% human win rate). Five one-change ablations with majority-of-5 votes on 307 prompts are the most controlled design evidence in the early video line. The freeze-versus-fine-tune result (55.0 / 58.1) is near chance, and by construction it cannot see the first frame. | Read |
| NOTE-362 | Integrated Gradients | Integrating the gradient along the straight line from a baseline gives the unique path attribution that is implementation-invariant, complete and symmetry-preserving. The justification is axioms, not a measurement: the paper argues that empirical evaluation of attributions is confounded, and its one comparison with another method is by eye. | Read |
| NOTE-365 v3 | Sanity Checks for Saliency Maps | Randomize the model’s weights, or train it on permuted labels, and see whether the saliency map changes. Gradients and GradCAM change. Guided Backprop is invariant to the higher layers. Integrated Gradients keeps the input’s structure while its sign decorrelates. Visual inspection cannot tell these cases apart, and that is the paper’s point. | Read |
| NOTE-371 | FLIP | Removing 50% of image patches in CLIP training halves the image encoder’s cost and lets the contrastive batch double at the same memory; ViT-L/16 on LAION-400M then matches its unmasked reproduction more than 3× faster and beats it by about a point at equal epochs. At equal batch the masking gains nothing, and at ViT-B/16 the full-schedule model is 0.2 behind — the improvement is the batch and the extra samples that masking pays for. | Read |
| NOTE-376 | Surya: heliophysics foundation model | A 366M-parameter transformer pretrained on full-resolution (4096²) multi-instrument SDO imagery to forecast the next frame, then rollout-tuned, forecasts solar dynamics zero-shot. With LoRA fine-tuning it beats task-specific baselines on four downstream heliophysics tasks. | Skimmed |