generative-modeling.
63 of 378 NOTE documents. Back to the full index.
| # | Title | Summary | Status |
|---|---|---|---|
| NOTE-009 | Latent Diffusion Models | Train the diffusion model in the latent space of a pretrained autoencoder rather than in pixels. The autoencoder removes imperceptible detail; the diffusion model then spends its capacity on semantics instead of on high-frequency content nobody sees. | Read |
| NOTE-019 v2 | Elucidating the Design Space of Diffusion-Based Generative Models | Pulls diffusion’s tangled formulations apart into independent axes — sampler, training noise distribution, and preconditioning — and derives the preconditioning coefficients from a unit-variance requirement rather than choosing them. FID 1.79 on class-conditional CIFAR-10 at 35 network evaluations; a pretrained ImageNet-64 model improves 2.07 to 1.55 from the sampler alone. | Read |
| NOTE-040 | Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding | Finds that scaling a frozen text-only language model improves text-to-image sample quality significantly more than scaling the image diffusion model — an allocation result, not an architecture one. Also diagnoses why high guidance weights destroy images (predictions leave the training range and compound through iterative sampling) and fixes it with dynamic thresholding. | Read |
| NOTE-043 v2 | DPM-Solver: A Fast ODE Solver for Diffusion Probabilistic Model Sampling | Notices that the diffusion ODE is semi-linear, solves its linear part analytically, and applies a numerical method only to the intractable neural-network integral. Training-free, 10–20 function evaluations, 4.70 FID at 10 NFE on CIFAR-10, and DDIM turns out to be exactly the first-order case. | Read |
| NOTE-048 | Progressive Distillation for Fast Sampling of Diffusion Models | Repeatedly halves a diffusion sampler’s step count by distilling two teacher steps into one student step, from 8192 down to 4 at FID 3.0 on CIFAR-10, for no more total compute than training the original model. Also where v-prediction comes from — and the reason for it is that ε-prediction’s implied x̂ blows up as signal-to-noise goes to zero. | Read |
| NOTE-053 v2 | Consistency Models | Trains a model to map any point on a diffusion trajectory directly to its origin, so one network evaluation generates a sample — FID 3.55 on CIFAR-10 and 6.20 on ImageNet-64 in one step. Trainable either by distilling a diffusion model or from scratch, and the from-scratch mode is what makes it a model family rather than a distillation trick. | Read |
| NOTE-063 | One Transformer Fits All Distributions in Multi-Modal Diffusion at Scale | Observes that marginal, conditional and joint distributions are all “predict the noise in perturbed data” if you allow each modality its own timestep. One network trained with per-modality timesteps then serves text-to-image, image-to-text, joint generation and unconditional generation — and classifier-free guidance falls out for free. | Read |
| NOTE-064 | Hierarchical Text-Conditional Image Generation with CLIP Latents | Splits text-to-image into a prior that predicts a CLIP image embedding from a caption and a decoder that renders from that embedding. The split buys diversity rather than fidelity — humans slightly prefer GLIDE’s photorealism, and strongly prefer unCLIP’s diversity. Its own reconstruction figures show the CLIP embedding loses attribute binding. | Read |
| NOTE-067 v2 | Denoising Diffusion Implicit Models | Generalises DDPM to a family of non-Markovian forward processes that share DDPM’s exact training objective, so the sampler can be replaced without retraining. Picking the deterministic member gives 10–50× fewer steps, plus latent interpolation and encoding that a stochastic sampler cannot do. Not a continuous-time or SDE paper — that is a different one. | Read |
| NOTE-068 | Denoising Diffusion Probabilistic Models | The paper that made diffusion work. Its Table 2 is the underquoted part: predicting ε rather than the posterior mean matters only under the simplified objective, learned reverse-process variances destabilise training, and the principled variational bound gives better codelengths while the unprincipled simplified loss gives better samples. | Read |
| NOTE-166 | Scaling Diffusion Language Models via Adaptation from Autoregressive Models | Continual pretraining on under 200B tokens converts GPT2 and LLaMA at 127M-7B into diffusion language models competitive with their AR counterparts. The argument is economic before it is technical — diffusion LMs are hard to train at scale and AR checkpoints are abundant — and the capability that pays for it is infilling without prompt re-ordering. | Read |
| NOTE-192 | Diffusion Beats AR in Data-Constrained Settings | Refits LIT-166’s data-constrained scaling law with the objective swapped from autoregressive to masked diffusion. The half-life of data reuse R* goes from 31.93 to 512.85. Diffusion loses badly at Chinchilla-optimal compute and wins past a crossover given in closed form. | Read |
| NOTE-196 | Scaling Rectified Flow Transformers | A 61-way sweep over diffusion and rectified-flow formulations settles on a straight-line path with logit-normal timestep sampling — rectified flow with uniform timesteps does not win. Adds a resolution-dependent timestep shift, MMDiT’s per-modality weights with joint attention, and QK-RMSNorm for high-resolution stability. 8B, weights released. | Read |
| NOTE-197 v2 | SiT: Scalable Interpolant Transformers | Walks one variable at a time from DiT to SiT with the architecture, parameter count and GFLOPs held fixed: discrete to continuous time, score to velocity prediction, variance-preserving to linear interpolant, ODE to a tuned SDE sampler. Velocity and the linear interpolant carry most of the gain, and the sampler’s diffusion coefficient turns out to be choosable after training. | Read |
| NOTE-198 | Variational Diffusion Models | Integrating the diffusion VLB over signal-to-noise ratio rather than time collapses the schedule out of the expression: in continuous time the bound depends on it only through the endpoint SNRs. VP and VE specifications are therefore equivalent, and the freed schedule shape is spent on minimizing the loss estimator’s variance. | Read |
| NOTE-226 | NoProp | Reading it: the interesting number is in the train column, not the test one. NoProp matches backprop on test while sitting 8-15 points behind on train, which on MNIST and CIFAR costs nothing and would be the whole story anywhere fitting is the binding constraint. The paper does not mention it. | Read |
| NOTE-228 v2 | The Diffusion Duality | Reading it: the theorem is elegant and the ablation says it accounts for under half the likelihood gain. What the paper actually establishes is a regime split — masked diffusion wins on quality at many steps, uniform-state wins below about 32, and the reason is architectural rather than tuned. | Read |
| NOTE-232 | Mixture-of-Transformers | Read to test whether it satisfies SOTA-262’s promotion condition. It does, as that condition was written — second group, per-modality weights against a shared-weight backbone, FLOPs held identical. The component ablation is a bonus the condition did not ask for and the practice needed. | Read |
| NOTE-237 | gen2seg | Read as the counter-route to SAM’s data engine. The headline (narrow finetuning, broad generalization) is well evidenced but the argument lives in the controls, which are unusually good: the same backbone under a conventional head scores ~1.5 mIoU, and swapping only the decoder recovers part of the gap. | Read |
| NOTE-239 | Denoising-corrected gradient guidance | Read for a practice the record’s diffusion line was missing. The ordering change is well motivated and cheaply adopted; the theory is proved under idealized assumptions the authors name; and the useful experimental result is that the standard method was optimizing the plan rather than the executed outcome. | Read |
| NOTE-240 | Noise hypernetworks | Read as the amortized counterpart to SOTA-301. The headline — recover half of test-time optimization at 1/100th the latency — is well measured. The finding that carries further is the control: direct reward fine-tuning of the same model makes it worse, and worse the more steps you sample. | Read |
| NOTE-243 | GaussianToken and the cost of the continuous half | Read from the #180 worklist and filed without a practice. The idea — let the quantization units move and size themselves instead of sitting on a grid — is a good one, and the evidence for it cannot be separated from the five continuous parameters each token carries alongside its index. The comparison is at equal token count, and nobody computed bits. | Read |
| NOTE-245 | VQ-VAE: what the three-term loss actually does | Read to give the discrete-tokenizer line a root. The durable content is three loss terms with three different owners and a gradient that is copied rather than derived — an engineering settlement that every descendant still runs unchanged, and that the paper is unusually honest about having chosen rather than justified. | Read |
| NOTE-246 | VQGAN: the loss that made f=16 survivable | Read as the second step of the tokenizer trunk. The architecture is VQ-VAE’s; what is new is a perceptual loss, a patch discriminator, and an adaptive weight that balances them without per-dataset tuning. That loss is the reason a 256×256 image can become 256 tokens and still look like anything, and it is why this line exists at all. | Read |
| NOTE-247 | ViT-VQGAN: where the codebook advice comes from | Read to give SOTA-306 its origin, and it corrected the practice in three places. The advice is not “use small code vectors” but “factorize lookup from embedding”; ℓ₂-normalization is ablated and is the largest single effect in the table; and the dimension-4 row shows 96% codebook usage with near-worst FID, so utilization is necessary and not sufficient. | Read |
| NOTE-248 | The FID Lottery: which lottery, and how big | Read from the #180 worklist, taken ahead of its dwell rank because SOTA-305 had just landed on the adjacent question. The finding that matters is not that FID is noisy but which noise dominates: the training run, by 3.2×, in a term no amount of extra sampling touches — and a control that rules out the boring explanation. | Read |
| NOTE-297 | Warm Diffusion | Blur and noise mixed in one forward process, with a two-headed model that denoises to the blurry image and deblurs the residual. A blur-to-noise ratio of 0.5, chosen from image and noise spectra, edges EDM on CIFAR-10 (1.85 against 1.97). High ratios degrade sharply, which is the evidence that noise keeps samples on the manifold. Read in full (8 pages). | Read |
| NOTE-298 | Diffusion Forcing | Independent per-token noise levels during training give one causal model both autoregressive sampling and horizon-wide guidance, with sampling schedules chosen at inference. Stable long video rollouts are shown qualitatively. Planning and robot results are measured. Read §1–5; the appendix proofs and time-series results were not read. | Read |
| NOTE-299 | Cold Diffusion | Diffusion generalized to deterministic degradations (blur, masking, downsampling, snow) with a restorer and a sampler that is exact for degradations linear in severity. Conditional restoration works. Noiseless unconditional generation is far worse than noise diffusion, and adding a little noise recovers half the gap. Read §1–5.3; the appendix was not read. | Read |
| NOTE-300 | StyleGAN3 | Aliasing in the generator glues fine detail to pixel coordinates. Continuous-signal design, with 2×-upsampled filtered nonlinearities, Fourier input and no noise, gives translation- and rotation-equivariant generators at StyleGAN2’s FID. Read §1–3.2 and Figure 3; the internal representation analysis and the appendices were skimmed. | Read |
| NOTE-301 | StyleGAN2 | Diagnoses two StyleGAN artifacts and fixes each at its cause: weight demodulation instead of instance normalization, and a fixed skip/residual architecture instead of progressive growing. Adds lazy regularization and path-length regularization. Read §1–4.1 and Tables 1–2; §4.2 onward and the appendices were skimmed. | Read |
| NOTE-302 | CycleGAN | Unpaired translation with two adversarial losses and a bidirectional cycle-consistency loss. It works for appearance changes and fails for geometric ones. The ablation supports the cycle term against GAN-only, but not strictly the bidirectional form over one direction. Read §1–6; the appendix was skimmed. | Read |
| NOTE-303 | Projected GAN | A GAN discriminator on frozen, randomly mixed, multi-scale pretrained features reaches prior-best FIDs up to 40× faster and sets new FIDs on 22 datasets. Everything is measured in FID or ImageNet-derived metrics, and the discriminator is ImageNet-pretrained, which LIT-563 later shows distorts FID. Read §1–5; the appendix was not read. | Read |
| NOTE-304 | The Role of ImageNet Classes in FID | FID can be lowered by two-thirds without changing a generator, by resampling its outputs to match ImageNet-class statistics. Non-ImageNet feature spaces barely move. An ImageNet-pretrained discriminator gets the same effect by accident. Read §1–5; the appendices were skimmed. | Read |
| NOTE-306 | SDXL | A larger latent diffusion UNet with two text encoders, micro-conditioning on original size and crop, multi-aspect training and a refiner. Size conditioning is ablated on ImageNet: most of the gain is from keeping the data, and conditioning adds about 3 FID. Read §1–2.5 and Appendix B; the rest was skimmed. | Read |
| NOTE-310 | FLUX.1 Kontext | The FLUX.1 architecture, and an editing model built by appending context latents to the token sequence. It is a system report: human preference against other systems, latency, one reconstruction table, and no ablations. Main text and appendix A read. | Read |
| NOTE-311 | FLUX.2 latent-space report | A controlled comparison of four autoencoder latents for flow models on ImageNet, with the timestep schedule swept per latent. The schedule decides the ranking. FLUX.2’s VAE gets near-RAE learnability with the best reconstruction. Read in full from the rendered report, except the interactive Table 2, which did not load. | Read |
| NOTE-320 | AlphaFold 3 | AlphaFold generalized to all biomolecular complexes, with a diffusion head over atom coordinates. It gains on ligands, nucleic acids and antibodies, and inherits generative failure modes (hallucination, chirality) that it patches with distillation and ranking. Main text read. Extended Data and supplementary methods not read. | Read |
| NOTE-322 v2 | Latent Dirichlet Allocation | The bag-of-words assumption, read through de Finetti, implies a document-level mixture. LDA is that mixture with Dirichlet topic proportions. It generalizes better than one-topic-per-document and per-document-parameter models by held-out perplexity. Main text read in full; the variational derivations in the appendices skimmed. | Read |
| NOTE-333 v2 | Wan | An open 14B text-to-video recipe whose evidence is mostly systems measurement: 2D context parallelism cuts communication overhead from over 10% to under 1%, and caching and 8-bit kernels give 1.62× and 1.27×. Its three modelling ablations run on 1.3B text-to-image. Two are read from training loss, and on FID one of them favours the rejected option. The applications in §5 are demonstrated almost entirely by figures. | Read |
| NOTE-334 | Movie Gen | At 5B, flow matching beats v-prediction diffusion on quality by 3.3 annotation σ, but on text alignment by only 1.9σ. Video-native captions beat captions stitched from three frames on alignment by 2.9σ. Every σ in the paper is rater variance, and each ablation arm was trained once. The Llama 3 block beats a DiT block that differs only in norm, activation and bias. The scaling claim rests on image-only runs covering less than a quarter of a decade of compute. | Read |
| NOTE-335 | Self Forcing | Training a few-step autoregressive video model on its own rollouts, with a distribution-matching loss on the whole clip, beats teacher- and diffusion-forced training in matched few-step comparisons (VBench 84.31 against 82.76 and 82.32 chunk-wise), and does not drop when moved to frame-wise generation, where the baselines do. It runs at 17 fps with 0.69s latency on one H100, and its parity with the slower 1.3B base rests on a 14B teacher: the SiD variant, with a 1.3B teacher, scores 84.07 against the base’s 84.26. | Read |
| NOTE-336 | Stable Video Diffusion | At fixed architecture and a fixed step count, a video model pretrained on a curated quarter of a 9.8M-clip pool is preferred by human raters to one trained on the whole pool, and the lead survives high-quality fine-tuning (Elo +127 against +89 over an image-only start at 10k steps). Almost every result is a preference bar from 64 prompts with no numbers in the text, and two of the curation choices the paper shipped lost its own ablation. | Read |
| NOTE-337 | Rectified Flow | Regressing a velocity onto x1 − x0 along the straight interpolation gives an ODE that preserves the marginals and does not increase any convex transport cost, and retraining on its own couplings (“reflow”) provably straightens it at O(1/K). On CIFAR-10 the single-pass model scores FID 2.58 against 3.93 for the VP probability-flow ODE at the same architecture, with training budget unstated. Most of the one-step result comes from distillation (378 → 6.18 with no reflow), and reflow adds 6.18 → 4.85. | Read |
| NOTE-338 v2 | Step-Video-T2V | A 30B text-to-video DiT on a 16×16×8 VAE. By the authors’ own benchmark the VAE reconstructs about as well as HunyuanVideo’s 8×8×4, and an independent measurement in Open-Sora 2.0 disagrees. Video DPO raises a tie-weighted preference score from 45% to 55% on 300 prompts. Every architecture comparison is reported without numbers, and the “state of the art” in the abstract is contradicted by the paper’s own Table 9. | Read |
| NOTE-339 | AnimateDiff | Temporal-attention modules trained on video over a frozen Stable Diffusion 1.5 plug into community checkpoints fine-tuned from the same base without further tuning. Against two baselines not built for the task it wins clearly only on smoothness: users ranked the training-free Text2Video-Zero higher on preserving the personalized domain (2.620 against 2.280), which is the property the design exists for. Version 1’s reported failure on stylized domains was dropped in version 2, and its “most checkpoints” became “any”. | Read |
| NOTE-340 v2 | Flow Matching | Regressing onto a per-sample conditional vector field trains a continuous normalizing flow without simulation, with the same gradient as the intractable marginal objective. With the U-Net, hyperparameters and epochs held fixed, the straight “OT” path beats the VP diffusion path under that objective at CIFAR-10 and ImageNet 32/64 with uniform timesteps (FID 6.35 against 8.06 on CIFAR-10). The objective’s own advantage over score matching is mixed, and the likelihood gains at ImageNet scale are 0.01 bpd. | Read |
| NOTE-341 | Video LDM (Align your Latents) | Video LDM makes a latent image diffusion model into a video model by inserting temporal layers and fine-tuning the decoder and upsampler on video. On driving scenes, an image-pretrained LDM with frozen spatial layers reaches FVD 534.17 against 1155.10 for the same design trained from scratch. The “images” are 1 fps frames of the same driving videos, and the pretrained arm gets 73K image steps the other does not. Decoder video fine-tuning cuts reconstruction FVD 390.88 → 32.94. | Read |
| NOTE-342 | MAGVIT-v2 | A causal 3D-CNN video tokenizer with lookup-free quantization (sign-binary latents plus an entropy penalty) improves a fixed ~307M masked token model on Kinetics-600 frame prediction from FVD 9.9 to 5.2 at equal decoding steps. The claim that LFQ, unlike VQ, keeps improving generation as the vocabulary grows rests on a figure with no numbers. “Beats diffusion” holds only with guidance (1.78 against MDT’s 1.79 at 256, 1.91 against VDM++‘s 2.65 at 512), and the strongest diffusion baselines at 512 are about 6× its size. | Read |
| NOTE-343 v2 | CausVid | Distilling a bidirectional video DiT into a 4-step block-causal student with distribution matching gives a streaming generator: 1.3s to the first frame and 9.4 fps, against 219.2s and 0.6 fps for the teacher. At equal initialization a bidirectional teacher makes a better causal student than a causal one (VBench 94.7 / 64.4 / 30.1 against 91.9 / 61.7 / 28.2). The causal teacher is also the weaker model, so the paper’s error-accumulation explanation is not isolated. | Read |
| NOTE-344 | HunyuanVideo | A 13B open text-to-video model whose one quantitative design argument is a pair of compute-optimal fits on a proxy family: the image fit puts more of each added FLOP into parameters (0.5634 against 0.4325), and the video fit, initialized from the image envelope, reverses it (0.3618 against 0.6289). No architectural or data choice is ablated, and the proxy differs from the shipped model in objective, text encoder and conditioning. | Read |
| NOTE-345 | Open-Sora 2.0 | An 11B video model initialized from FLUX, trained mostly at 256px, and adapted to 768px as text/image-to-video. The “$200k” is 4,160 H200 GPU-days for one final run on HunyuanVideo’s VAE. The 4×32×32 Video DC-AE is a separate, unconverged experiment whose cost is outside that figure. Every training-strategy choice that makes the run cheap is asserted, not ablated. | Read |
| NOTE-346 | CogVideoX | An open text-to-video DiT whose design choices are backed by curves rather than numbers. On full 3D against factorized attention, the text says only that factorized FVD is “much higher … in early steps” and that factorized training is “unstable and prone to collapse”. Model size, steps and resolution are unstated. The one measured number is the cost: full attention’s forward pass is 1.08×, 1.67× and 2.30× slower at three sizes. | Read |
| NOTE-347 | Video Diffusion Models | A standard Gaussian diffusion model with a space-time factorized 3D U-Net generates video. Appending 0, 4 or 8 independent frames to each training video takes text-to-video FVD from 202.28 to 68.11 to 57.84 at fixed batch size and steps, but the added frames are extra per-step compute and the paper’s own gloss is “more independent examples in a batch”. Reconstruction guidance beats the replacement method for autoregressive extension, 136 against 451 FVD. | Read |
| NOTE-348 | Imagen Video | A seven-model pixel-space cascade whose one well-measured result is that guided progressive distillation to 8 steps per stage cuts sampling from 618s to 35s. The CLIP metrics cannot tell the two apart: they fall slightly under constant guidance, rise under oscillating guidance, and rate every sample above the real videos. Joint image-video training, the choice later reports copied, is asserted without a number. | Read |
| NOTE-349 | Make-A-Video | Make-A-Video extends an unCLIP text-to-image model to video with identity-initialized pseudo-3D conv and attention layers trained on video with no captions. Text reaches the video decoder only through one CLIP image embedding, so text-only-in-video associations cannot be learned, and the discussion concedes this. System-level results are strong (zero-shot UCF-101 FVD 367.23, MSR-VTT CLIPSIM 0.3049), and there is no ablation of any design choice, image initialization included. | Read |
| NOTE-350 v2 | LTX-Video | A 1.9B video DiT on a 32×32×8, 128-channel VAE with no patchifier (1:8192 pixels to tokens), whose decoder also does the last denoising step. It is fast and preferred over same-size 2024 open models in a 20-rater survey. The VAE has no reconstruction metric at all, and the denoising decoder, the paper’s central idea, rests on one unquantified internal study. | Read |
| NOTE-351 | Emu Video | Generating a first frame and conditioning the video on it beats direct text-to-video at matched data, steps and trainable parameters (70.5% / 63.3% human win rate). Five one-change ablations with majority-of-5 votes on 307 prompts are the most controlled design evidence in the early video line. The freeze-versus-fine-tune result (55.0 / 58.1) is near chance, and by construction it cannot see the first frame. | Read |
| NOTE-357 | SEDD | Score entropy makes the ratio view of discrete diffusion trainable and gives it a likelihood bound; with it, absorbing-state diffusion reaches a near-tie with a retrained autoregressive model on LM1B. “Beats GPT-2” is 3 of 5 zero-shot sets against a model trained on different data. The controlled result that lasts is that masking beats uniform corruption on every table. | Read |
| NOTE-358 | Diffusion Models Beat GANs | An ablated U-Net (ADM) and a classifier-gradient scale that trades recall for precision take diffusion past BigGAN-deep on ImageNet FID. The architecture search is single-run; every guided number is the best of a small scale sweep scored on the metric it reports; and the trade is non-monotone — a scale of 1 makes an unconditional model’s FID worse (26.21 → 33.03). The claim that large scales produce no adversarial examples is asserted in the introduction and checked only by eye and in Inception feature space. | Read |
| NOTE-360 | MDLM | Masked diffusion’s ELBO is a schedule-weighted average of masked-LM losses, and with a modern recipe it is the best discrete diffusion language model at 110M parameters. It does not beat autoregression in-domain, even with the AR baseline given half the optimizer steps; the paper’s own checklist names that as its limitation. | Read |
| NOTE-369 | Min-SNR weighting | Capping the x0-space loss weight at min(SNR, 5) speeds diffusion training under x0, ε and v prediction, and makes ε output trainable under weightings that otherwise diverge. The 3.4× headline is against constant-weighted x0-prediction. Against plain ε-MSE on a UNet, FID goes 8.55 → 7.32 at 200K iterations and 4.21 → 4.14 at 1M, single runs. The multi-task “conflicting gradients” account is motivated by one probe and a proxy objective, and is not tested against FID. | Read |
| NOTE-370 | EDM2 | Magnitude-preserving layers plus forced weight normalization take the ADM U-Net from FID 8.00 to 2.56 on ImageNet-512 at equal compute, in a cumulative single-model ladder whose steps are all far larger than FID’s noise. Post-hoc EMA reconstructs any averaging length from two stored power-function averages, and what it shows is that the best length depends on architecture, learning rate, guidance weight and the metric — the last by 2% against 14% under guidance. | Read |