Bao et al. (2023) — ARXIV-2303.06555
Key takeaways
- One insight, stated by the authors: “learning diffusion models for marginal, conditional, and joint distributions can be unified as predicting the noise in the perturbed data, where the perturbation levels (i.e. timesteps) can be different for different modalities”
- So the timestep vector is the task specification. Set a modality’s timestep to 0 and it is the condition; perturb both and you are learning the joint. One network covers text-to-image, image-to-text, joint and unconditional generation with no architectural change between them
- Classifier-free guidance comes for free (§3.2): the unconditional model is already present as the same network at a different timestep setting, so no separate condition-dropping training run is needed
- Conditioning becomes a value in the input rather than a structure in the architecture, which turns a discrete set of tasks into a continuous space
- The authors state their own negative result: “the text generated by our implementation is not that smooth, mainly because the text data is noisy” — an attribution they do not isolate
Standing in the anthology
Read — NOTE-063. Retagged to generative-modeling: the contribution is a
training-and-conditioning scheme, and the architecture is a stock transformer by
the paper’s own account.
Nothing is sourced here and the reading files no practice — the record’s generative-modelling practices are single-modality.
Of the takeaways replaced, “universal sequence modeling” is not what this paper is about. It is a diffusion paper, and the unification is over distributions — marginal, conditional, joint — not over sequences.