SOTA-414: To make an image diffusion model a dense geometric predictor, fine-tune it end to end as a one-step model at t = T with a task loss

Source

Martin Garcia, Knaebel, Schmidt, de Geus, Hermans and Leibe (2024; WACV 2025), LIT-687.

What to do

Start from an image diffusion model (Stable Diffusion, or a diffusion-fine-tuned depth model). Condition on the input image as usual. Then:

1. Fix t = T, and replace the noise input with zeros.
2. Convert the network’s prediction to a clean latent (with v-prediction, ẑ₀ = √ᾱ_T z_T − √(1−ᾱ_T) v̂) and decode it through the frozen VAE decoder.
3. Train on the task loss in output space: an affine-invariant L1 after a least-squares scale and shift for depth, and angular error for normals.

Use one forward pass at inference, with no ensembling.

Why it should work, and how far that is shown

For a near-unimodal conditional distribution, which is roughly what depth given an image is, the optimal prediction at t = T is close to the conditional mean. One step should then suffice, and a task loss trains exactly that prediction. The source argues this from EDM’s result on the final-step prediction and does not isolate it. Its measured finding is that the fine-tuned one-step model beats both the fixed one-step model and the ensembled multi-step one.

Evidence

Depth AbsRel, zero-shot: 5.2 / 9.6 / 6.2 / 5.8 / 30.2 on NYUv2 / KITTI / ETH3D / ScanNet / DIODE, against 5.5 / 9.9 / 6.5 / 6.4 / 30.8 for Marigold at 50 steps × 10. That is 1 network evaluation against 500. Starting from plain SD gives 5.4 / 9.6 / 6.4 / 5.8 / 30.3.

The recipe and these numbers are Martin Garcia et al.’s, LIT-687, which arrived at end-to-end fine-tuning after finding that Marigold’s single-step inference was broken by DDIM’s leading timestep spacing. Fixing the spacing alone still leaves one step worse than the 50 × 10 ensemble on all five depth sets (5.7 / 10.8 / 6.9 / 6.6 / 31.1), so the gain over the ensemble belongs to the fine-tuning, not the scheduler fix. On normals the fine-tuned one-step model reaches 16.2 / 14.7 / 15.8 / 33.5 mean angular error on NYUv2 / ScanNet / iBims-1 / Sintel, against Marigold’s 18.8 / 17.7 / 18.4 / 39.1.

Conditions

• Dense geometry with a near-unimodal target. The argument fails where the conditional distribution is multimodal, and image generation is the obvious case.
• One backbone, one fine-tuning budget, no seeds. The SD-against-Marigold initialization comparison rests on 0.0–0.2 AbsRel.
• Depends on SOTA-416 only for the baseline. The fine-tuning recipe fixes t = T directly. Scheduler spacing matters only to anyone comparing against multi-step inference, and that comparison is unfair until the spacing is trailing.
• Relies on v-prediction to make the t = T prediction a clean one. The source does not compare parameterizations. See SOTA-195.

Known implementations

• diffusion-e2e-ft, the source’s code.