Antonio Sclocchi, Alessandro Favero and Matthieu Wyart (2024), Proceedings of the National Academy of Sciences 122(1), e2408799121 (2025) — ARXIV-2402.16991, DOI-10.1073/pnas.2408799121
Key takeaways
- The question is inference, not learning. The data are the Random Hierarchy Model of LIT-877 with v classes, a vocabulary of v symbols, branching s, depth L and m rules per symbol drawn without replacement from the v^s tuples. The rules are assumed known, so the denoiser is the exact posterior, computed by belief propagation on the tree. This is the ideal a perfectly generalising diffusion model would match, not a model of one trained on finite data.
- A sharp threshold for the class, smooth decay below it. Corrupt each leaf with belief 1 − ε in its true value and ε spread uniformly. Averaging the upward messages over the random rules gives a one-dimensional map p′ = F(p) = (p^s + f (m−1)/(mv−1) (1 − p^s)) / (p^s + f (1 − p^s)), with f = (mv − 1)/(v^s − 1) ≈ m/v^(s−1) the fraction of tuples a level uses. When F′(1) < 1, roughly s f < 1, the perfect state p = 1 is stable and an unstable fixed point between 1/v and 1 separates noise levels from which the class is recovered from those from which it is lost: as L → ∞, a phase transition at ε*(s f). When s f > 1 the class is lost at any noise. Leaves and low latents have no threshold: their reconstruction falls smoothly with ε (Figs. 5–7, S2–S6).
- Hence old parts in a new whole. Past the threshold the resampled datum belongs to a random class while many of its low-level features are the original’s, so curves of “still correct” for the top and bottom levels cross. This is the paper’s account of observation (iii), contrary to Ho et al.’s remark that at large noise “only large scale features are preserved”.
- The approximation is checked, not proved. The annealed messages match exact BP on single realisations (v = 32, m = 8, s = 2, L = 10) for the upward pass and for the marginals; downward messages fluctuate strongly at small ε, because synonyms make a child underdetermined by its parent. The ε-noise model stands in for Gaussian diffusion of one-hot leaves through an effective ε(t) (SI 3).
- The ImageNet observation. Noising ImageNet images to time t with an unconditional 256×256 DDPM (Dhariwal and Nichol) and denoising back, the cosine similarity of a ConvNeXt’s logits between original and sample falls sharply near t ≈ T/2 while early layers decline smoothly, and the curves invert order around the drop; three ResNets agree (Fig. 2, S8). A CNN trained on an RHM (L = 7, v = 16, m = 4), fed BP-resampled data, shows the same inversion (Fig. 7b); a classifier of a two-mode Gaussian mixture shows the drop but no inversion (SI 5).
Standing in the record
Filed on 2026-10-09 at the owner’s request, as one of the works cited by the batch of thirteen works on hierarchy and hyperbolic geometry (nucleation#113, LIT-865 to LIT-877; see that day’s curation entry) that neither record held. The batch work that cites it is LIT-877, Cagnetta et al.’s Random Hierarchy Model, which names it as a later paper of the same group building on that model. Read on its own merits (NOTE-685); the reading produces THEORY-199.
Beside THEORY-195 it is the other half of the RHM programme: that account is about how many examples a network needs to learn the grammar, this one about what an ideal denoiser that knows the grammar does with noise. It does not bear on QUESTION-025: its hierarchy is constituency, as in LIT-877, and it measures no co-occurrence, embedding direction or concept lattice. It is flagged for the anthology because its experimental half is about diffusion models.