nucleation

LIT-677: Unveiling mode-connectivity of the ELBO landscape

2 min read

Properties3
StatusProposed — read in full 2026-10-03 (NOTE-522); watching, as the only work in the record to carry mode connectivity from neural networks to variational inference. It claims that local maxima of the LDA ELBO, from two SVI runs on the New York Times corpus, are joined by essentially flat maximum-energy paths found with the simplified string method, that paths flatten as the number of topics grows, and that statistical degeneracy explains this. The three-page workshop paper reports none of the results it describes: no path, no ELBO values, no synthetic-data experiment; its one figure is a schematic. What would settle it: the experiments themselves, which are not in this version.
Tagsloss-landscapes, probabilistic-modeling, anthology-candidate
Cited as “paper” byNOTE-522 — Unveiling mode-connectivity of the ELBO landscape

Graph View

Edith Zhang and David Blei (2021), Bayesian Deep Learning workshop, NeurIPS 2021 — https://bayesiandeeplearning.org/2021/papers/3.pdf

Key takeaways

  • The claim. Local optima of the ELBO for latent Dirichlet allocation, fitted with stochastic variational inference, are not isolated: two optima from different random initialisations are connected by a path along which the ELBO stays essentially at its optimal value. The optima “lie on a connected subset in parameter space” (abstract, §1.1).
  • The method. The simplified string method of E, Ren and Vanden-Eijnden, built to find transition paths in chemistry: 15 beads on the line between two optima, each taking a natural-gradient (batch VI) step, then re-spaced evenly along the string, repeated to convergence (§2, App. C).
  • The explanation. Over-parameterisation, with more topics K than the corpus has, lets a reassignment of one word be absorbed by small shifts in the topics, as with neurons in an over-fitted network (§3). Beyond that, “continuous statistical degeneracy”, a connected set of optimal parameters, would produce near-optimal paths even in an under-parameterised model (§1).

Standing in the record

Filed with the owner’s batch on mode connectivity and model merging, under loss-landscapes (ADR-027), with probabilistic-modeling for its subject: variational inference on a topic model, outside deep learning. The anthology flag follows ADR-027’s batch rule; of the nine works in this filing, it is the one least likely to belong there.

Proposed, not Active. The paper says its experiments show near-optimal paths, flattening with data size and K, and paths in an under-parameterised model. None of these results appears in it. The only figure is a schematic of two topic sets, §1 refers to an “experimental section” that the paper does not have, and App. C’s loop says “repeat steps 3 and 4” where steps 2 and 3 are meant. Its value now is the conjecture and the method, transplanted from chemistry by way of Draxler et al. (LIT-653), whose method it says it resembles. It sets its theoretical discussion closest to Kuditipudi et al. (LIT-671), on over-parameterisation and resilience. It builds on neither and measures neither, so no relation is declared.

“Statistical degeneracy” is the record’s singular-model question in another vocabulary: Watanabe’s singular learning theory (LIT-354, unread) is the frame in which non-identifiable models have connected sets of optima, and the batch’s paper on generative and contrastive models (LIT-660) cites it for the same intuition. Neither paper connects the two.

Backlinks