nucleation

LIT-1898: Stochastic Gradient Descent as Approximate Bayesian Inference

2 min read

Properties5
StatusActive — read in full 2026-10-31 (NOTE-1475); the paper the owner recalled at U4, and the right one, but what it shows is narrower than "the same process": near one non-degenerate minimum, the optimiser's noise scale ε/S plays the part of 1/N, and training time enters only as the wait for stationarity (THEORY-1255, THEORY-1257).
Version2
Tagslearning-theory, probabilistic-modeling, anthology-candidate
Cited as “paper” byNOTE-1475 — Mandt, Hoffman & Blei — SGD as approximate Bayesian inference
Cited as “source” byTHEORY-1255 — Near a non-degenerate minimum, constant-rate SGD's stationary distribution is a Gaussian set by learning rate, batch size and preconditioner, which can be tuned to match the posterior at a given sample size; training time enters only as the wait for stationarity, THEORY-1257 — Averaging constant-rate SGD iterates at stationarity gives a covariance that shrinks with the number of examples processed, and equals the Gaussian posterior's for that many observations when the noise covariance equals the Hessian

Graph View

Mandt, S., Hoffman, M. D., & Blei, D. M. (2017). Stochastic gradient descent as approximate Bayesian inference. Journal of Machine Learning Research, 18(134), 1–35.

Read in full in NOTE-1475.

Why this is here

The paper the owner recalled at U4 (“I think there’s a David M Blei paper that discusses this”) against A3’s claim that Bayesian posterior dominance as n grows “is not the same process” as gradient-based optimization over training time. A4 conceded the separation was “too strong”, with a limit kept here: the result relates SGD’s long-run distribution to a posterior for a fixed dataset, not training time t to sample size n. The reading bears A4’s limit out and sharpens it. The result assumes a positive-definite Hessian, so it concerns a regular model in one basin, where the posterior has no phases to order. What it maps to sample size is the optimiser’s noise scale, ε/S against 1/N (THEORY-1255). The one place examples processed map to a sample size is iterate averaging after equilibration (THEORY-1257). The record already reads this line of work through Kunin et al. (THEORY-209) and the underdamped stationary Gaussian (THEORY-779), which is this paper’s well-specified case with momentum; NOTE-293 cites this paper for McCandlish’s temperature. It grounds QUESTION-363. Its precursor is LIT-1899.

Backlinks