Contribution
It proposes a description of SGD’s “long runtime behaviour” (abstract) in which three quantities meet: the local learning coefficient (LLC) λ(w) of singular learning theory, read as a mass fractal dimension of low-loss regions (§3.2.1, App. B.1); a spectral dimension d_s, the exponent of the volume of visited states, V_s(t) ∝ t^(d_s/2) (Definition 3.1); and a walk dimension d_walk = 2λ/d_s, the Alexander–Orbach relation (Theorem 3.1), which sets displacement R(t) ∝ t^(1/d_walk). From a time-fractional Fokker–Planck equation it derives a local steady state exp(−γL_m/D_ξ) (Lemma 3.3) and reads it as a tempered posterior (Corollary 3.2). The experiments fit d_s from displacement and compare it with LLC estimates, and in one toy setting compare cluster masses of SGD solutions with an SGLD posterior.
Key insight
Late in training SGD does not diffuse like Brownian motion. The displacement grows more slowly, as if the weights moved through a medium whose open volume scales with the LLC. If a region’s effective diffusion is a constant D_ξ, SGD’s stationary law there is a Gibbs distribution at temperature D_ξ, which is the posterior at sample size 1/D_ξ. On this account SGD is “almost Bayesian”: Bayesian within a region, at a temperature set by the dynamics.
Assumptions
- Large batch “so that the estimation noise … doesn’t dominate the dynamics” (§3.1).
- A steady state exists. The key lemma assumes “there exists steady-state solutions” (Lemma 3.3); the main text adds “(if it exists)”. §5.1 concedes “In general, SGD iterates do not converge to exact equilibria” and that a non-equilibrium steady state is possible. Proposition D.1 shows there is none if displacement has even a small linear component.
- Diffusion tensor ∝ Hessian, and nearly all Hessian eigenvalues ≈ 0 (App. D.2, citing Xie et al. and Sagun et al.).
- An effective scalar diffusion coefficient, D_ξ = ξ^(2 − d_walk), at a chosen length scale ξ (Lemma 3.2, Corollary 3.1), constant on the region considered.
- The Alexander–Orbach relation holds locally near critical points (Theorem 3.1, stated from Hambly et al. and not proved here); “in cases where there are no nearby saddle points (which is more common in early training), this relation does not need to hold” (p. 5).
- A single spectral dimension over training for vanilla SGD (§3.3.1).
Key results
- Theorem 3.1. d_walk(t) = 2λ(w_t)/d_s near critical points.
- Theorem 3.2 / D.2. With isotropic noise D along degenerate directions, the first-passage time through a ball around w* scales as ε^λ(w*)/(DC), C the capacity of the escape set. The proof replaces |Ω| by V(ε) in a cited mean-first-passage formula for porous media; it is a one-line substitution.
- Lemma 3.3. Where D_ξ is constant on W ⊂ 𝒲 and a steady state exists, p_s(w) ∝ exp(−γL_m(w)/D_ξ).
- Corollary 3.2. With γ = 1 and log loss, p_s(w)^(mD_ξ) ∝ p(X_m|w), so p(w|X_m) ∝ ρ(w)p_s(w)^(mD_ξ). This is the identity (e^(−L_m/D_ξ))^(mD_ξ) = e^(−mL_m). Equivalently, p_s is the posterior with a flat prior at effective sample size 1/D_ξ.
- Lemma 3.4 and Corollary 3.3. d_s ≤ λ(w(t)) as t → ∞, and d_s ≤ the time-averaged λ. The proof argues d_walk > 2 from the loss being non-constant, then d_s = 2λ/d_walk < λ.
- Table 1. λ, d_s, α = d_s/(2λ) (the displacement exponent; I checked the column is this ratio) and r²: TinyStories-1M 32, 21.4, 0.33, 0.98; TinyLlama-15M 76.1, 48.3, 0.32, 0.98; TinyStories-33M 39.3, 38.7, 0.49, 0.98; ResNet18 72.05, 0.57, 0.004, ≈1; ResNet34 73.5, 0.62, 0.004, ≈1; VGG16 159.7, 0.14, 0.001, ≈1. The vision models’ α ≈ 0.004 means their displacement barely grows; the fit is then near-flat and r² ≈ 1 says little.
- Posterior concentration (§4.2, Table 2). 500 two-hidden-layer ReLU networks on 512 moons samples, SGD at rate 0.01 for 200 epochs; the “Bayesian posterior” is SGLD seeded at the SGD solutions with lowest loss and LLC (App. G.4), 1000 draws at step 10⁻⁵. After tempering with ξ = 0.5, chosen by the KL between empirical and theoretical SGD distributions (Fig. 3), the cluster masses agree: KL 0.009, Wasserstein 0.002, JS 0.003. Untempered, SGD prefers a different cluster (C1) from the posterior (Fig. 4b).
- Appendices H–J. Ablations on MNIST: Adam has much smaller d_s than SGD (0.41 against 7.8 mean) and its LLC tracks displacement less well; CIFAR-10 convnets and resnets show early super-diffusion growing with model size; Table 8’s 50 identical MNIST runs give d_s from −0.33 to 15.5 at LLC 18.4–23.8.
Claims
| id | claim | strength | support |
|---|---|---|---|
| C1 | Late-training SGD displacement is sub-diffusive, R ∝ t^α with α < ½ | moderate (experiment) | Table 1, Figs. 1, 14–20; fits of a power law to displacement |
| C2 | α = d_s/(2λ) with d_s ≤ λ | weak | d_s is defined by fitting this relation (Eq. 17), so the fit cannot test it; Lemma 3.4’s proof is heuristic |
| C3 | Within a region of constant D_ξ the steady state is a tempered posterior | strong for the algebra, conditional | Lemma 3.3, Corollary 3.2, given that a steady state exists and D_ξ is constant |
| C4 | SGD’s solution distribution is the posterior “tempered” by accessibility | weak | one toy experiment; cross-region weights not derived; ξ fitted; the posterior is SGLD seeded at SGD’s own solutions |
| C5 | The diffusion tensor is well approximated by a scalar for small γ/n | weak, and vacuous as stated | Lemma D.2 bounds ‖D − aI‖ absolutely; relative to ‖D‖ the error does not shrink (my check) |
Method
Fit the LLC by the SGLD estimator of Lau et al. (devinterp, Table 3 settings), record weight displacement from initialisation every 100 steps, regress log R(t) on log t with slope d_s/(2λ) (Eq. 17) to get d_s, and compare d_s with λ. For the posterior test, cluster SGD solutions, sample SGLD from the best of them, and compare cluster masses before and after raising the SGD distribution to the power set by ξ.
Concepts
- Spectral dimension d_s — the exponent of the volume of visited states, V_s(t) ∝ t^(d_s/2); here estimated from displacement given λ.
- Walk dimension — R(t) ∝ t^(1/d_walk); > 2 means sub-diffusion.
- Accessibility constraints — the paper’s phrase (abstract) for the difference between where SGD’s steady state puts mass and where the posterior does, “since SGD simply cannot reasonably reach those areas” (p. 7). It is not given a formula beyond D_ξ.
- Characteristic length ξ — the homogenisation scale at which D_ξ is defined (App. E); in the experiment, a fitted parameter.
Connections
- Mandt et al. (LIT-1899, LIT-1898) is the non-degenerate case this paper sets out to generalise (pp. 1–2).
- Chen et al. (LIT-1894) is cited among the SLT works it builds on.
- Wang et al.’s refined LLC (LIT-1535) is cited for stage boundaries (App. F). Lau et al.’s LLC estimator, used throughout, is not held here.
- Kunin et al.’s anomalous diffusion at the end of training (LIT-1011, THEORY-209) reports anomalous drift of converged networks too, but super-diffusive: squared displacement ∝ t^c with c from about 1.07 to 1.62, explained by damping in a regular quadratic model. This paper does not cite it.
- LaMont and Wiggins on thermodynamics and inference is cited (§2.1); it is registered as Deferred from the same exchange.
Bearing on the record
- QUESTION-363. The question reads this paper as “a tempered
posterior that accounts for local accessibility constraints”, the third
candidate in its title. Two corrections.
- The tempering is a temperature, and it maps to sample size, not to training time. Corollary 3.2 says SGD’s local steady state is the posterior at effective sample size 1/D_ξ. That is the same kind of bridge as Mandt et al.’s ε/S against 1/N: a noise scale standing in for n. Nothing in the paper relates training time t to n; time enters only through displacement and the fractional derivative, and the theory is of “late stage” dynamics.
- “Accessibility” is not derived where it would matter. Within a region of constant D_ξ there is no accessibility correction at all: the steady state there is exactly a tempered posterior. Between regions, the paper’s own Fokker–Planck equation with a spatially varying D gives a zero-flux steady state ∝ exp(−∫∇L/D), which depends on D along the paths between regions, not on each region’s own D (THEORY-1258). The relative mass of regions is therefore not given by per-region tempering, and in §4.2 it is matched after fitting ξ. The paper does not give a case of the kind the question asks for, a region dominant in the posterior that SGD cannot reach.
- Phase transitions are outside the theory. The paper ties its steady state to regions where “λ(w) be locally constant … away from phase transitions” (p. 7), and treats LLC jumps in grokking as needing a space-fractional term it does not develop (App. F). So it says nothing about the order of transitions over training.
- QUESTION-039. The sub-diffusion results bear on how fast SGD moves late in training, not on what sets the order of acquisition.
- THEORY-209. Both describe a converged network that keeps moving. Kunin’s model is regular and quadratic and explains super-diffusive drift by overdamped modes, with larger rates lowering the exponent; this paper reports sub-diffusion and attributes it to degeneracy. The settings differ (resumed ImageNet training at the original schedule there, low-rate fine-tuning here), and neither paper addresses the other.
- New: THEORY-1258, Active, and THEORY-1272, Proposed.
- ML practice. Appendix C proposes uses for transfer learning,
schedulers and optimiser evaluation; the LIT keeps
anthology-candidate.
Limitations
- The existence of a steady state is assumed, and the authors say it may fail (§5.1, Prop. D.1).
- The scalar diffusion coefficient is not justified. Lemma D.2 shows ‖D − aI‖ is small in absolute terms when D is small, which any small matrix satisfies; relative to ‖D‖ the error does not shrink.
- d_s is defined by fitting the relation it is meant to test (Eq. 17). The check of Lemma 3.4 (Fig. 2a, Table 1) compares a fitted d_s with λ; d_s ≤ λ holds wherever the fitted exponent is below ½, which is what sub-diffusion means.
- The posterior test is one toy problem, with a fitted ξ, and an SGLD “posterior” seeded at SGD’s own lowest-loss, lowest-LLC solutions and run at step 10⁻⁵ for 1000 draws, which is unlikely to mix between clusters.
- Negative spectral dimensions appear in Table 8, which the authors attribute to early super-diffusion and keep “for transparency”.
- The vision rows of Table 1 have α ≈ 0.001–0.004, displacement that barely moves, so their r² ≈ 1 is weak evidence.
- Theorem 3.1 is the Alexander–Orbach relation carried over by analogy; its conditions are not checked for loss landscapes.
Open questions
- What does the paper’s own Fokker–Planck equation give for the relative mass of two regions with different D, and does it match SGD’s cluster frequencies without fitting ξ?
- Is the late-training sub-diffusion a property of degeneracy, as claimed, or of a small learning rate near a minimum, as Kunin et al.’s rate-dependent drift would suggest? A sweep crossing both regimes in one architecture would separate them.