Source
Chen, Lau, Mendel, Wei and Murfet (2023), LIT-1894, §§4–5 and Appendices C, D, F and K of arXiv 2310.06301v1; read in NOTE-1462.
What was actually shown
The setting. TMS with r = 2 hidden and c = 6 input dimensions in the high-sparsity limit (LIT-323). Phases are regions dominated by k-gon critical points. The local free energy is F_n(W_α) ≈ nL_α + λ_α log n + c_α, and a Bayesian transition α → β happens where the two cross (Eq. 12).
The posterior. 5-gon (λ = 7, L = 0.06874) to 6-gon (λ = 8.5, L = 0.04819) at n_cr ≈ 601 by the formula, between 600 and 700 by NUTS sampling (Fig. 2). I recomputed both losses from the model by quadrature and n_cr from Eq. 12.
Training. SGD on n = 1000 samples (batch 20, rate 0.005, 4500 epochs, from a perturbed 4-gon, 30 seeds; and 400 runs from posterior samples at n = 100) plateaus at k-gon critical points and makes the transitions of Eq. 10, among them 4⁺⁺⁻ → 4⁺ → 5 → 5⁺. “the Bayesian phase transition 5 → 6 has not been observed as a dynamical transition” (§5.1).
The hypothesis and its scope. “The dynamical transitions α → β encountered in neural network training have Bayesian antecedents” (§5.1). The paper adds: “There is no necessary relation between these two kinds of transitions”, and explains the missing 5 → 6 by distance or barriers. It does not claim that every Bayesian transition occurs in training, or that the two orders coincide. App. C.4 gives the condition under which they could: with constants ignored, successive transitions occur at increasing n only if the slope ΔL/Δλ rises along the chain.
My computation from their numbers. With Table C.2’s constants, Table K.1’s λ̂ and the losses above, the antecedents fall at n ≈ 190 (4⁺⁺⁻ → 4⁺), 370 (4⁺ → 5) and 2,500 (5 → 5⁺; 1,600 with the theoretical λ). The slopes, −0.029, −0.018, −0.003, rise. Along this chain the order in training time is the order in sample size. At n = 1000 the free energies are about 113 for the 6-gon, 121 for the 5-gon and 124 for the 5⁺-gon: the posterior favours the 6-gon, and of the two phases SGD ends in it prefers the one SGD left.
What could have failed. The antecedents’ critical sizes could have fallen along the chain, and SGD could have reached the 6-gon. Neither happened in this model at this setting.
What this does not say
- That training time maps onto sample size. A monotone map along one chain would still have to send the end of training at n = 1000 to a sample size above 1,600 for the 5⁺ step, and could not send it to the 6-gon that the posterior at 1000 favours. No single t ↦ n map makes training’s endpoint the posterior’s state at its own n.
- That the 6-gon is inaccessible. One learning rate, one batch size and chosen initialisations; “it may occur, just with low probability per SGD step” (§5.1). Inaccessible and not reached in 4500 epochs are not yet told apart.
- That the critical sizes are exact. The free energy formula is asymptotic and used at n of a few hundred, the constants are heuristic (App. C.1), and the λ̂ for 4-gon variants have standard deviations near 1 (Table K.1). The lowest-n antecedents are inconclusive by the authors’ account (App. D).
- That the pattern generalises. One toy model with exact learning coefficients for a few critical points.