Chen, Z., Lau, E., Mendel, J., Wei, S., & Murfet, D. (2023). Dynamical versus Bayesian phase transitions in a toy model of superposition. arXiv:2310.06301.
Read in NOTE-1462, with the derivation appendices skimmed.
Why this is here
A3 cited it to separate Bayesian and dynamical transitions. The paper does state that separation: “There is no necessary relation between these two kinds of transitions” (§5.1), and its main Bayesian transition, 5-gon to 6-gon, was never seen in training. What A3 left out is the rest: the two processes share their landmarks, the critical points, and their direction, from high loss and low complexity to low loss and high complexity; and the paper conjectures that training’s transitions have Bayesian antecedents. That conjecture runs one way and does not say the two orders coincide. From the paper’s numbers, the antecedents of one observed chain do fall in training order, while SGD trained on 1000 samples ends where the posterior at 1000 does not (THEORY-1275). Read with LIT-323, the toy model it studies. It bears on QUESTION-363.