Shreyas Gokhale (2023), arXiv:2307.09550.
Key takeaways
From the reading, NOTE-807.
- A vocabulary, not a result. Training is a trajectory over a graph of “heuristic models”, the algorithms a network’s weights implement (conjecture C1), with fallibility F = Σ_x D_KL(P_GT ‖ P_H) over all inputs as the landscape. Nothing is measured or proved.
- Thresholds as connectivity. Below a critical data size the graph disconnects the memorising model from the generalising one, so grokking cannot happen however long training runs; with an assumed rate λ_g ∝ (D − D_c), grokking time is 1/(D − D_c). Emergence with scale is a node appearing as N grows. Both are called percolation; no percolation model is specified.
- Scaling laws as waiting times. Bouchaud’s trap model with exponentially distributed barriers, plus F ∝ 1/N_b, gives F(S) = (S₀/S)^μ, F(D) = (D₀/D)^μ and F(N) = (N₀/N)^(μ/(d_f·α)), on a graph assumed connected. Kaplan et al.’s data exponent gives μ ≈ 0.1.
- Two kinds of event. Thresholds and power laws come from different parts of the picture: connectivity for the first, barrier statistics for the second.
Standing in the record
Filed 2026-10-12 from the owner’s papers-feed for QUESTION-039, as a
candidate for the “one kind of event” answer. Read in full on 2026-10-14
in NOTE-807, which finds it is not that candidate: it puts
grokking, emergence and scaling into one vocabulary but treats
thresholds and power laws as different events. It has no experiment and
no proof, so no THEORY is filed from it. For grokking’s critical data
size the record holds a concrete, measured mechanism in THEORY-181. It
keeps the anthology-candidate flag: it is about training dynamics in
machine learning, which an anthology topic could hold.