THEORY-072: Training and test loss disagree as functions of weight norm, and shrinking the norm through the gap is what takes so long

Source

Liu, Michaud and Tegmark (2022), LIT-540 — ARXIV-2210.01117. The LU mechanism is this paper’s: it defines the reduced landscape, derives the γ-dependence and measures it in the teacher–student model, and uses the picture to induce grokking on MNIST, IMDb and QM9.

The account

Write any loss as a function of the weight norm w = ‖w‖₂ and an angular direction, and define the reduced loss by minimizing over directions at fixed norm. Then:

• Reduced training loss is “L”-shaped. It falls as the norm grows and then stays near zero, because at any sufficiently large norm an overparameterized network can fit a small training set.
• Reduced test loss is “U”-shaped, with a minimum at a critical norm w_c — the “Goldilocks zone”.

Above w_c the two disagree: training loss says done, test loss says no. A model initialized there fits fast and sits. Only weight decay — or implicit regularization — moves the norm back toward w_c, and that walk is slow. Grokking is the walk.

With weight decay γ, the reduced training landscape becomes l̃_train(α) + γα²C², which gives the quantitative signature: time to fit is independent of γ, time to generalize is inversely proportional to it.

What supports it

• The γ-dependence is measured and matches, in the teacher–student model at large initialization α = 2.0.
• Small initializations (α = 0.5) generalize fast at every γ, as the picture requires — there is no gap to cross.
• At α = 2.0: γ = 0 never generalizes, small γ groks, large γ generalizes quickly.
• The picture predicts where to look to make grokking happen, and the predictions worked on MNIST, IMDb and QM9. That the recipe was derived from the mechanism is the account’s best argument.
• LIT-085 independently reports the weight norm rising during memorization and then dropping below its initial value as the model generalizes.

Why it is Proposed

A landscape is not a trajectory. The reduced loss minimizes over angular directions at each norm. That is a well-defined object; it is not the path the optimizer takes, and the inference from the shape of one to the timing of the other is the step carrying the argument.

And there is a counterexample. LIT-537 groks on modular arithmetic with a two-layer MLP, no weight decay, and a weight norm that rises through the transition. If there is no regularization there is no walk down, and if the norm increases the model is moving away from w_c rather than toward it. The account cannot be the general mechanism.

That run is the paper’s evidence for a rival account of the same delay, THEORY-070: grokking as the late end of lazy training, controlled by output scale and kernel alignment rather than weight norm. The two cannot both be the general mechanism, and neither has absorbed the other: the record holds no lazy-to-rich account of the γ-dependence measured above, and this account has nothing to say about a run whose norm rises.

It is not Rejected, and the distinction matters. The counterexample shows the LU walk is not necessary for grokking. It does not touch the runs where the norm does come down — which LIT-085 observed independently — nor the γ-dependence, nor the induce-and-eliminate experiments, which no one disputes and which are filed under THEORY-069 rather than here.