Source
- Nanda et al. (2023), LIT-345, §5.2, App. C.2, D.1, as read in NOTE-283.
- Zucchet et al. (2025), LIT-371, §§2.1–2.2, App. D, as read in NOTE-319.
- Saxe et al. (2018), LIT-372, §§2, 4, App. C, E, I, as read in NOTE-320.
- Prakash & Martin (2025), LIT-369, Figs. 1, 4–6, Tables 1–2, as read in NOTE-317.
- Zhou et al. (2025), LIT-370, Figs. 3–6, as read in NOTE-318.
What was actually shown
Each work divides training into phases. None of them compares its phases with the others’.
| Work | Phases, and the quantity that defines them | What the later phase does |
|---|---|---|
| LIT-345 | Memorisation → circuit formation → cleanup; restricted and excluded loss on a known Fourier algorithm, 10³–10⁴ epochs | Simplifies the weights. The weight norm falls and Fourier sparsity rises, and test loss drops |
| LIT-371 | Marginals → plateau → knowledge; attribute loss against an exact entropy baseline | Acquires information. The model comes to tell individuals apart and stores their attributes |
| LIT-369 | Pre-grokking → grokking → anti-grokking; train and test accuracy, 10⁵–10⁷ steps | Loses generalisation. Test accuracy halves, and the weight spectrum concentrates (α < 2, outlier eigenvalues) |
| LIT-370 | Chaos → cone; basin sensitivity and eNTK cosine distance, 10²–10³ iterations | Confines the dynamics. The basin is fixed, and the kernel moves only within a narrow angle |
| LIT-372 | Drift → diffusion in gradient SNR; rise then fall of binned I(X;T), tanh only | Saturates tanh units, which binning reads as lower entropy. The SNR drop is generic near any minimum |
A shared feature, in two works only. Both works with a mechanistic intervention find a circuit forming while the loss is flat. In LIT-345 it forms while train and test loss stay flat. Ablating its frequencies, or keeping only them, confirms the circuit. In LIT-371 it forms during the plateau, and patching in post-plateau attention patterns removes the plateau. In LIT-345 the circuit forms after memorisation and generalisation follows the cleanup. In LIT-371 the circuit forms before the item-specific learning it enables (NOTE-319). LIT-370’s early settling of the kernel is of the same shape, but it is measured in a different quantity with no intervention on structure. LIT-369 and LIT-372 measure no circuit.
No later phase is IB compression. Of the five, only LIT-372 measures I(X;T). There its fall occurs only for double-saturating units under an imposed binning, and full-batch GD shows it as SGD does (THEORY-035). The other four measure no information quantity. In LIT-371 the later phase must raise the information the recall position carries about the named individual. That is the reader’s inference in NOTE-319, not a measurement. The later phases of LIT-345 and LIT-369 are “simplifications” of the weights, and they point in opposite directions. One comes with generalisation, the other with its loss, under WD = 0 (NOTE-317).
What this does not say
- That training has no general phase structure. The five works use different tasks, architectures, optimisers, regularisers and time scales. The differences here may belong to those settings rather than to training. The claim is that the record’s evidence shows no shared later phase, not that none exists.
- That the five transitions are the same event. They range from 10² iterations (LIT-370) to 10⁶ steps (LIT-369). No work measures another’s quantity, so whether any two coincide is open.
- That the circuit-under-a-flat-loss pattern is general. It rests on one synthetic algorithmic task and one synthetic factual-recall task. Both are small transformers, and the phase boundaries in LIT-345 are drawn by eye.
- That LIT-369’s diagnostics hold. Its own weight-decay control shows α < 2 with traps and no collapse. It is cited here only for the accuracy phases and the direction of the spectral change.
- That no network discards input information. LIT-372 §5 finds task-irrelevant information discarded during fitting, not in a later phase.
Connections
- THEORY-035 rejects the IB account of a later compression phase. This document generalises its negative finding across the record’s other phase sequences, without depending on it. Neither relation is declared, because this claim is about the phase literature, not about the mechanism that document rejects.
- LIT-324 (NOTE-298) is the origin of the drift → diffusion and fitting → compression picture. LIT-341 (NOTE-287) is the grokking phenomenon behind LIT-345 and LIT-369. LIT-348 names the noise scale that the gradient-SNR transition precedes.
Note of 2026-10-26: simplification measured per head
The refined-LLC study (NOTE-1188), read on 2026-10-26 for T24 of the exchange “Experimental Infrastructure Review” (QUESTION-275), tracks the local learning coefficient of each attention head of a two-layer attention-only language model over training. Its appendices C.2–C.3 report stages in which a head’s coefficient falls, which the authors read as “periods of refinement” following periods of rapid specialization and liken to critical periods and synaptic pruning. That is a measured, per-component instance of this account’s first kind of later phase, a network simplifying, and it sits in the same run beside heads that grow (THEORY-975). It is one more phase sequence in one more quantity, which the promote_when says cannot settle the account. The account is unchanged.