Source
- McCandlish, Kaplan, Amodei & the OpenAI Dota Team (2018), LIT-348, §§2–3, Table 1 and Apps. A, C and D, as read in NOTE-293.
- Shwartz-Ziv & Tishby (2017), LIT-324, §3.5, the drift/diffusion SNR phases LIT-348 cites, as read in NOTE-298.
What was actually shown
The mechanism (derivation), now THEORY-735. A minibatch gradient is unbiased for G, with covariance Σ/B. Put it into a second-order step L(θ − εV) and choose the optimal ε. The expected improvement is then ΔL_max/(1 + B_noise/B), with B_noise = tr(HΣ)/GᵀHG (eqs. 2.4–2.8). Below B_noise, examples in a batch substitute almost one-for-one for steps. Above it they are mostly wasted, and at B = B_noise speed is half the maximum. Summed over a run at fixed B, this gives (S/S_min − 1)(E/E_min − 1) = 1, whose corner defines B_crit = E_min/S_min (eqs. 2.11–2.12). NOTE-293 re-derived the hyperbola, and it holds even when the noise scale varies over the run.
The noise-to-signal reading is the paper’s own. For B_simple = tr(Σ)/|G|², E|G_est − G|²/|G|² = B_simple/B (eq. 2.10). So B_simple is the batch at which the batch gradient’s noise equals its signal. The paper calls it “essentially a measure of the signal-to-noise ratio”.
The evidence (measurement). Across eight tasks B_crit spans about six orders of magnitude (Table 1). The run-averaged B_simple lies within about 10× of it on most tasks. The SVHN autoencoder (40 vs 2) and VAE (200 vs 10) miss by about 20×, and the paper says so. Dota 5v5 has no measured front. The noise scale rises by an order of magnitude or more during training. It scales as 1/(ε/B): lowering the temperature 16× raised B_simple about 16× (App. C, Fig. 15).
A link across two readings. LIT-324 holds the batch fixed and watches per-layer gradient SNR fall from drift to diffusion at about 350 epochs. In LIT-348’s terms, that is plausibly where the rising B_simple overtakes the batch in use. This reading is NOTE-298’s and NOTE-293’s. Neither paper states it, and no one has tested it.
What this does not say
- That the model is derived from first principles. The authors call the derivation one with “many unfounded assumptions” (§2.2): a local quadratic, a greedy optimal step, and H ∝ I for B_simple. Its justification is the fit.
- That B_crit is predicted rather than fitted. The functional form is derived, but S_min and E_min are fitted per goal (App. A.3). B_simple/B_crit varies about 10× across tasks, and the paper does not explain why (§5).
- That the noise scale is a property of the task alone. It depends strongly on learning rate through the temperature. The claim that model size acts only through the loss reached rests on single-layer LSTMs at three widths, and the paper states it as a conjecture (NOTE-293, C5).
- Anything about generalisation. The model concerns training loss only (§2.4).
- That the knee is a channel-capacity effect. Per-step progress saturates to a finite ceiling. A Shannon–Hartley log(1 + SNR) with SNR ∝ B grows without bound. These are different laws, and the paper contains no entropy, channel or capacity (NOTE-293; curation 2026-09-29, 230637).
- That the temperature ε/B is physical. It is an SGD analogy and has no connection to Landauer’s k_BT.
Connections
- LIT-347 and LIT-356 own the “learner as a channel” framing that the prior-art map’s row 12 attaches to this knee. Whether a per-step information term rises with B/B_simple is an open question in NOTE-306.
- The IB-compression reading of LIT-324’s phases is not relied on here. See THEORY-035, which rejects it.