Explanation statuses — Whether the record believes the ACCOUNT, which is not what the same word means on a practice. An explanation can be Rejected while the thing it explained goes on working exactly as well as it ever did (ADR-031).
Disbelieved — tested and failed, or contradicted by later work — and the thing it explained may still work perfectly well.
6 of 129 THEORY documents. Back to the full index.
| # | Title | Summary | Status |
|---|---|---|---|
| THEORY-001 | Batch normalization works by reducing internal covariate shift | Ioffe and Szegedy (2015), LIT-002 — the explanation batch normalization was named after and introduced with: that training is slowed by each layer’s input distribution shifting as the layers below it update, and that holding those distributions steady is what buys the speed. Refuted in 2018; the technique was not. | Rejected — the experiment that would have confirmed it was eventually run, and the benefit survived the shift being put back |
| THEORY-013 | The SGD noise scale g = eps*N/B sets generalization, so the optimal batch size grows with the training set | Smith and Le (2017), LIT-305, read SGD as a stochastic differential equation with noise scale g = eps*N/B, argued that minibatch noise is what drives the optimizer toward minima that generalize, and derived that the optimal batch size is proportional to the learning rate and to the training set size. Filed Rejected and kept: LIT-058 swept 35 workloads thirteen months later, found no evidence that larger batches degrade out-of-sample performance once the metaparameters are retuned, and reviewed this paper by name as one that did not retune. The phenomenon the account explains is, on the larger measurement, mostly an artefact of holding the learning rate fixed. | Rejected |
| THEORY-017 | An influence-function estimate predicts the effect of removing a training point and retraining | Koh and Liang (2017), LIT-403 — the account the method was derived from and named for: the estimate approximates leave-one-out retraining, so a large influence means the model would have been meaningfully different without that point. True for convex models, where it was checked. Corrected for neural networks in 2022; the method was not. | Rejected — correct where it was validated, which was the convex case; for neural networks the three largest terms of the discrepancy turn out not to be approximation error at all |
| THEORY-068 | In-context learning is gradient descent on an implicit model | The identification the field read off LIT-533’s title: that what a transformer does with its context is gradient descent on a model held in its activations. Filed Rejected and kept, on three separate grounds. Beyond one layer the source paper’s own models match GD++ — gradient descent on data transformed by I − γXXᵀ — not gradient descent. LIT-535 matches layers to steps and finds a linear correspondence with Iterative Newton and an exponential one with GD, and a conditioning regime GD cannot reach. LIT-536 proves an equivalent algorithm must share ICL’s order sensitivity, which GD has none of. What survives is THEORY-067, the frame without the name. | Rejected |
| THEORY-079 | What makes a diffusion model generative is iterated restoration of any degradation, not Gaussian noise | Bansal et al. (2022), LIT-553 — the account cold diffusion was named for: the train-to-restore, sample-by-restore-and-redegrade procedure generates whatever the degradation is, so noise is incidental. Its own Table 5 has noiseless blur generation at FID 97.00 on CelebA against 23.11 with noise, and Warm Diffusion’s sweep degrades from 1.85 to 11.97 as noise is taken out. The procedure survives. The claim that noise is not what matters does not. | Rejected — its own paper’s unconditional numbers contradict it, and a blur-to-noise sweep shows generation degrading sharply as noise is removed |
| THEORY-082 | Causal tracing shows where a language model stores a fact, and that is where the fact should be edited | Meng et al. (2022), LIT-578 — tracing puts factual recall in mid-layer MLPs at the last subject token, and ROME and MEMIT edit there on that basis. Hase et al. (2023), LIT-574, test the link per fact on GPT-J. Tracing peaks are spread across layers, while edits succeed at layer 6 for almost everything. The edit layer explains 94.7% of the variance in success and tracing 0.1% more. The editors work. The reason given for where they edit does not hold. | Rejected — per-fact causal-tracing localization explains 0.1% of the variance in edit success once the edit layer is accounted for, and edits succeed at layers tracing does not point to |