Paper statuses — A paper’s standing in the anthology: whether it is worth reading. NOT whether its findings are true. Adam is Active here because it is worth reading; whether any particular thing it recommends is still what you should do is the SOTA scheme’s business, and the two can disagree in either direction (ADR-002).
Superseded — later work covers this better, and the note names it.
6 of 737 LIT documents. Back to the full index.
| # | Title | Summary | Status |
|---|---|---|---|
| LIT-031 v2 | Pruned Neural Networks are Surprisingly Modular | Filan et al. (2020), ARXIV-2003.04881. Analysis of neural network modularity. | Superseded — by LIT-041, ARXIV-2110.08058; by later work on clusterability and mechanistic interpretability |
| LIT-042 v2 | SGDR: Stochastic Gradient Descent with Warm Restarts | Loshchilov and Hutter (2016), ARXIV-1608.03983. Warm restarts: anneal the learning rate on a cosine to a floor, then jump it back up and repeat, with each cycle longer than the last. | Superseded — by LIT-035; single-cycle cosine decay proved better at scale |
| LIT-141 | Hyper-Connections | Zhu et al. (2024), ARXIV-2409.19606. Replace the residual connection with several parallel streams and learnable mixing between them, at expansion rate 2 or more, to escape the seesaw between vanishing gradients and representation collapse; gains on dense and MoE pretraining — and, per its successor, an identity-mapping property lost along the way. | Superseded — by LIT-140; mHC restores the identity mapping that free hyper-connection mixing gave up, and scales where this did not |
| LIT-609 v2 | Distributed Representations of Words and Phrases and their Compositionality | Mikolov et al. (2013), ARXIV-1310.4546. Where negative sampling comes from — and it is NCE with the consistency deliberately dropped, kept only because the vectors stay good. Also: the number of negatives that helps falls as the dataset grows, which is the opposite of what the vision contrastive line later assumed. | Superseded — by LIT-603; a duplicate of LIT-603, filed concurrently by |
| LIT-610 v2 | Efficient Estimation of Word Representations in Vector Space | Mikolov et al. (2013), ARXIV-1301.3781. CBOW and Skip-gram: drop the non-linear hidden layer that made neural language models expensive, and word vectors become cheap enough to train on 1.6 billion words in under a day. The linear-analogy finding is here, and so is the test set built to stop it being an anecdote. | Superseded — by LIT-604; a duplicate of LIT-604, filed concurrently by |
| LIT-720 | Analyzing and Improving the Training Dynamics of Diffusion Models | Karras, Aittala, Lehtinen, Hellsten, Aila and Laine (2023), ARXIV-2312.02696. Post-hoc EMA: keep two power-function weight averages during training, snapshot them periodically, and afterwards reconstruct the average for any decay length by least squares — reconstruction error falling as O(1/n⁴) in the snapshot count. That makes the averaging length a sweep instead of a pre-training guess, and the consequence is the headline number here: with the length fixed at 13%, varying the learning-rate decay over t_ref ∈ [30k, 160k] moves FID by up to 72%; sweeping the length post hoc brings the whole bracket within 10% of the optimum. Also EDM2’s magnitude-preserving architecture, reaching FID 1.91 on ImageNet-512 without guidance. | Superseded — by LIT-714; A duplicate. 2312.02696 was already held as LIT-714, filed hours earlier in #375 and merged while this note was being written; LIT-714 is the fuller reading — the whole ablation ladder, the metric-disagreement finding, and NOTE-370 with the appendices — and ten documents cite it against this note’s four, all of which were its own. Retired rather than deleted so the mistake stays visible and the formerly: alias keeps any stray citation resolving. Nothing here is lost: everything this note said, LIT-714 or SOTA-428 says already, including the learning-rate bracket. |