Topics — The twenty-eight kinds of claim this anthology files against — the primary axis of both the practice registry and the reading list, and the same twenty-eight for each (ADR-026). The filing rule is the point: take a domain topic when the claim is ABOUT the domain; a claim merely discovered in one still takes its kind. The axis organizes the record, it does not bound it: content the list cannot place is evidence the list is short (ADR-059).
Signal structure — what the data itself is like, independent of any model — the statistics and structure of language, images and other signals that methods exploit or must respect, such as frequency spectra, heavy-tailed and Zipfian distributions, and what counts as a unit.
26 of 737 LIT documents. Back to the full index.
| # | Title | Summary | Status |
|---|---|---|---|
| LIT-062 v3 | High-Resolution Image Synthesis with Latent Diffusion Models | Rombach et al. (2022), ARXIV-2112.10752. Latent space diffusion. | Active |
| LIT-205 v2 | How to Synthesize Text Data without Model Collapse? | Zhu et al. (2024), ARXIV-2412.14689. Pre-training across proportions of synthetic data gives a negative correlation between that proportion and performance, and the statistical signature is distributional shift plus over-concentration of n-gram features. The proposed answer is not better generation but token-level editing of human text into semi-synthetic data, with a proof that the test error is then bounded. | Active |
| LIT-328 v2 | Spectrum Dependent Learning Curves in Kernel Regression and Wide Neural Networks | Bordelon et al. (2020), ARXIV-2002.02561. Learning curves for kernel regression follow a power law whose exponent is set by the decay of the kernel-target spectrum, which connects spectral structure to observed scaling exponents. | Active |
| LIT-409 v2 | Token Erasure as a Footprint of Implicit Vocabulary Items in LLMs | Feucht et al. (2024), ARXIV-2406.20086, EMNLP 2024. The last token of a multi-token word or named entity shows a pronounced “erasure”: probes that recover neighbouring and current token identity elsewhere fail there, in early layers specifically. The paper reads that as the footprint of a mechanism forming lexical items, scores sequences for “lexicality” and reads out an LLM’s implicit vocabulary — the first attempt to do so. Its framing is the sharp part for this record: lexical items are units of meaning that are non-compositional, and the example given is a multi-word expression. | Active |
| LIT-410 v2 | The English phrase-as-lemma construction: When a phrase masquerades as a word, people play along | Goldberg and Shirtz (2025), Language 101(2):291-320. English has a construction — a don't-mess-with-me driver — in which a unit with the internal syntax of a phrase, up to a whole sentence, occupies a slot reserved for a word, and speakers treat it as a word: four preregistered surveys (N = 685 plus 70) find such sentences judged to presuppose more shared common knowledge, and to be wittier and more sarcastic, than normed near-paraphrases. The effect survives restricting to high-frequency phrases and shows no frequency dependence. No model was run, and the finding is about the signal rather than anything trained on it: it sources THEORY-022, the claim that a lexical unit is delimited by construction rather than by frequency — which is what SOTA-007’s vocabulary is fit to. See ADR-045 for the scope call it forced. | Active |
| LIT-450 v3 | How do language models learn facts? Dynamics, curricula and hallucinations | Zucchet et al. (2025), ARXIV-2503.21676. Factual recall is learned in three phases with a long plateau between generic statistics and individual knowledge, and the plateau is the attention recall circuit forming — shown by patching a reference model’s attention patterns in, at which point the plateau disappears. Imbalanced data shortens the plateau, up to a point, and the paper argues it slows acquisition afterwards; a warm-up on a subset of individuals, then all of them, beat the fixed distributions tried (single seed). | Active |
| LIT-457 v2 | When do spectral gradient updates help in deep learning? | Davis and Drusvyatskiy (2025), ARXIV-2512.04299. A spectral update beats a Euclidean one on a block exactly when the gradient’s squared nuclear-to-Frobenius ratio exceeds the stable rank of that block’s incoming activations — a measurable condition, and they show it is satisfied throughout transformer training. | Active |
| LIT-484 v2 | TinyStories: How Small Can Language Models Be and Still Speak Coherent English? | Eldan and Li (2023), ARXIV-2305.07759. Restrict the training corpus to what a three-year-old’s vocabulary can say and a model below 10M parameters — or one with a single transformer block — writes fluent, grammatical, multi-paragraph stories. The finding is about the corpus, not the architecture: what stops small models speaking is the breadth of what they are asked to learn, not the complexity of language. | Active |
| LIT-555 v2 | Warm Diffusion: Recipe for Blur-Noise Mixture Diffusion Models | Hsueh et al. (2025), ARXIV-2511.16904. Degrade with blur and noise together, and predict the blurry image and the missing high frequencies with two heads. Blur lets the model use the dependency of fine detail on coarse structure. Noise keeps intermediate states on the data manifold. Sweeping the blur-to-noise ratio on CIFAR-10 at 35 steps gives FID 1.97 at BNR = 0 (EDM), 1.85 at BNR = 0.5 and 11.97 at BNR = 10. A small gain, and a large loss as the process goes cold. | Active |
| LIT-592 | Latent Dirichlet Allocation | Blei, Ng and Jordan (2003), JMLR 3:993–1022. If the words of a document are exchangeable (bag of words), de Finetti’s theorem says the document is a mixture over a latent parameter. LDA makes that parameter a Dirichlet-distributed vector of topic proportions, one per document, with each topic a distribution over words. Fitted by variational EM, it has lower held-out perplexity than a unigram, a mixture-of-unigrams and a pLSI model on two corpora, and the two baselines overfit badly as the number of topics grows. Its 50 topic proportions stand in for 15,818 word features in Reuters classification. | Active |
| LIT-601 | Masked Autoencoders Are Scalable Vision Learners | He et al. (2021), ARXIV-2111.06377. Mask 75% of the patches, run the encoder on the visible quarter only, and reconstruct pixels with a small decoder you then throw away. 3x faster pretraining, 87.8% with ViT-Huge on ImageNet-1K alone. The 75% ratio, five times BERT’s 15%, is found by a sweep, and the paper explains it with an argument about the signal’s redundancy. | Active |
| LIT-602 v2 | GloVe: Global Vectors for Word Representation | Pennington, Socher and Manning (EMNLP 2014), DOI 10.3115/v1/D14-1162. Meaning shows in ratios of co-occurrence probabilities, not in the probabilities themselves, so word vectors should make differences encode log ratios. That leads to a weighted least-squares fit of wᵢ·w̃ⱼ + bᵢ + b̃ⱼ to log Xᵢⱼ over nonzero counts, with a weight capped at x_max = 100 and exponent 3/4. It scores 75.0% on the word analogy test with 42B tokens, and 71.7% against word2vec skip-gram’s 69.1% on the same 6B tokens. The word2vec side of that comparison ran at default settings. | Active |
| LIT-603 v3 | Distributed Representations of Words and Phrases and their Compositionality | Mikolov, Sutskever, Chen, Corrado and Dean (NIPS 2013), ARXIV-1310.4546. Makes skip-gram fast and better. Negative sampling replaces the softmax with logistic regression against k noise words drawn from the unigram distribution raised to the 3/4 power. Frequent words are subsampled, with discard probability 1 − √(t/f) at t ≈ 10⁻⁵, which runs 2–10× faster and helps rare words. High-PMI bigrams become single tokens. Vector sums compose meanings, for example Russian + river ≈ Volga River. | Active |
| LIT-607 v3 | Improving Distributional Similarity with Lessons Learned from Word Embeddings | Levy, Goldberg and Dagan (TACL 2015). Takes the design choices bundled into word2vec and GloVe (dynamic windows, subsampling, shifted PMI, context-distribution smoothing, adding context vectors, eigenvalue weighting), transfers them to count-based PPMI and SVD, and compares all four methods under the same tuning: 672 representations, 8 similarity and analogy datasets, and cross-validated selection. Once tuned alike, no method wins consistently. A single hyperparameter often matters more than the choice of method. SGNS beats GloVe on every task, reversing GloVe’s headline comparison. Smoothing the context distribution by 0.75 is the one setting that helps everywhere. | Active |
| LIT-609 v2 | Distributed Representations of Words and Phrases and their Compositionality | Mikolov et al. (2013), ARXIV-1310.4546. Where negative sampling comes from — and it is NCE with the consistency deliberately dropped, kept only because the vectors stay good. Also: the number of negatives that helps falls as the dataset grows, which is the opposite of what the vision contrastive line later assumed. | Superseded — by LIT-603; a duplicate of LIT-603, filed concurrently by |
| LIT-610 v2 | Efficient Estimation of Word Representations in Vector Space | Mikolov et al. (2013), ARXIV-1301.3781. CBOW and Skip-gram: drop the non-linear hidden layer that made neural language models expensive, and word vectors become cheap enough to train on 1.6 billion words in under a day. The linear-analogy finding is here, and so is the test set built to stop it being an anecdote. | Superseded — by LIT-604; a duplicate of LIT-604, filed concurrently by |
| LIT-612 | Neural Word Embedding as Implicit Matrix Factorization | Levy and Goldberg (NeurIPS 2014). With enough dimensions, skip-gram with negative sampling reaches its optimum when w·c = PMI(w, c) − log k, so it implicitly factorizes the word-context PMI matrix shifted by log of the number of negatives. NCE factorizes log P(w|c) − log k in the same way. The factorization is weighted: frequent pairs cost more to get wrong. Shifted PPMI and SVD over it match SGNS on word similarity. SGNS stays ahead on syntactic analogies, which the authors attribute to the weighting. | Active |
| LIT-613 | A Latent Variable Model Approach to PMI-based Word Embeddings | Arora, Li, Liang, Ma and Risteski (TACL 2016). A generative model of text: a discourse vector drifts slowly on a random walk, and each word is emitted with probability ∝ exp(⟨c, v_w⟩). If word vectors are isotropic, the partition function is nearly constant, and integrating out the discourse gives log p(w,w’) ≈ ‖v_w + v_w’‖²/2d − 2 log Z, so PMI ≈ ⟨v_w, v_w’⟩/d in low dimensions. Isotropy also explains why analogies survive large fitting error: solving for v_a − v_b is a regression that denoises by √(d/n). The squared norms of the fitted vectors correlate 0.75 with log frequency, and the partition function concentrates within about ±10%. | Active |
| LIT-654 | Muon Outperforms Adam in Tail-End Associative Memory Learning | Wang et al. (2025), ARXIV-2509.26030. Ninety-four documents in this record name Muon and seventeen practices recommend or condition on it; five theories mention it and none says why it works. This one ablates Muon block by block: applying it to the value-output weights and the FFN alone almost recovers the full-Muon trajectory, and applying it to query-key contributes little. The account is that Muon’s update matches the outer-product structure of associative memory, giving isotropic spectra and better learning of rare classes. | Active |
| LIT-659 | On the Importance of Noise Scheduling for Diffusion Models | Chen, Google Brain (2023), ARXIV-2301.10972. Higher-resolution images are less corrupted by the same per-pixel noise, so the best schedule moves with resolution. It sweeps schedule functions (Table 3) and an input scaling factor b, with x_t = sqrt(γ)·b·x0 + sqrt(1−γ)·ε (Table 4), on ImageNet 64, 128 and 256 with RIN. The best b falls as resolution rises. At 256 with a 1−t schedule, b = 1 gives FID 7.21 and b = 0.4 gives 3.52. This is concurrent with simple diffusion and is the second origin of the resolution-dependent shift. | Active |
| LIT-660 | simple diffusion: End-to-end diffusion for high resolution images | Hoogeboom, Heek and Salimans, Google Brain (2023), ARXIV-2301.11093. Trains single-stage pixel-space diffusion up to 512x512 with four changes. The first is to define the noise schedule at a reference resolution and shift its log-SNR by 2·log(ref/d) at resolution d. Table 2 is the controlled measurement. On ImageNet 256 the unshifted cosine schedule gives FID 7.65 train / 6.87 eval, and the schedule shifted to 32 gives 3.76 / 3.71. This is the origin, with the concurrent Chen (2023), of the resolution-dependent schedule shift SD3 later derives again. | Active |
| LIT-672 | Should You Mask 15% in Masked Language Modeling? | Wettig, Gao, Zhong and Chen (2022), ARXIV-2202.08005. The sweep BERT never ran, and the answer the record has been openly missing: 15% is not universally optimal, and the optimum tracks model size — 40% at 354M, 20% at 124M, 15% at 51M. It also decomposes the masking rate into a corruption rate and a prediction rate whose effects are antagonistic, and uses that to find BERT’s 80-10-10 substitution rule worse than plain [MASK]. | Active |
| LIT-689 | Common Diffusion Noise Schedules and Sample Steps are Flawed | Lin, Liu, Li and Yang (WACV 2024), ARXIV-2305.08891. Common schedules leave signal at the last timestep. Stable Diffusion’s x_T is 0.068·x₀ + 0.998·ε. The model learns to keep the leaked per-channel mean, and pure-noise inference then pins samples to “medium brightness”. Many samplers also never start at t = T. It proposes four fixes: zero terminal SNR, v-prediction, sampling from the last timestep, and a guidance rescale. The only numeric result applies all four at once (COCO FID 22.96 → 21.66). Every single-fix ablation is qualitative, on one to three prompts. It credits the trailing discretization to DPM-Solver, and it never calls the spacing fix “slight”. | Active |
| LIT-695 | Language Design as Information Renormalization | Gallego et al. (2022), ARXIV-1708.01525. The paper models Chomsky’s MERGE as a coarse-graining tensor, so that sentence probabilities form mostly loop-free stochastic tensor networks. It argues from this that tree-like syntax gives polynomially decaying (long-range) correlations. It then concludes that tree-structured networks (it names deep CNNs, as Tree Tensor Networks) fit language, while MPS-like models (RNNs, HMMs) with exponentially decaying correlations do not. | Deferred — seeded from the abstract and a skim on 2026-09-25; not read in full |
| LIT-751 | ”Principal Components” Enable A New Language of Images | Wen, Zhao et al. (2025), ARXIV-2503.08685 — Semanticist. A ViT encoder emits a causal 1D sequence of continuous tokens that conditions a DiT diffusion decoder. During training, every token after a uniformly sampled index is replaced by a learned null token (“nested CFG”), so earlier tokens carry more. The result is coarse-to-fine and PCA-like, and any prefix decodes. ImageNet-256 rFID 0.78 with 256×16 floats against ImageFolder’s 0.80 at 286×32. An autoregressive model trained on only the first 32 tokens reaches gFID 2.57. The diffusion decoder is argued to keep image spectrum out of the tokens, but that is shown against other tokenizers, not ablated. | Active |
| LIT-767 | Understanding Deep Learning via Notions of Rank | Razin (2024), ARXIV-2408.02111 — PhD thesis. Gradient descent’s implicit regularization is better read as rank minimization than norm minimization: on a 2×2 matrix completion problem, a depth L ≥ 2 linear network trained by gradient flow drives every norm and quasi-norm to infinity, with probability at least 0.5 over arbitrarily small initialization, while rank falls to its minimum. The same bias toward low (tensor, hierarchical-tensor) rank is proved for polynomial-activation networks equivalent to tensor factorizations; it predicts convolutional networks’ locality, which a regularizer derived from it partly undoes on long-range tasks. | Active |