Topics — The twenty-eight kinds of claim this anthology files against — the primary axis of both the practice registry and the reading list, and the same twenty-eight for each (ADR-026). The filing rule is the point: take a domain topic when the claim is ABOUT the domain; a claim merely discovered in one still takes its kind. The axis organizes the record, it does not bound it: content the list cannot place is evidence the list is short (ADR-059).
Signal structure — what the data itself is like, independent of any model — the statistics and structure of language, images and other signals that methods exploit or must respect, such as frequency spectra, heavy-tailed and Zipfian distributions, and what counts as a unit.
10 of 449 SOTA documents. Back to the full index.
| # | Title | Summary | Status |
|---|---|---|---|
| SOTA-172 v2 | Build synthetic pretraining data by editing human text at the token level, not by generating from scratch | Zhu et al. (2024), LIT-205 — pretraining across proportions of synthetic data shows a negative correlation between the proportion and performance, caused by distributional shift and over-concentration of n-gram features. The remedy inverts the usual move: edit human data at the token level, so the result stays anchored to a real distribution and the test error is provably bounded. | Proposed |
| SOTA-187 v5 | Train the generative model in a learned compressed latent, not at full resolution | Active | |
| SOTA-238 v4 | Set domain weights with a small proxy model under group DRO on excess loss, then transfer them | Xie et al. (2023), LIT-391 — ARXIV-2305.10429. Train a small proxy under group DRO to produce domain weights, then resample and train the real model with them. Optimise worst-case EXCESS loss against a reference model, not worst-case loss — the naive form upweights whichever domain is noisiest, because every domain has a different irreducible entropy. | Active |
| SOTA-243 v2 | Discard the easy examples when data is abundant and the hard ones when it is scarce | Sorscher et al. (2022), LIT-399 — power-law scaling in dataset size is evidence of redundancy, and given a difficulty ranking it can be beaten toward exponential. But which end of the ranking to discard flips with how much data you started with, and the fraction to discard has to grow with the corpus. | Proposed |
| SOTA-263 v4 | Shift the timestep schedule when the resolution changes, because more pixels need more noise | Hoogeboom et al. (2023), LIT-660, and Chen (2023), LIT-659. A timestep is not a fixed amount of corruption. Destroying the signal in an image with more pixels takes more noise, so a schedule set at one resolution under-corrupts at a higher one. Shifting log-SNR down with resolution takes ImageNet 256 FID from 7.65 to 3.76 in pixel space. The direction is measured. The magnitude is tuned in every source. | Active |
| SOTA-267 v3 | Warm up on a subset of the entities before training on all of them, rather than holding one distribution fixed throughout | Zucchet et al. (2025), LIT-450 — the paper argues two quantities move in opposite directions: plateau length is governed by how often the most common entities appear, acquisition speed after the plateau by how often the least common ones do. So concentration buys an early exit from the plateau and costs the tail, and a warm-up that trains on a subset of the entities first, then on all of them, beat the best fixed distribution tried, in single-seed runs on synthetic biographies. | Proposed |
| SOTA-274 v2 | Before adopting a spectral optimizer, measure the stable rank of each block’s incoming activations against its gradient’s nuclear rank | Davis and Drusvyatskiy (2025), LIT-457 — the spectral step’s one-step guarantee beats the Euclidean one by the ratio of gradient nuclear rank to incoming activation stable rank. Both are cheap to compute from a run you are already doing, and the paper never uses them to route. | Proposed |
| SOTA-373 v3 | Set the masking ratio by the signal’s redundancy, not by the ratio that worked on text | Active | |
| SOTA-374 v4 | When learning embeddings from co-occurrence, subsample very frequent tokens, discarding each occurrence with probability 1 − √(t/f) with t around 10⁻⁵ | Mikolov et al. (2013), LIT-603 — in a Zipfian corpus, the few most frequent words co-occur with everything and dominate the updates while teaching little. Discard each occurrence of word w with probability 1 − √(t/f(w)). This cut skip-gram training time by 2–3× and raised accuracy, only slightly for words (59 → 60% at NEG-5) but a lot for phrases (27 → 42% at NEG-15, 19 → 47% with hierarchical softmax). | Active |
| SOTA-382 v3 | Subsample frequent tokens: it is faster and it improves the rare ones | Superseded — by SOTA-374; the same recommendation from the same paper and section; SOTA-374 states the discard rule and carries the independent ablation |