Topics — The twenty-eight kinds of claim this anthology files against — the primary axis of both the practice registry and the reading list, and the same twenty-eight for each (ADR-026). The filing rule is the point: take a domain topic when the claim is ABOUT the domain; a claim merely discovered in one still takes its kind. The axis organizes the record, it does not bound it: content the list cannot place is evidence the list is short (ADR-059).
Data pipeline — loading, quality assessment and selection, preprocessing, batch preparation.
54 of 449 SOTA documents. Back to the full index.
| # | Title | Summary | Status |
|---|---|---|---|
| SOTA-042 | Memory-map large datasets | Mohan et al. (2020), LIT-050 — ARXIV-2007.06775. | Active |
| SOTA-043 | Use mixed precision during data loading | Mohan et al. (2020), LIT-050 — ARXIV-2007.06775. | Active |
| SOTA-044 | Pin memory for CPU-GPU transfers | Afroz et al. (2025), LIT-737 — ARXIV-2511.14124. Host-to-device copies from pinned memory ran at 24.74 GB/s against 10.16 GB/s from pageable memory on PCIe 4.0; the advice itself is NVIDIA’s, not a paper’s. | Active |
| SOTA-045 | Profile data loading separate from training | Mohan et al. (2020), LIT-050 — ARXIV-2007.06775. | Active |
| SOTA-077 | Use tar archives for dataset storage | Aizman et al. (2020), LIT-053 — ARXIV-2001.01858. | Active |
| SOTA-078 | Buffer size should be 2-3x batch size | Aizman et al. (2020), LIT-053 — ARXIV-2001.01858. | Active |
| SOTA-079 | Pre-fetch next batch during compute | Aizman et al. (2020), LIT-053 — ARXIV-2001.01858. | Active |
| SOTA-080 | Use multiple worker processes (num_workers = 4 * num_gpus) | Aizman et al. (2020), LIT-053 — ARXIV-2001.01858. | Active |
| SOTA-101 v2 | Use perplexity-based filtering for quality assessment | Albalak et al. (2023), LIT-117 — ARXIV-2312.02406. | Rejected — The cited paper performs no filtering — it reweights sampling over domains — and filtering has the opposite failure mode to what it recommends. The string “filter” does not appear in it |
| SOTA-102 v2 | Implement dynamic temperature scaling for mixing | Albalak et al. (2023), LIT-117 — ARXIV-2312.02406. | Superseded — by SOTA-103; Names a real component of ODM’s policy imprecisely rather than something the paper does not do — the two were one practice split in the import, and SOTA-103 now describes the whole method |
| SOTA-103 v2 | Adjust data mixing proportions online from per-domain training loss | Albalak et al. (2023), LIT-117 — ARXIV-2312.02406. | Active |
| SOTA-104 v2 | Monitor domain coverage during training | Albalak et al. (2023), LIT-117 — ARXIV-2312.02406. | Rejected — Names no signal and no response, and is in tension with its own source: under ODM the sampling distribution is supposed to move away from uniform coverage. The string “domain coverage” does not appear in the paper |
| SOTA-124 v5 | Repeat high-quality data freely when its epoch size exceeds the model’s memorization window | Falcon-LLM Team (2026), LIT-119 — the Falcon-H1-Tiny technical blogpost. Roughly 100–500 GT for a 7B model, scaling linearly; the authors call the understanding early. | Proposed |
| SOTA-127 v2 | Filter chain-of-thought traces out of the training data of tiny specialized models | Falcon-LLM Team (2026), LIT-119 — the Falcon-H1-Tiny technical blogpost. Reasoning traces interleaved with tool-calling data sent a 90M model into repetition loops; removing them fixed it at once. | Active |
| SOTA-128 | Train fill-in-the-middle data without masking the loss on non-FIM tokens | Falcon-LLM Team (2026), LIT-119 — the Falcon-H1-Tiny technical blogpost. At 80 GT and 90M, computing the loss on every token beat masking the prefix and suffix; matches what the reference FIM recipes appear to do. | Active |
| SOTA-152 | Pack training documents by best fit instead of concatenating and splitting | Ding et al. (2024), LIT-203 — concatenate-then-split truncates documents that would have fit, and the model then learns to continue text whose beginning it never saw. Treating the grouping as bin packing removes those truncations at the same efficiency, no padding added: +4.7% reading comprehension, +16.8% context following, +9.2% program synthesis, and up to 58.3% less closed-domain hallucination. | Proposed |
| SOTA-164 v2 | Deduplicate the pretraining corpus at both substring and document granularity before training on it | Lee et al. (2021), LIT-202 — a single 61-word sentence appears thousands of times in C4. Deduplicating cuts verbatim emission about tenfold, reaches equal or better accuracy in fewer steps, and removes train-test contamination from standard validation sets. Two granularities, because a repeated boilerplate paragraph is not a duplicated document. | Active |
| SOTA-166 v2 | Set the pretraining data proportions by fitting a mixing law on small runs, not by argument | Ye et al. (2024), LIT-201 — model performance is quantitatively predictable as a function of the mixture proportions, so fitting the law on a sample of mixtures reads off performance on mixtures never trained. Nested with the existing scaling laws, small runs predict a large model. Matched the default mixture trained for 48% more steps. | Proposed |
| SOTA-170 v2 | Choose filtering aggressiveness by the token horizon: at long horizons rephrase what a filter would discard | Su et al. (2024), LIT-178 — aggressive model-based filtering wins at 1T tokens and loses at 15T, because removing 90% of the data optimises mean quality and mean quality is not what binds when the corpus runs out. Ensemble the classifiers, rephrase text synthetically instead of discarding it, and lean less on heuristics. | Active |
| SOTA-171 v2 | Repeat a data-constrained corpus for up to about four epochs; past that, added compute stops paying | Muennighoff et al. (2023), LIT-166 — with constrained data at fixed compute, up to four epochs of repeated data changes the loss negligibly against having that much unique data. Past that point the value of adding compute decays toward zero. 400 training runs, up to 900B tokens and 9B parameters, with a scaling law that prices repeated tokens and excess parameters. | Active |
| SOTA-172 v2 | Build synthetic pretraining data by editing human text at the token level, not by generating from scratch | Zhu et al. (2024), LIT-205 — pretraining across proportions of synthetic data shows a negative correlation between the proportion and performance, caused by distributional shift and over-concentration of n-gram features. The remedy inverts the usual move: edit human data at the token level, so the result stays anchored to a real distribution and the test error is provably bounded. | Proposed |
| SOTA-173 v2 | Augment the objective to make multi-epoch pretraining productive on a fixed corpus | Chen et al. (2026), LIT-175 — standard autoregressive pretraining overfits severely on a fixed corpus, reaching its optimum early and deteriorating continuously. Three orthogonal augmentation families each delay that and lower validation loss; combining them lowers the minimum further. The claim is hundreds of productive epochs on the same data. | Proposed |
| SOTA-174 | Train autoregressive models with fill-in-the-middle by default: it is a data transformation, and it is free | Bavarian et al. (2022), LIT-124 — cut a document into prefix, middle and suffix, move the middle to the end with sentinel tokens, and the model learns to infill. Transforming a large fraction of the training data does not harm left-to-right perplexity or sampling quality across a wide range of scales, so infilling is an added capability rather than a trade. | Active |
| SOTA-175 | Mask whole syntactic units for code fill-in-the-middle, not random character spans | Gong et al. (2025), LIT-126 — standard FIM masks random character spans, which produces training examples that rarely correspond to an edit anyone makes. AST-FIM masks whole syntactic units at scale, so the middle span is a coherent structure. Up to 5 points over random-character FIM at 1B and 8B, and most useful on real editing. | Proposed |
| SOTA-183 | Generate the harmlessness preference labels with the model itself, against a written set of principles | Bai et al. (2022), LIT-082 — ARXIV-2212.08073. Replace the human harmlessness comparisons with model-generated ones: the model critiques and revises its own responses against an explicit list of principles, and the preference model trains on that. | Active |
| SOTA-186 | Bootstrap a large annotation set with the model you are training, staging the automation as it improves | Active | |
| SOTA-194 | When an objective arrives with its own dataset, state the composition before crediting the objective | Active | |
| SOTA-238 v4 | Set domain weights with a small proxy model under group DRO on excess loss, then transfer them | Xie et al. (2023), LIT-391 — ARXIV-2305.10429. Train a small proxy under group DRO to produce domain weights, then resample and train the real model with them. Optimise worst-case EXCESS loss against a reference model, not worst-case loss — the naive form upweights whichever domain is noisiest, because every domain has a different irreducible entropy. | Active |
| SOTA-239 | Order training data by skill prerequisite: teach the prerequisite and the dependent skill costs less data | Chen et al. (2023), LIT-392 — ARXIV-2307.14430. Training data has a prerequisite structure, and it is exploitable: train on a prerequisite skill and the dependent skill is reachable with less data. Sample over skill mixtures online rather than fixing proportions in advance. | Proposed |
| SOTA-241 | Rank candidate data selections with a model-free distributional proxy before spending a training run | Xie et al. (2023), LIT-087 — KL reduction in a hashed n-gram space correlates with downstream performance across selection methods, including manual curation, which uses no n-grams at all. A selection can therefore be ranked against its alternatives before anything is trained on it. | Proposed |
| SOTA-242 | Select pretraining data by matching a target distribution you can sample from, not by scoring it for quality | Xie et al. (2023), LIT-087 — data selection needs a description of what you want, not a criterion for what is good. Supply a small sample of the target text and resample the raw corpus to match it. The target is an artifact you can produce; a quality rubric is an argument you have to win. | Proposed |
| SOTA-243 v2 | Discard the easy examples when data is abundant and the hard ones when it is scarce | Sorscher et al. (2022), LIT-399 — power-law scaling in dataset size is evidence of redundancy, and given a difficulty ranking it can be beaten toward exponential. But which end of the ranking to discard flips with how much data you started with, and the fraction to discard has to grow with the corpus. | Proposed |
| SOTA-244 | Score example importance a few epochs into training, averaged over several initializations | Paul et al. (2021), LIT-397 — the importance ranking is available from a single early checkpoint, not only from a finished run. EL2N at epoch 20, averaged over ten initializations, prunes half of CIFAR-10 with accuracy slightly up, and the ranking transfers to other architectures. | Proposed |
| SOTA-245 | State a relation in both orders in the training corpus if you want it usable in both directions | Grosse et al. (2023), LIT-401 — a training sequence influences a completion only when the phrase related to the prompt comes before the phrase related to the completion. Identical content in the reverse order scores barely above an unrelated baseline. | Proposed |
| SOTA-246 | Find mislabelled training data by self-influence, not by training loss | Koh and Liang (2017), LIT-403, with LIT-400 — rank training points by their influence on their own loss and inspect from the top. Mislabelled examples are strong proponents of themselves, so they sort to the front; high training loss does not separate them nearly as well. | Proposed |
| SOTA-267 v3 | Warm up on a subset of the entities before training on all of them, rather than holding one distribution fixed throughout | Zucchet et al. (2025), LIT-450 — the paper argues two quantities move in opposite directions: plateau length is governed by how often the most common entities appear, acquisition speed after the plateau by how often the least common ones do. So concentration buys an early exit from the plateau and costs the tail, and a warm-up that trains on a subset of the entities first, then on all of them, beat the best fixed distribution tried, in single-seed runs on synthetic biographies. | Proposed |
| SOTA-268 | Do not transfer a data mixing ratio across model scales; the threshold where a domain starts being learned moves with size | Gu et al. (2025), LIT-451 — knowledge acquisition from a knowledge-dense dataset mixed into web text has thresholds, not a smooth curve. Below a critical model size, or a critical mixing ratio, the model memorises almost nothing however long it trains; past it, accuracy jumps to over 60%. The critical ratio follows a power law in model size, so a ratio chosen at one scale can be on the wrong side of a transition at another. | Proposed |
| SOTA-271 | Train on a second modality even when the target is single-modality | Huh et al. (2024), LIT-458 — if representations converge on a modality-agnostic model of what generated the data, then data from any modality helps find it. The vision direction is already common practice; the language direction is not, and the paper cites a case where adding image data improved text performance. | Proposed |
| SOTA-295 | Parameterize the generating prompt above the word list, and constrain how each sample opens | Finke et al. (2025), LIT-485 — requiring a story to contain three words drawn from a vocabulary constrains the lexicon and leaves the frame alone, and the frame is where a generator repeats itself: 59.38% of TinyStories begins “once upon a time”. Name a theme, a topic, a style and a narrative feature per sample, and require the sample to open with a given part of speech and letter. Five diversity metrics move; measured simplicity does not. | Proposed |
| SOTA-341 | Condition an image generator on each training image’s original size instead of discarding or upsampling small images | Podell et al. (2023), LIT-566 — embed each training image’s original height and width, like the timestep, and give it to the model. Then keep the small images that a minimum-resolution filter would drop (39% of SDXL’s pretraining data below 256²) without teaching the model their blur. At inference, set the size you want. On class-conditional ImageNet 512²: FID-5k 43.84 discarding, 39.76 keeping unconditioned, 36.53 keeping with size conditioning. | Proposed |
| SOTA-352 | When a structure predictor is made generative, add a regression model’s predictions to the training data so disordered regions are learned as disorder rather than invented structure | Abramson et al. (2024), LIT-584 — a diffusion head generates plausible compact structure even where the protein has none. AlphaFold- Multimer, a regression model, renders the same regions as extended loops. Mixing its predictions into training teaches the generative model that convention, and “greatly reduced” hallucination. Remaining hallucinations are flagged by low confidence but do not look disordered. | Proposed |
| SOTA-354 | When unlabelled inputs vastly outnumber labelled ones, retrain from scratch on the labelled data mixed with the model’s own high-confidence predictions for unlabelled inputs, under augmentation that stops it copying them | Jumper et al. (2021), LIT-583 — about 100,000 solved structures and billions of sequences. AlphaFold 2 predicted structures for about 350,000 unlabeled sequences, kept the confident ones, and retrained the same architecture from scratch on those plus the PDB. Cropping and alignment subsampling mean the student cannot just reproduce the teacher. The ablation shows a clear gain. The filter is the model’s own calibrated confidence (SOTA-353). | Proposed |
| SOTA-359 v2 | Supervise vision from the caption, and match image to caption rather than predicting its words | Active | |
| SOTA-361 | Compose augmentations so the shortcut dies, and tune them for the contrastive objective rather than the supervised one | Active | |
| SOTA-369 | Curate a pretraining set by retrieving neighbours of curated seeds, rather than scaling the uncurated pool | Active | |
| SOTA-370 | Add many low-resolution views alongside the two full ones, because view count is what helps and resolution is what costs | Active | |
| SOTA-374 v4 | When learning embeddings from co-occurrence, subsample very frequent tokens, discarding each occurrence with probability 1 − √(t/f) with t around 10⁻⁵ | Mikolov et al. (2013), LIT-603 — in a Zipfian corpus, the few most frequent words co-occur with everything and dominate the updates while teaching little. Discard each occurrence of word w with probability 1 − √(t/f(w)). This cut skip-gram training time by 2–3× and raised accuracy, only slightly for words (59 → 60% at NEG-5) but a lot for phrases (27 → 42% at NEG-15, 19 → 47% with hierarchical softmax). | Active |
| SOTA-382 v3 | Subsample frequent tokens: it is faster and it improves the rare ones | Superseded — by SOTA-374; the same recommendation from the same paper and section; SOTA-374 states the discard rule and carries the independent ablation | |
| SOTA-387 | Hold out a slice of the training set for model selection instead of selecting on the validation split | Beyer et al. (2022), LIT-628 — they train on the first 99% of ImageNet-1k and keep 1% as a “minival”, stated as being “to encourage the community to stop selecting design choices on the validation (de-facto test) set.” A convention, asserted rather than measured, and the record held no document on it. | Proposed |
| SOTA-389 v3 | Caption training video with a model that watches the video, not with captions of its frames | Proposed | |
| SOTA-402 | To teach a model to reason over facts it holds, spend added data on facts derived from other facts rather than on more atomic facts | Wang et al. (2024), LIT-667. Two sweeps that separate what is normally varied together: with the inferred/atomic ratio φ fixed, scaling the training set changes nothing about how a transformer acquires a two-fact inference rule; with the size effect thus accounted for, raising φ moves the acquisition monotonically, and at φ = 18.0 the delay is gone — 96.7% before training accuracy has even saturated. Synthetic knowledge graphs only, and it buys rule application on facts already seen in compositions, not systematicity. | Proposed |
| SOTA-417 | Filter pretraining text by how well the tokenizer compresses it | Geiping and Goldstein (2023), LIT-681. Drop every document whose token count exceeds t times its character count. Text a tokenizer trained on the corpus compresses badly is mostly markup, code fragments and debris. No model is needed. On C4 it lifts GLUE from 75.9 to 79.3, and on already-clean bookcorpus-wikipedia from 78.1 to 78.7. The gain tracks how dirty the source was. | Proposed |
| SOTA-440 | Supervise a feed-forward 3D reconstructor with a frozen camera-controlled video model’s renderings instead of captured multi-view data | Bahmani et al. (2025), LIT-743 — Lyra trains a 3D Gaussian Splatting decoder only on a frozen video model’s RGB decodings along six camera paths per image, with no captured multi-view data. In its one controlled arm, training on real RealEstate10K + DL3DV instead scores PSNR 19.08 against 24.77, and adding the real data to the generated data gives 24.74, no better. Both are scored against the teacher’s own renderings on its own prompts, which is the distribution the generated supervision was drawn from — so the arm shows the generated data is sufficient there, not that it beats captured data on real scenes. | Proposed |
| SOTA-441 | When a web-scraped pair corpus is noisy, filter it by self-consistency: keep only the pairs whose passage a model trained on the noisy set ranks near the top | Wang et al. (2022), LIT-744, Table 7. Train a dual encoder on the noisy pairs, rank each pair’s passage against 1M random passages, and keep the pair only if it lands in the top 2. On six BEIR datasets, filtered beat unfiltered at 1M pairs (40.7 against 34.9 nDCG@10) and with all pairs (51.6 against 50.0), though the filtered set is about a quarter the size. One cutoff, chosen by inspection and never varied, and the filtering model’s cost is not counted. | Proposed |