data-pipeline.
20 of 378 NOTE documents. Back to the full index.
| # | Title | Summary | Status |
|---|---|---|---|
| NOTE-002 | Efficient Online Data Mixing | An Exp3 bandit over data domains, rewarded by per-domain training loss on the batches the run is already taking, revising the mixture during training. 19% fewer iterations to the next best method’s final perplexity, at negligible wall-clock cost. | Read |
| NOTE-018 | Segment Anything | The data engine is the transferable part. Three stages — assisted-manual, semi-automatic, fully automatic — each stage’s labels training the model that produces the next stage’s, ending in 1.1B masks. The model and the dataset are built together. | Read |
| NOTE-031 v2 | Data Selection for Language Models via Importance Resampling | Selects pretraining data by importance resampling in a hashed n-gram feature space — 10,000 buckets of unigram and bigram counts — to match a target distribution. The more useful result is the metric: KL reduction in that feature space predicts downstream performance across selection methods, including manual curation, which uses no n-grams at all. | Read |
| NOTE-039 | PaLM 2 Technical Report | Independently re-derives compute-optimal scaling at larger compute than Chinchilla and lands on the same answer — grow data and parameters roughly 1:1, against the earlier 3:1 model-first trend. States the inference-economics corollary plainly: for a fixed training and inference budget, train a smaller model on more tokens rather than making the architecture inference-efficient. | Read |
| NOTE-069 | PaLI: A Jointly-Scaled Multilingual Language-Image Model | Scales language and vision components jointly across 100+ languages and shows a properly scaled model handles many languages while keeping English-only state of the art. Its most useful passage is an ablation honest enough to explain a regression by the language ratio of the data that caused it. | Read |
| NOTE-175 | DoReMi: Optimizing Data Mixtures Speeds Up Language Model Pretraining | A 280M proxy under group DRO sets the domain weights for an 8B run: +6.5 points few-shot and the baseline in 2.6x fewer steps, with no downstream task in the loop. The transferable part is the objective — worst-case EXCESS loss against a reference model, because worst-case raw loss would upweight whichever domain is noisiest. | Read |
| NOTE-176 | Beyond neural scaling laws: beating power law scaling via data pruning | Power-law scaling of error in dataset size is a symptom of redundancy, not a ceiling. With a ranking of example difficulty and a pruning fraction that grows with the corpus, error can fall exponentially instead. Which end of the ranking to discard inverts with data abundance: keep hard examples when data is plentiful, easy ones when it is scarce. | Read |
| NOTE-177 | Deep Learning on a Data Diet: Finding Important Examples Early in Training | Two scores that rank training examples by importance from a single early checkpoint — GraNd (expected loss-gradient norm) and EL2N (norm of the error vector) — averaged over several initializations. They prune half of CIFAR-10 without loss, transfer across architectures, and come with the finding that the very highest-scoring examples should be excluded, more so under label noise. | Read |
| NOTE-178 | Studying Large Language Model Generalization with Influence Functions | Scales influence functions to 52B-parameter language models with an EK-FAC approximation to the IHVP and query batching over the gradient bottleneck, then uses them as an instrument. Generalization gets more abstract with scale; influence is heavy-tailed but not concentrated; and a training sequence only counts when the prompt-related phrase comes first. | Read |
| NOTE-180 | Estimating Training Data Influence by Tracing Gradient Descent | Defines a training example’s influence as the total change it caused in a test point’s loss over the iterations where it was used, and approximates that by step-size-weighted gradient dot products at saved checkpoints. No Hessian, no convexity. Self-influence ranks mislabelled examples to the top. | Read |
| NOTE-181 | Understanding Black-box Predictions via Influence Functions | Traces a prediction back to the training points responsible for it by upweighting each point infinitesimally and differentiating, which needs only gradients and Hessian-vector products rather than retraining. Establishes the four uses of training-data attribution — explanation, poisoning, domain-mismatch debugging, and finding mislabelled data. | Read |
| NOTE-191 | Pre-training under infinite compute | Under a fixed 200M-token corpus and no compute limit, the standard recipe overfits; tuning weight decay to ~30x the customary 0.1 makes loss monotone in parameter count. Ensembling independently trained models then beats parameter scaling on asymptote, and distils back down at ~83% retention. | Read |
| NOTE-192 | Diffusion Beats AR in Data-Constrained Settings | Refits LIT-166’s data-constrained scaling law with the objective swapped from autoregressive to masked diffusion. The half-life of data reuse R* goes from 31.93 to 512.85. Diffusion loses badly at Chinchilla-optimal compute and wins past a crossover given in closed form. | Read |
| NOTE-199 v2 | How do language models learn facts? | Factual recall is learned in three phases, and the plateau between generic statistics and individual knowledge is the attention recall circuit being built — patching in a trained model’s attention patterns removes the plateau. Imbalanced data shortens the plateau and, the paper argues, slows acquisition after it; a warm-up on a subset of individuals before training on all of them beat the fixed distributions tried, in single-seed runs. | Read |
| NOTE-201 | Data Mixing Can Induce Phase Transitions in Knowledge Acquisition | Linear scaling of acquired knowledge in model size holds when a knowledge-dense dataset is trained on alone and breaks once it is mixed into web text: below a critical model size, or a critical mixing ratio, the model memorises almost nothing. Attributed to capacity allocation as a knapsack, with the critical ratio a power law in model size. | Read |
| NOTE-233 | TinyStories | Read as the missing trunk under eight tiny-models practices. The result that matters is a separation, not a record: what stops a 125M model speaking coherent English is the breadth of the corpus, not the complexity of language — established by holding the architecture and shrinking the corpus. | Read |
| NOTE-234 | SimpleStories | Read to see what, if anything, it establishes that its trunk did not. The dataset comparison is direct and well instrumented; the model comparison is confounded three ways and the authors say so. The practice this record takes from it is about the generation procedure, which was measured. | Read |
| NOTE-321 | AlphaFold 2 | Near-experimental protein structure accuracy at CASP14, a calibrated confidence head, self-distillation on confident predictions, and a hard dependence on alignment depth below about 30 sequences. Main text and Methods read. The supplementary methods (architecture details, ablation descriptions) were not read. | Read |
| NOTE-333 v2 | Wan | An open 14B text-to-video recipe whose evidence is mostly systems measurement: 2D context parallelism cuts communication overhead from over 10% to under 1%, and caching and 8-bit kernels give 1.62× and 1.27×. Its three modelling ablations run on 1.3B text-to-image. Two are read from training loss, and on FID one of them favours the rejected option. The applications in §5 are demonstrated almost entirely by figures. | Read |
| NOTE-336 | Stable Video Diffusion | At fixed architecture and a fixed step count, a video model pretrained on a curated quarter of a 9.8M-clip pool is preferred by human raters to one trained on the whole pool, and the lead survives high-quality fine-tuning (Elo +127 against +89 over an image-only start at 10k steps). Almost every result is a preference bar from 64 prompts with no numbers in the text, and two of the curation choices the paper shipped lost its own ablation. | Read |