LIT-087: Data Selection for Language Models via Importance Resampling

Xie et al. (2023) — ARXIV-2302.03169

Key takeaways

• Frames pretraining data selection as matching a target distribution you can sample from — not as scoring data for quality — and solves it with classical importance resampling made tractable by working in a reduced feature space: unigrams and bigrams hashed into m = 10,000 buckets
• The metric is the bigger result. KL reduction — how much a selection reduces distance to the target versus random selection, measured in that hashed space — correlates strongly with downstream performance across selection methods, including ones that use no n-grams at all, such as manual curation. A model-free proxy that ranks data selections before any training
• The crude features are deliberate: the space is not claimed to represent text well, only to preserve the ordering of candidate selections
• +0.9% over heuristic classification, comparable to top-k heuristic classification — which the authors note may suit domain-specific selection where “diversity may be less important than in the general-domain setting”
• Within-domain F1 82.9% vs cross-domain 81.2%: a 1.7-point average cost to selecting for a target in a different domain

Standing in the anthology

Read — NOTE-031. Sources SOTA-242 and SOTA-241.

The reading puts a cheaper instrument beside one the record carries. SOTA-166 says to set data proportions by fitting a mixing law on small runs rather than by argument — right, and it costs a set of training runs. If KL reduction really ranks selections without training, it is a pre-run filter that makes searching a mixing law cheaper. Whether the record should carry it is a registry decision: the evidence is correlational, from 2023, at a scale below where mixing laws are now fitted.

Of the takeaways replaced, “data quality assessment” and “quality-aware data mixing” describe scoring data for quality, which this paper deliberately does not do.

Audited by #121, which named two candidates and filed neither. R2 — frame selection as matching a target you can sample from rather than as scoring quality — was the gap: no data-pipeline practice was about selection as distribution matching. R1 — evaluate a selection with a model-free proxy before training on it — is the cheap check that follows from it, and the reading qualifies it as correlational.

The audit declined on two grounds: that reconciling the two frames was larger than its scope, which is fair, and that whether the record should carry this at all was an open registry decision, which is not — DP-006 had already answered it. Both are now filed: R2 as SOTA-242, R1 as SOTA-241, both Proposed, with the doubt the audit recorded moved into their promote_when: conditions where something can come due against it. ADR-043 is the general rule that came out of the case.