Source
Xie et al. (2023), LIT-087 — ARXIV-2302.03169.
The two frames
Almost everything the record holds about choosing pretraining data is
quality-shaped: score each document for how good it is, keep the good
ones. SOTA-170 is the record’s central filtering practice and the thing it
tunes is how aggressive a quality classifier to run; the FineWeb-Edu and
DCLM work it argues with differs on the setting, not on the question. The
question is always is this document good enough.
DSIR asks a different one. Given a small sample of the text you actually want — the target — and a large raw corpus, select a subset distributed like the target. That is classical importance resampling, and the paper makes it tractable by estimating the weights in a reduced feature space rather than over text.
The frame is LIT-087’s: Xie et al. pose data selection as matching a target
you can sample from rather than as scoring quality, and implement it with
importance weights estimated over unigrams and bigrams hashed into 10,000
buckets. Selecting from the Pile this way, DSIR beats heuristic
classification by 0.9% and is comparable to its top-k variant, the margin
the conditions below turn on.
Why the frame is the recommendation and the method is not
A quality criterion is a claim about documents in the abstract, and it has to be defended: whose definition of quality, good for what, measured how. Every argument about filtering aggressiveness is an argument about that criterion, and it is unfalsifiable until somebody trains on it.
A target distribution is an artifact. You produce it by pointing at text you want more of. It is not a claim that the target is good, only that it is what you are aiming at, so the argument moves from a rubric nobody can settle to a sample anybody can inspect. The specification is also reusable: the same target admits any selection method, which is what makes selections comparable at all — and is the precondition for the proxy filed alongside this one.
The method is incidental to that. Hashed n-grams are 2023’s tractable estimator and will not be the last one; the instruction here is about what you supply as input, not about how the weights get computed.
Conditions
The target has to exist, and it has to be sampleable. This is the real cost, and it is where the frame can simply fail to apply. If what you want is “generally capable”, there is no sample to point at, and the quality frame is not being lazy — it is answering the only question anyone has stated.
Breadth changes which frame wins. The paper’s own qualification: for
domain-specific targets, plain top-k heuristic classification is
competitive, and the authors reason that “diversity may be less important
than in the general-domain setting”. DSIR’s margin over heuristic
classification is 0.9%, and its advantage is attributed to selecting the
most diverse source distribution. So the frame buys most where the target is
broad enough that a scorer collapses it, and least where the target is narrow
enough that a scorer captures it.
Selecting for the wrong target costs about 1.7 points. Within-domain F1 82.9% against cross-domain 81.2% — small, and the right order of magnitude to know before reusing somebody else’s target.
Hashed n-grams cannot express structure, reasoning or correctness. “Match the target” is only as good as the target, and nothing in the mechanism will warn you that the target was badly chosen.
Why Proposed
The evidence is a 0.9% margin from 2023, at a scale well below where the
record’s other data practices operate, and the correlational result it rests
on has not been re-run since. That is a reason not to mark it Active and
not a reason for the record to be silent on the frame — DP-006. What the
condition above is waiting for is somebody stating a selection target as a
sample, or running the two frames against each other; neither has happened in
the literature this record holds.
Known implementations
- DSIR — the paper’s own release, selecting from the Pile.