Source
Xie et al. (2023), LIT-087 — ARXIV-2302.03169.
The result this rests on is not the method
The paper’s headline is a selection method. The finding that matters here is the one it needed in order to argue for itself: KL reduction — how much a selection reduces distance to the target relative to random selection, measured in a hashed n-gram space — correlates strongly with downstream performance across selection methods, including methods that use no n-grams at all, such as manual curation.
The proxy, and the suggestion to use it before training, both come from LIT-087: Xie et al. define KL reduction in the same 10,000-bucket hashed unigram-and-bigram space DSIR selects in, and report it tracking downstream performance across the selection methods they compare. The correlation is measured across a handful of methods at small model scale, and it is the whole of the evidence here.
That last clause is the whole claim. A metric that only ranked n-gram selections would be the method’s own objective wearing a metric’s clothes. One that also ranks a hand-curated corpus is a proxy for data selection in general, computable with a hash and a counter, before any model exists.
Why a bad feature space is the right feature space
Ten thousand hash buckets of unigram and bigram counts cannot represent text and are not claimed to. What is claimed is that the space preserves the ordering of candidate selections, and ordering is all a screen needs.
The general form, which is the transferable part: when a decision only needs candidates ranked, the proxy has to be monotone with the outcome, not faithful to the object. Faithfulness is the expensive property and it is routinely bought when it was not the one required.
What it is for
Every other way the record has of choosing data costs training runs.
SOTA-166 fits a mixing law on a sample of mixtures; SOTA-238 runs a proxy
model to convergence to produce one weighting. Both are cheaper than
training at target scale and neither is free, and both of them have to pick
which candidates to spend their budget on.
A model-free proxy is the stage before that: it does not tell you the best mixture, it tells you which candidates are not worth a run. Used that way the correlational weakness below matters much less than it would if the proxy were making the final call — a screen that is right about the ordering most of the time still removes most of the cost.
Conditions
The evidence is a correlation across a handful of selection methods, at one scale, in 2023. It is the paper’s own second result rather than the thing it set out to show, and nothing in this record re-runs it.
It inherits the target requirement. The proxy measures distance to a target distribution, so it presupposes the frame filed beside it — with no sampleable target there is nothing to compute a KL reduction against.
Below the scale its consumers operate at. The record’s data practices are argued at 8B parameters and 15T tokens. Whether an n-gram distance still orders selections correctly where selection is model-based and targets are implicit is exactly the open question, and the one the condition above names.
Why Proposed
DP-006: the evidence is weak-but-honest and the instruction is real, so the
document exists and the status carries the doubt. The cheapness is what makes
it worth filing at this strength — a screen that costs a hash and is wrong
sometimes is a different proposition from a recommendation that costs a
training run and is wrong sometimes.
Known implementations
- DSIR — the paper’s own release; KL reduction is reported there as the metric used to compare selections, including ones DSIR did not produce.