Su et al., NVIDIA (2024) — ARXIV-2412.02595
Key takeaways
- The critique is of a method the field had just adopted. FineWeb-Edu and DCLM won benchmark gains through aggressive model-based filtering, at the cost of removing 90% of the data — which is a good trade at a short token horizon and a bad one at 15T tokens, where you run out of corpus.
- The alternative is three things together: ensembling classifiers rather than trusting one, rephrasing text synthetically instead of discarding it, and leaning less on heuristic filters.
- Short horizon: a high-quality subset improves MMLU by 5.6 over DCLM when training 8B models for 1T tokens.
- Long horizon, which is the point: the full 6.3T-token dataset matches DCLM on MMLU while containing four times more unique real tokens. An 8B model trained for 15T tokens, 7.2T of them from this dataset, beats Llama 3.1 8B by +5 MMLU, +3.1 ARC-Challenge and +0.5 averaged over ten tasks.
Standing in the anthology
The other side of the filtering trade-off from LIT-184, and the pair is the point: how hard to filter is not a quality question with one answer but a question about the token horizon you are training to. Aggressive filtering wins at 1T and loses at 15T, because the thing it optimises — mean quality — is not the thing that binds when the corpus runs out.
It also connects to a disagreement the record already holds. SOTA-124 asks when high-quality data may be repeated; this asks how much data stays high-quality if you filter less; and LIT-166 prices repetition against unique tokens. All three are answers to “the corpus is finite” and none of them cites the others.