Source
Lee et al., Google Research (2021), LIT-202 — ARXIV-2107.06499.
Lee et al. are where this practice starts: LIT-202 built and released both tools below, and measured what deduplicating standard language-modelling corpora with them bought — the three results in the next section.
Run two passes, because they catch different things:
- Exact substring matching, which finds long repeated spans wherever they sit — the paper’s example is one 61-word English sentence appearing thousands of times in C4. C4 (LIT-425) had already been deduplicated by its own authors when Lee et al. found that, which is what makes the second pass below a requirement rather than a refinement.
- Approximate near-duplicate detection at document granularity, which finds documents that are the same page with different boilerplate.
A repeated paragraph is not a duplicated document, and a pipeline that runs only one of the two passes leaves the other failure mode in the corpus.
What it buys, which is three things usually conflated into one
- Memorisation. Deduplicated models emit memorised training text about ten times less frequently in unprompted generation.
- Efficiency. They reach the same or better accuracy in fewer training steps.
- Honest evaluation. Train-test overlap contaminates a large fraction of the validation sets of standard datasets, so the numbers mean more afterwards.
Only the second is an efficiency argument. A pipeline that deduplicates for speed alone and stops when the speed-up looks small has given up the other two without noticing.
Filed late, and the reason is worth keeping
This was in every pipeline the record describes before it was in the record. A step nobody argues about produces no debate to notice, which is a different way of going unrecorded from a technique that is too new to endorse — and the registry was missing nodes at both ends. Its practical consequence is SOTA-124, which says when deliberate repetition of good data is safe; this is the paper establishing that repetition is not free by default, and that the damage shows up as verbatim emission rather than as loss.
Known implementations
- Falcon-H1-Tiny, Kimi K3, DeepSeek-V4, Olmo 3 — and, so far as the record can tell, everything else.