Raffel et al. (2019) — ARXIV-1910.10683
Key takeaways
- C4, the Colossal Clean Crawled Corpus, is introduced here: a single Common Crawl snapshot put through heuristic cleaning — English-only detection, discarding lines that do not end in terminal punctuation, dropping boilerplate and placeholder text, a badwords filter, and deduplication of repeated spans. Roughly 750GB of English text
- The cleaning is heuristic and English-only, which is the property to carry. C4 is not “the web”; it is one crawl under one filter design, and what those filters keep or drop is a set of choices nobody has to agree with
- The paper’s own subject is the text-to-text framework and a systematic ablation of transfer-learning choices. The corpus was infrastructure built to run that study, and it is the part this record depends on
Standing in the anthology
Carries no practice, deliberately — ADR-032.
Forty-six documents mention C4 — thirty-nine readings, four literature notes, two practices and one explanation — and the paper that made it was not in the record at all. It plays two distinct roles here, and they want different caveats.
As a corpus with a known defect, it is load-bearing. SOTA-164 says to
deduplicate the pretraining corpus at both substring and document
granularity, and its headline evidence is a fact about C4: LIT-202 found
one 61-word English sentence repeated thousands of times in it. That practice
is not really a claim about corpora in general — it is a claim grounded in
what survived this paper’s filters, which already included a deduplication
step. The interesting reading is that C4 was deduplicated by its authors and
the repeats were still there, which is what makes the two-granularity
argument necessary rather than obvious.
As a standard workload, it is a convention. SOTA-155’s 500×
communication reduction, LIT-212, LIT-252, LIT-276 and THEORY-014’s
comparison table all use C4 as the language-modelling task that distributed
and communication-efficient methods are measured on. Nothing about those
results depends on C4 specifically; it is there because everyone else used it,
which DP-007 is the name for. SOTA-155 already marks the narrow
part — “eight workers on C4 is a long way from the arrangement the motivation
describes”.
The same shape as LIT-424: a dataset introduced inside a paper about
something else, which outlived the argument it was built to serve.
The T5 set-up itself is also a baseline others have measured against. LIT-709 (Primer) reran T5 at 537M on Raffel et al.’s exact schedule: the original T5 reached 13.25 perplexity in 15.7K TPUv3-hours, against 4.6K for T5++ (SwiGLU + RMSNorm) and 3.8K for Primer, which is where Primer’s “4.2×” comes from. On T5’s own encoder–decoder masked-LM setting, Primer-EZ did no better than T5++.
Unread — no NOTE.