Gururangan et al. (2020) — ARXIV-2004.10964
Key takeaways
- Two stages, and the second is the surprise. Domain-adaptive pretraining (DAPT) continues pretraining on a large unlabelled corpus from the target domain; task-adaptive pretraining (TAPT) continues it on the task’s own unlabelled training text
- Both help, and they are complementary — doing both beats either, which is what makes them two effects rather than one
- TAPT is the unobvious half: a few thousand unlabelled task examples are worth their own pretraining stage even after the domain corpus has been used
- Gains are largest where the target domain is furthest from the pretraining corpus
- RoBERTa, four domains, eight classification tasks, 2020 scales
Standing in the anthology
The source of SOTA-033.
Read in full for #114; the reading is NOTE-020. The practice is confirmed.
The reading adds the paper’s own names, DAPT and TAPT, which matter
because this is two findings and the practice’s title carries neither —
least of all the second, which is the one that was not obvious.