Source
Chen et al. (2023), LIT-392 — ARXIV-2307.14430.
The claim
Data has an order, and the order is not arbitrary. The framing is explicit: “just as humans acquire interdependent skills in a deliberate order, language models also follow a natural order when learning a set of skills from their training data.”
What makes it a practice rather than an analogy is the consequence: ordered skill sets exist, and “their existence enables more advanced skills to be learned with less data when we train on their prerequisite skills.”
So the recommendation is to model the prerequisite structure and sample accordingly — online over skill mixtures, in continual pretraining where the goal is many skills, and in fine-tuning where it is one.
The recommendation originates with LIT-392, which proposes both the framing and the sampler, Skill-It, and measures it in both regimes. On the synthetic LEGO task in continual pretraining it is 36.5 points more accurate than random sampling. On Natural Instructions in fine-tuning it cuts validation loss on the target skill by 13.6% against training only on that skill’s own data, which is the prerequisite claim in its plainest form. Applied to RedPajama to continue pretraining a 3B model, it beats uniform sampling over data sources at 3B tokens using 1B. These are the abstract’s figures; the record has not read the body.
The axis this adds
The record’s data-pipeline practices answer what to include and in
what proportion: filtering (SOTA-170), deduplication (SOTA-164),
proportions (SOTA-166, SOTA-103, SOTA-238), repetition (SOTA-171).
None answers in what order.
The distinction to hold, because the record’s vocabulary invites collapsing them:
- Reweighting changes how much of each domain, possibly over time. That is SOTA-103 and SOTA-238.
- Ordering changes what must come before what, because of a dependency between capabilities. That is this.
- Staging orders the objectives — pretrain, SFT, RL (SOTA-129, SOTA-130). The record’s existing uses of the word “curriculum” are all this third thing, and it is a different axis again.
Conditions
Proposed, and the reason is scope rather than doubt. The evidence is in
continual-pretraining and fine-tuning regimes; nothing here demonstrates an
ordered skill set governing a pretraining run from scratch, which is where the
data-ordering question has the most money attached to it.
Identifying skills and their prerequisite structure is itself work, and the framework defines a skill “in terms of the associated data” — so applying this means committing to a decomposition of your corpus that you then have to defend. That cost is real and the practice should not be read as free.