Chen et al. (2023) — ARXIV-2307.14430
Key takeaways
- The hypothesis is an ordering claim, not a quality claim: “just as humans acquire interdependent skills in a deliberate order, language models also follow a natural order when learning a set of skills from their training data.”
- A skill is defined by its data. The framework formalises a skill, and an ordered set of skills, “in terms of the associated data” — which is what makes the claim testable rather than a metaphor.
- The payoff is data efficiency: ordered skill sets exist, and “their existence enables more advanced skills to be learned with less data when we train on their prerequisite skills.”
- Skill-It is the resulting online sampler over skill mixtures, in two regimes — continual pretraining, where the goal is many skills, and fine-tuning, where it is one.
Standing in the anthology
Sources SOTA-239. Filed on the abstract and the framework statement
rather than a full reading, so it carries no NOTE (ADR-025) — the ordering
claim and the data-efficiency consequence are what the practice rests on, and
both are stated there.
It occupies a slot the record had entirely empty. Twenty-six practices carry
data-pipeline and they answer what to include — filtering (SOTA-170),
deduplication (SOTA-164), proportions (SOTA-166, SOTA-103), repetition
(SOTA-171) — and none answers in what order. The record’s only “curriculum”
documents are about training stages (pretrain, SFT, RL — SOTA-129,
SOTA-130), which is a different axis: that is ordering the objectives, this is
ordering the data.