Source
Wang, Yue, Su and Sun (2024), LIT-667 — ARXIV-2405.15071.
The setting is a transformer trained from scratch on a mixture of atomic
facts — (subject, relation, object) edges of a random knowledge graph — and
inferred facts, which a latent rule deduces from pairs of atomic ones. The
quantity the recommendation is about is
φ = |inferred facts in training| / |atomic facts|
and the model’s ability to complete inferred facts it has not seen arrives by grokking: training accuracy saturates above 99% long before test accuracy leaves single digits.
What the two sweeps establish, and why it takes two. Raising φ also
raises the total amount of training data, so on its own the φ sweep would not
say which of the two did the work. The second sweep is what settles it: hold
φ = 9.0 and vary the number of entities — and therefore the training-set
size, which scales linearly with it — and nothing qualitatively changes,
not the gap between the train and test curves, not the level of generalization
reached. Then the φ sweep at fixed size moves grokking speed monotonically,
and at φ = 18.0 the model reaches 96.7% before training accuracy has
saturated — the delay the phenomenon is named for is simply gone. Both sweeps
are Wang et al.’s, LIT-667, run on two-hop composition with the φ sweep at
2,000 entities, and the recommendation originates with them too: they state the
result as a correction to the critical-data-size hypothesis, name critical data
distribution in its place, and offer it as guidance for setting up data “to
better induce implicit reasoning”.
So the operational form of this is not “add data” and not “add less data”. It is: when you have a budget of additional examples and a model that already holds the facts, spend the budget on statements derived from those facts. Another million atomic facts buys the acquisition of the rule nothing.
Conditions, and what it does not buy
It buys in-distribution rule application, not systematicity. Across every
φ the authors tried, out-of-distribution composition — applying the rule to
atomic facts that appeared in training only in atomic form — stayed at zero,
out to two million optimization steps. Whatever this knob is for, it is not
for getting a model to compose facts it has only ever seen stated alone. That
limit has its own explanation in THEORY-103, and the fix
there is architectural, not a data ratio.
It assumes you can tell derived from atomic. The whole effect is indexed on
a partition of the training data that a synthetic generator hands you for free
and a natural corpus does not. On real text the analogous quantity — how much
of the corpus states a consequence of something else it states — is not
something anybody has measured, let alone moved deliberately. The
promote_when is about exactly that gap.
Weight decay is part of the setup. The source trains with AdamW at weight decay 0.1 and reports that raising it accelerates grokking further. The account that explains the ratio effect (THEORY-071) needs the regularizer, so a run with the regularization turned off is outside what has been measured here.
Everything is synthetic, with a unique token per entity and an 8-layer GPT-2. The authors argue in their limitations that the control this buys is worth the distance from practice, and the record agrees — while pricing the distance in, which is what the status is for.
Known implementations
- None known. The source is a controlled study, not a training report.