SOTA-229: Scale the draft model's training data, once nothing constrains it to predict the target's features

What to do

If you train your own draft model for speculative decoding (SOTA-227), treat its training data as a lever that keeps paying, and remove the thing that stops it paying:

• Predict tokens directly. Do not train the draft to reproduce the target’s hidden features and derive tokens from them.
• Fuse features from several depths of the target rather than reusing the top layer alone.
• Simulate the multi-step draft during training — feed the draft’s own step-one output into step two — which LIT-185 calls training-time test and which is not optional once the feature loss is gone.

Then scale the data. LIT-185 uses roughly 8x EAGLE’s and reports the speedup still climbing.

Why

The ceiling was an objective, not a capacity limit. Predicting the target’s features is a useful scaffold: it gives the draft a signal for multi-step generation that pure token prediction does not. It is also a constraint on what the draft can express, and LIT-185’s observation is that it is the binding one — remove it and first-token acceptance improves immediately with more data.

Removing it alone breaks the second step, which is the part worth understanding rather than copying. Without the feature loss, the draft’s step-one output no longer lands near the ground-truth feature that step two was trained against, so step-two acceptance collapses. Training-time test closes that gap by making the training distribution the serving distribution. The three changes are one change.

And the reason top-layer features were never enough: for a full-rank LM head, the top-layer feature is information-equivalent to the next token’s logits. Predicting the next-next token from it is asking for information it provably does not carry. Intermediate layers have it, and only become usable once the feature loss is removed.

What it is worth

Up to 6.5x speedup, about 1.4x over EAGLE-2 at batch size 1, across five tasks on both chat and reasoning targets — MT-bench and GSM8K, at temperature 0.

And, separately: +40% throughput at batch 64 in SGLang. That number belongs to SOTA-227’s conditions more than to this practice, and it is recorded there.

Conditions

You have to be training the draft. Everything here is about how a draft is built. If you are using someone else’s, SOTA-227 is the practice and this one is why theirs improved.

The draft needs the target’s intermediate layers. That rules it out for a target served behind an API, and couples the draft to a specific target — with nothing published about what acceptance does when that target is updated.

“Scaling law” is the paper’s word for a trend. One target, one benchmark, a data range that is not large, and no fitted exponent or range of validity. The claim this practice makes is that returns had flattened and no longer do — not that a particular curve holds.

The two architectural changes are not ablated apart. Direct token prediction and multi-layer fusion are motivated together and measured together, so their separate contributions are not recoverable from this paper.

The settings are algorithm-local and stay in the reading. Layer choices, fusion details and data volumes belong to NOTE-163 under ADR-041.

Known implementations

• SGLang ships the method. That is adoption of the architecture, not an independent measurement of the scaling claim, and is counted in consensus_note per DP-005.