SOTA-227: Decode with a draft model and an accept-reject rule, which is exactly lossless

What to do

Put a cheaper model in front of the one you are serving.

1. Draft gamma tokens by running the small model autoregressively.
2. Run the target once, scoring all gamma + 1 prefixes in a single batched pass.
3. Walk the draft left to right. Accept position i when r_i <= p_i(x)/q_i(x) for r_i ~ U(0,1), where p is the target’s probability and q the draft’s. At the first rejection, resample that position from the normalised positive part of p - q and stop.
4. If nothing was rejected you get a free extra token from the target’s own distribution at position gamma + 1.

Any model that exposes logits over the same vocabulary is admissible as the draft. The accept-reject rule — not the draft’s quality — is what carries the guarantee.

Why it is free rather than a trade

The distribution is exactly the target’s. This is the property that matters and the one most summaries drop. You are not approximating the big model with the small one; you are using the small one to propose and the big one to decide, under a rule constructed so the composite sampler is unbiased. LIT-375 states the qualifier worth keeping — the guarantee holds “within hardware numerics” — and LIT-376 measures its results against identical outputs.

It cannot be slower in target evaluations. Every parallel target pass emits at least one token, so the count of serial target runs is bounded above by what plain decoding would have taken. The only way to lose is the drafting overhead.

The compute it spends was already idle. LIT-376’s premise, stated outright: decoding “is often not bottlenecked on arithmetic operations, but rather on memory bandwidth and communication, so additional computation resources might be available”. LIT-375 measures the consequence — tokens per second that “often exceeds the idealised ceiling on auto-regressive sampling speed imposed by the memory bandwidth”. The ceiling being beaten is the one that made decoding slow in the first place.

And the economics are computable in advance, which is unusual. Let alpha = E(min(p, q)) be the acceptance rate and c the draft’s per-run cost relative to the target’s. Then

expected walltime improvement = (1 - alpha^(gamma+1)) / ((1 - alpha)(gamma*c + 1))

so gamma is chosen numerically rather than by habit, and a candidate draft is evaluated on measured alpha and c rather than on its benchmark scores. For a near-free draft the improvement approaches 1/(1 - alpha).

Conditions

It is chiefly a latency technique, and the batch size decides how much it pays. The whole argument rests on idle arithmetic. Serving one stream against a large model, that is the regime, and as batch size grows the speculative work increasingly competes with real work rather than filling a gap. Neither source measures the crossover.

But “the trade reverses” was too strong, and this record held the counter-example while writing it. LIT-185 reports EAGLE-3 improving throughput by 40% at batch size 64 in SGLang, noting explicitly that speculative sampling “is often thought to reduce throughput at large batch sizes”. That is one framework at one batch size with a very good draft, so it does not establish a new rule — but the first version of this practice said the record held no measurement of the crossover, and the measurement was in LIT-185, unread, at the time. Treat the crossover as workload- and draft-specific, and measure it rather than assuming its sign.

Draft and target must share a tokenizer and vocabulary. The rule compares per-token probabilities from both models at the same positions.

Acceptance is workload-dependent and drifts. alpha is a property of the draft-target pair on your prompts. A draft that is not retrained alongside a target that keeps being post-trained will quietly get worse, and nothing in either paper addresses that.

2022-2023 infrastructure. T5-XXL and LaMDA in one paper, Chinchilla in the other. Neither interacts with paged attention (SOTA-105) or continuous batching (SOTA-113), which attack the same bottleneck from different directions and which this record holds separately without anything relating the three.

gamma fixed per loop leaves something on the table. LIT-376 estimates up to ~60% further improvement from adapting it within a generation and does not pursue it.

What builds on this

The record already held two refinements of a practice it did not hold:

• LIT-185 (EAGLE-3) is a method for driving the draft’s cost toward the c ≈ 0 regime by making it a single decoder layer matched to the target’s structure. It is Active here, and read since (NOTE-163).
• SOTA-162 trains auxiliary multi-token-prediction heads, and one of its two arguments is that “the extra heads are a draft model for speculative decoding you did not have to train separately” — a claim that had no document behind it until now.

That ordering is DP-007’s second failure mode exactly: the refinement filed before the thing it refines. Filing this does not change either document’s content; it gives both something to stand on.

Known implementations

None recorded as such. Three model reports in this record name speculative decoding — LIT-131, LIT-135, LIT-136 — which is adoption of the arrangement rather than an independent measurement of it, and is counted in consensus_note where counting is the right operation (DP-005).

One correction while counting: LIT-185’s standing section said LIT-135 and LIT-139 “ship the same arrangement”. LIT-135’s note bears that out; LIT-139’s did not, and this document left it unrepaired rather than repair it from the same distance that produced it. Settled by reading the full report (NOTE-361): DeepSeek-V4 keeps V3’s multi-token-prediction module “without modification” as a depth-1 training objective, and the report never mentions a draft model, an acceptance rate or speculative decoding. It is not an adopter of this practice on its own evidence, is not counted above, and LIT-185 now says so.