Speculative decoding: draft models, acceptance rates, and what the speed-ups actually were

About this article: AI-assisted writing, reviewed against the cited sources. This is source research, not a lab validation of the procedures. Check the stated versions and your own recovery requirements before making changes.

A large language model is slow at generation because each new token waits for a full pass over the weights. Speculative decoding does not make that pass cheaper. It tries to make one pass pay for several tokens. A smaller draft model guesses a short continuation. The large model scores those guesses in one go. Guesses that the large model would have been willing to produce are kept. The first guess it would not have produced is replaced by a corrected sample. If the acceptance rule is the one in the papers, the tokens you keep are still samples from the large model.

An AI writing system produced this article for NetLeverage's editorial studio. "Guess" here means a sample from a smaller distribution, not a claim about experience. No speed-up was measured here. Numbers are from the papers, on their models and hardware.

Why it matters

Decode is often limited by memory traffic, not by arithmetic. Chen et al. put it in hardware terms: for a sharded transformer, a ceiling on autoregressive speed is about the model size divided by the total memory bandwidth, because every new token has to move the weights. Scoring a handful of already-drafted tokens moves those weights once. If the draft is right often enough, and cheap enough, the time per accepted token falls. If either condition fails, you have added a second model and a rejection step and you are slower.

The same papers are explicit about what does not change. You do not retrain the target. You do not accept a biased distribution in exchange for speed, provided the rejection rule is implemented. What does change is latency variance, memory for the draft, and the assumption that the accelerator has spare capacity to score several positions at once. A serving stack that is already full of concurrent decodes may not have that spare capacity. Both papers measured the small-batch case.

How it works

One loop

Leviathan, Kalman, and Matias (2022) and Chen et al. (2023) describe the same loop. Chen et al. cite the earlier paper. The letters are a trap: Leviathan et al. use $p$ for the target and $q$ for the draft. Chen et al. use $q$ for the target and $p$ for the draft. This article says target and draft, and uses $p$ for the target and $q$ for the draft, matching Leviathan et al.

1. The draft samples $\gamma$ tokens (Chen et al. call this length $K$) one after another, each conditioned on the prefix plus its own earlier guesses. This part is serial, but the draft is small.
2. The target scores the whole guess in one forward pass, producing the target probability of each drafted token given the true prefix up to there.
3. Walking left to right, each drafted token $x$ is kept with probability $min(1,p(x)/q(x))$. If $q(x)\le p(x)$, it is always kept. If the draft was more confident than the target, it is kept only with probability $p(x)/q(x)$.
4. The first rejection is replaced by a sample from the residual distribution: renormalise $max(0,p(x)-q(x))$ over the vocabulary. Leviathan et al. write this as $p^{\prime }(x)=\mathrm{n}\mathrm{o}\mathrm{r}\mathrm{m}(max(0,p-q))$. Chen et al. write $(q-p{)}_{+}$ in their lettering, which is the same residual of target minus draft.
5. If every drafted token is accepted, one extra token is sampled from the target at the end of the guess. A loop therefore returns between 1 and $\gamma +1$ tokens. Chen et al. state the lower bound directly: if the first draft token is rejected, the residual sample is still a valid token, so the loop never returns empty.

Discard every draft token after the rejection. They were conditioned on a token you did not keep.

Why the distribution matches

The proof, in both papers, is a per-token rejection argument. The probability of emitting a particular token is the probability the draft proposed it and it was accepted, plus the probability that some proposal was rejected and the residual distribution then selected it. Those two paths add up to $p(x)$. Leviathan et al. show this for any draft distribution, with no requirement that the draft share the target's architecture. Chen et al. add the practical caveat "within hardware numerics": a different computation graph and a different use of the random-number generator mean you should not expect the same string as ordinary sampling, even when the distribution matches. They also say a single numeric disagreement under greedy decoding can send two strings apart. Their check was downstream metrics, not string equality.

Leviathan et al., comparing against the T5X implementation, report identical outputs in that setup. Read that as a result of their implementation and their sampling settings, not as a promise that every engine's speculative path is bit-identical. If an engine fuses the rule into a kernel, verify the kernel against this probability, not against a single chat transcript.

Acceptance rate is overlapping probability mass

Leviathan et al. define the acceptance probability $\beta$ for a prefix as the chance that a token drawn from the draft is kept. Under an assumption that these chances are independent across steps, $\alpha =E(\beta )$. They show

For a fixed prefix, this equation gives $\beta$; averaging over prefixes gives $\alpha =E(\beta )$. Here $D_{LK}$ is the total-variation distance between the two distributions (half the L1 distance, written by them via the midpoint distribution). $\alpha$ is high when the draft and the target put their probability on the same tokens. It is not the fraction of draft tokens that happened to match an argmax in one log, unless you are in the temperature-zero case, where the rule collapses toward "did the draft pick the target's chosen token".

If the $\beta$ values behave as independent draws with mean $\alpha$, the number of tokens produced by one loop is a capped geometric random variable. The expectation is

At $\alpha =0.8$ and $\gamma =4$, that expression is $(1-{0.8}^5)/0.2=3.36$ tokens per target call, before you charge for the draft. This is arithmetic on their formula, not a new measurement. Later tokens in the guess are only reached if earlier ones were accepted, so raising $\gamma$ has diminishing returns. Chen et al. plot that: the fraction of the $K+1$ slots that are actually used falls as $K$ grows, and on their XSum run the latency was best at $K=3$, not at the largest $K$ they tried.

The speed-up formula, and when it is less than one

Let $c$ be the time for one draft step divided by the time for one target step. Leviathan et al. assume the target can score $\gamma +1$ positions in parallel at the same wall time as one ordinary step. The expected wall-time improvement is then

If $\alpha >c$, some positive $\gamma$ helps, and the gain is at least $(1+\alpha )/(1+c)$. If the draft is almost as expensive as the target, $c$ is near 1 and the inequality fails even when $\alpha$ looks flattering. In their experiments, drafts about two orders of magnitude smaller kept $c$ under 0.05, often near zero. A larger draft raised $\alpha$ and raised $c$, and the wall time got worse. On English-to-German with T5-XXL, T5-large had a higher acceptance rate than T5-small and a lower speed-up (1.7× against 3.4× at temperature 0).

The assumption behind the formula is spare parallelism. Section 3.4 of Leviathan et al. says the arithmetic concurrency grows by about $\gamma +1$. Rejected positions are wasted work. If the device is already busy, scoring the guess is not free, and the formula does not apply. Chen et al. place the method in the small-batch latency regime. They describe quantisation as reducing sampling latency. Shrinking the cache, including through multi-query attention, becomes especially useful as batch size grows and cache bandwidth matters more. Their conclusion places speculative sampling alongside other techniques for reducing small-batch latency.

What they measured

Leviathan et al. ran batch size 1 on one TPU v4. The target was T5-XXL (11 billion parameters). Drafts were existing T5 checkpoints, not models trained for the purpose. The table below is their Table 2, for the draft they mark as best in each row group, plus the pattern that a bigger draft was not automatically better.

Temperature 0 accepted more often than temperature 1. A bigram draft on English-to-German had $\alpha \approx 0.2$. With $c\approx 0$ and $\gamma =3$ that still gave 1.25×, which they report as real and still worse than T5-small. They also computed $\alpha$ for a 97M model and a 137B LaMDA dialogue setup. Those are acceptance studies, not the T5X wall-time table. The abstract's "2×–3×" is the T5-XXL comparison against T5X, with identical outputs in that implementation.

Chen et al. used Chinchilla, 70 billion parameters, on 16 TPU v4 chips, again at batch size 1. The draft was a 4 billion parameter model they trained on the same tokeniser and dataset: narrower, and only 8 layers, so it could be sampled on the same 16-chip slice. Draft latency was 1.8 ms per token against 14.1 ms for Chinchilla. They explain why "use a 7B" is the wrong shortcut on that hardware. A Chinchilla-style 7B was fastest on 4 TPU v4s (5 ms per token). Putting that 7B on 16 chips made it slower, because the extra communication was not paid back by bandwidth. A draft has to be fast on the devices the target already occupies, or those devices sit idle while the draft runs somewhere else.

At $K=4$:

XSum was 11,305 sequences, capped at 128 tokens. HumanEval was 16,400 samples, capped at 512. They attribute the larger code speed-up to repetitive subsequences, shorter tokens, and a sharper temperature. Raising $K$ increased the spread of the time to finish a sequence. They flag that as a problem if you care about the slow tail, not only the mean. On HumanEval and greedy XSum, the speed-up beat the memory-bandwidth ceiling they state for ordinary one-token sampling. That is possible only because one weight-load produced more than one token.

Drafts that are not a second full model

Both papers allow any draft that exposes the probabilities the rejection rule needs. Leviathan et al. include n-gram tables, which are cheap lookups. Chen et al. list training extra heads on the target, citing Stern et al. (2018), and sequence-level distillation. They set those aside for large models because they need target samples or a change to the target, and they use the smaller standalone model instead. This article did not open Stern et al., so the head-based method is not described here.

Current engines name several drafts in documentation opened the same day, without new measurements in this draft:

• The vLLM README lists speculative decoding with n-gram, suffix, EAGLE, and DFlash.
• The llama.cpp server help lists --spec-type values including draft-simple, draft-eagle3, draft-mtp, draft-dflash, and several n-gram modes, defaulting to none. --spec-draft-n-max defaults to 3. The GGUF spec describes an optional mtp- filename prefix for a separate multi-token-prediction file; MTP tensors may instead be embedded in the main model file.
• TGI's serve help has an integer --speculate. The TGI README's "~2× latency" line is their wording. It is not one of the tables above.

Those names are not interchangeable with the 2022 rejection rule until you read the engine's sampler. An n-gram draft still needs a defined $q(x)$ if you want the distribution proof. A method that accepts a draft token whenever it matches the target's argmax, and does nothing else, matches temperature-zero behaviour more closely than it matches temperature-1 sampling.

What to choose and why

A workable check, without a lab claim attached, is the one the formula suggests. Log, per request, how many draft tokens were proposed and how many were accepted, and the time of a loop with the draft disabled. If accepted tokens per target forward are near 1, $\alpha$ is too low or $K$ is too long. If the draft's own step time is a large fraction of the target step, $c$ is too high. Either way the feature is not paying for itself, whatever the README says.

Risks and limits

• The two papers swap $p$ and $q$. An implementation comment that copies one paper's formula and the other paper's variable names will accept the wrong tokens and silently bias the model.
• Distribution match is not string match. Chen et al. say so, and they point at numerics and at the random-number stream. Leviathan et al. did observe identical outputs against T5X. Treat bit-identity as something to test on your build.
• Greedy decoding amplifies a one-token miss. Their warning is that two strings can diverge after a single disagreement. Comparing one prompt by eye is a weak test of correctness.
• The speed-ups are batch size 1, on TPU v4, for T5-XXL and Chinchilla. They are not a forecast for a quantised model on a GPU that is already batching. HumanEval's 2.46× was the friendly task, because code is repetitive. XSum nucleus was 1.92×.
• Tail latency can worsen as $K$ grows even when the mean improves. Chen et al. say the variance of time-to-finish rose with $K$.
• The formula assumes the target scores the guess in about one step's wall time. Long $K$, a slow attention kernel, or a batch that is already wide will break that. Their own figure shows loop time rising roughly linearly with $K$, reflecting additional draft calls and small increases in scoring time.
• A draft that does not share the tokeniser cannot be scored token-by-token against the target. Chen et al. trained the draft with Chinchilla's tokeniser. Neither paper measures a mismatched vocabulary.
• Extra heads and later engine drafters have different provenance. Chen et al. cite Stern et al. (2018) for extra heads; the later engine documentation names approaches such as EAGLE. Neither table establishes every later implementation's acceptance behaviour.
• No command was run. No NetLeverage product behaviour is claimed.

Sources

1. Yaniv Leviathan, Matan Kalman, and Yossi Matias, Fast Inference from Transformers via Speculative Decoding, arXiv:2211.17192 — https://arxiv.org/abs/2211.17192 (HTML https://arxiv.org/html/2211.17192 opened 10 November 2026). Rejection rule, fixed-prefix $\beta =\sum min(p,q)$ and mean $\alpha =E(\beta )$, expected tokens, wall-time theorem, T5-XXL table.
2. Charlie Chen, Sebastian Borgeaud, Geoffrey Irving, Jean-Baptiste Lespiau, Laurent Sifre, and John Jumper, Accelerating Large Language Model Decoding with Speculative Sampling, arXiv:2302.01318 — https://arxiv.org/abs/2302.01318 (HTML https://arxiv.org/html/2302.01318 opened 10 November 2026). Chinchilla 70B measurements, draft topology, $K$ trade-off, numerics caveat. Note their swapped letters.
3. Engine references, opened 10 November 2026: vLLM README, https://github.com/vllm-project/vllm; llama.cpp server README, https://github.com/ggml-org/llama.cpp/tree/master/tools/server; GGUF specification, https://github.com/ggml-org/ggml/blob/master/docs/gguf.md; TGI README, https://github.com/huggingface/text-generation-inference; TGI architecture, https://huggingface.co/docs/text-generation-inference/architecture. These support implementation names and flags, not independently measured performance.

This article was originally published on 10 November 2026.