Self-hosted inference cost is GPU-time per token. Speculative decoding raises tokens-per-second without changing the output, so it drops cost per token — but only if the draft is accurate enough that accepted tokens outrun the extra draft passes. The net speedup, not the headline one, is what shows up on the bill.
Baseline vs speculative
Plain autoregressive decoding is one token per target pass. Speculative decoding lifts throughput by the net speedup, and cost per token falls with it — provided the net speedup clears 1×.
| Mode | Throughput (tok/s) | Cost / 1M | Cost / month |
|---|
Sensitivity to acceptance rate
The single number that decides everything is how often the draft guesses right. Below the break-even acceptance rate for your draft depth and overhead, speculative decoding is a net loss.
| Acceptance rate | Tokens / pass | Net speedup | Cost / month |
|---|
Why accepted guesses are almost free
Decoding a large model one token at a time is bottlenecked by memory bandwidth: every single token requires streaming the whole weight set through the GPU, and that fixed cost dwarfs the arithmetic. The clever observation behind speculative decoding is that verifying k proposed tokens in one forward pass costs almost the same as decoding a single token — you stream the weights once either way. So if a cheap draft model proposes a handful of tokens and the big model accepts most of them, you got several tokens for the price of one. The output is identical to normal decoding because the target still checks every token and rejects anything it wouldn't have produced; only the throughput changes. On self-hosted inference, where your bill is essentially GPU-hours divided by tokens produced, more tokens per GPU-second is money straight off the top.
The formula, and the overhead that bites
With a per-token acceptance rate a and a draft depth k, the expected tokens the target emits per verification pass is E = (1 − ak+1) ÷ (1 − a) — a geometric run of accepted guesses plus one bonus token. That is the raw speedup. The catch is the draft itself: it runs k small forward passes per cycle, so if the draft costs a fraction c of the target per pass, each cycle really costs 1 + k·c target-equivalent units. The net speedup is E ÷ (1 + k·c), and cost per token drops by 1 − 1 ÷ netSpeedup. Push k too high on a workload where a is low and E grows slowly while k·c grows linearly — the net speedup sinks toward, and past, 1×. That is the failure mode this tool guards: a draft that is confidently wrong is worse than no draft at all.
Where it fits
Speculative decoding is a self-hosting lever, not an API one — hosted providers already price it in, so you capture it only when you run the GPU. Pair it with the GPU cloud cost calculator to price the hardware you're accelerating, the LLM VRAM & concurrency calculator to see how batching interacts with it, and the self-hosted vs API calculator to decide whether running your own model beats paying per token in the first place. A related trick, distilling a smaller model, cuts the target's own cost rather than accelerating it — the two stack.
How this calculator works
It computes the expected tokens per target verification pass, E = (1 − ak+1)/(1 − a), from your acceptance rate a and draft depth k, then the net speedup E/(1 + k·c) after the draft's per-pass overhead c. Baseline cost per million is GPU$/hour ÷ (throughput × 3600) × 1,000,000; the speculative cost divides that by the net speedup. Monthly cost is your output-token volume at each rate, the saving is the difference, and any configuration whose net speedup falls below 1× is flagged as a net loss.
Frequently asked questions
What is speculative decoding and why does it save money?
A small draft model proposes tokens the big model verifies in one pass; accepted guesses are near-free throughput. On self-hosted inference, cost is GPU-time per token, so higher throughput means lower cost per million — with identical output, since the target checks every token.
How is the net speedup calculated?
Expected tokens per verify pass E = (1 − a^(k+1))/(1 − a) divided by the draft overhead 1 + k·c. Cost per token drops by 1 − 1/netSpeedup.
When does it lose money?
When acceptance is low and the draft depth is high, the k extra draft passes cost more than the accepted tokens save, dropping net speedup below 1×. This tool flags that case.