The fallback rate is the whole ballgame. A distilled model that handles 90% of your traffic and routes the rest back to the teacher does not deliver the headline multiple — the escalated tenth is billed at full frontier rates and typically accounts for most of whatever bill is left. Enter an honest number below and watch the payback period move.
What the fallback rate does to your ROI
Same student model, same up-front spend — only the share of traffic that has to go back to the teacher changes. This is the variable that decides whether a distillation project is a quarter-long win or a rounding error.
| Fallback rate | Effective / mo | Saving / mo | Saving vs teacher | Payback |
|---|
Why the headline "20x cheaper" number is not your saving
Distillation works, and the published multiples are real: train a small student to imitate a large teacher on a narrow task and the per-token serving cost typically drops somewhere between five and twenty times. What those figures describe is the cost of a token passing through the student instead of the teacher. What they do not describe is the cost of running a production system, which is a different quantity, because production systems do not route 100% of traffic to a model that is merely almost as good. They route the confident cases to the student and escalate the rest — and the rest is charged at full teacher rates. At a 10% fallback rate the teacher is still handling a tenth of your volume at 30x the student's price, which means it is still responsible for the majority of your remaining bill. Modelling distillation as a clean swap is the single most common error in these business cases.
Up-front cost is mostly people, not GPUs
The compute line is almost always the cheapest part. Labelling 50,000 examples through the teacher costs a few hundred dollars at typical rates. A fine-tune on a small student is tens of GPU-hours — call it another hundred dollars at commodity rental prices. Then comes the part that does not appear in any vendor's cost calculator: someone has to curate the training set, build an eval harness that can actually tell you whether the student is good enough, define the confidence threshold for the fallback router, and validate it against production traffic before you dare shift real users onto it. Forty engineering hours is an optimistic estimate for that work, and at any realistic blended rate it dominates everything else in the up-front column. That is why the calculator asks for it explicitly instead of quietly leaving it at zero.
The clock you are racing
The last consideration is one no spreadsheet forces on you: frontier prices keep falling. A project with a two-month payback is safe against that; a project with an eighteen-month payback is betting that the teacher's price and the small-model landscape both stand still for a year and a half, which recent history says they will not. If the payback figure above lands past a couple of quarters, the honest read is usually that you should reach for a cheaper off-the-shelf model, prompt compression or caching first — see the prompt caching savings calculator and the model routing savings calculator, both of which deliver a fraction of the saving for a fraction of the commitment. Distillation earns its keep on high-volume, narrow, stable tasks, and gets punished everywhere else. Compare against a straight fine-tune with the fine-tuning vs prompting calculator, or against running the model yourself with the self-hosted LLM vs API calculator.