What each fallback rate costs
Same tokens, same traffic — only the primary→fallback trigger rate changes.
| Fallback rate | Monthly cost | Overhead vs primary | Overhead % |
|---|
The hidden cost of your reliability tier
A model fallback chain is the standard 2026 AI-gateway pattern: send the request to a cheap, fast primary model; if that call is rate-limited or errors, the gateway (OpenRouter, Portkey, LiteLLM, or a custom wrapper) automatically retries against a pricier, more reliable fallback model, and sometimes a third tier if the fallback also fails. It is the right architecture for uptime — but the fallback tier is usually priced 5–20× the primary per token, precisely because the primary was chosen for being cheap. A fallback rate that looks negligible in a dashboard — 5%, even 2% — can move the bill far more than that percentage suggests.
The math is not linear because the two tiers aren't priced the same. Blended cost per request = primary_cost × (1 − p1) + fallback_cost × p1 × (1 − p2) + tertiary_cost × p1 × p2, where p1 is the primary-to-fallback trigger rate and p2 is the (much rarer) fallback-to-tertiary rate. With a 14× price gap between tiers, a 5% fallback rate can add 50–65% to the blended bill even though 95% of requests still hit the cheap primary — because that 5% of traffic is paying more than half the total dollar cost.
This is a genuinely different shape of cost than deliberate model routing (where you choose the cheap/expensive split by design) or a straightforward retry-on-failure cost (where the same model retries itself). Here the escalation target is a different, pricier model, and it's triggered by infrastructure conditions — capacity, rate limits — not by task difficulty.
Related: model routing savings, API retry cost, LLM vendor migration cost, agent loop budget.
How to use it
1. Pick a preset chain or enter your own per-million prices for primary, fallback and tertiary tiers.
2. Enter monthly requests and typical input/output tokens per request.
3. Set your observed (or estimated) primary→fallback and fallback→tertiary trigger rates.
4. Read the blended monthly cost and the overhead versus a pure-primary baseline; use the table to see how sensitive your bill is to the fallback rate.
Common mistakes
Assuming fallback cost scales with the fallback rate. It doesn't — it scales with the fallback rate times the price ratio between tiers, which is usually much larger. Ignoring the tertiary tier. If your fallback also fails under sustained load, the tertiary tier (if any) usually costs the same as the fallback but adds latency risk — model it even at a low rate. Billing failed calls. Most providers don't charge for a hard rate-limit rejection; if yours does (partial generation before a mid-stream error), your real overhead is higher than this estimate. Chasing 100% uptime with the most expensive fallback. A mid-tier fallback often recovers most of the reliability at a fraction of the cost of jumping straight to the flagship model.
FAQ
What is an AI model fallback chain?
A gateway pattern where a failed or rate-limited request automatically retries against a backup model, and optionally a third tier if that also fails. Used by OpenRouter, Portkey, LiteLLM and similar tools.
Why does a small fallback rate cost so much?
Fallback models are usually priced far above the primary per token. Even a 5% trigger rate can add 50%+ to the bill if the fallback costs 10-14x more per request.
Are rejected requests billed?
Standard practice: a hard 429 rate-limit rejection bills $0, since no tokens were generated. This calculator assumes that convention.
How do I reduce the fallback rate cheaply?
Raise your primary tier's rate limit, add backoff-and-retry on the primary before escalating, spread traffic across keys/providers, or insert a mid-tier fallback instead of jumping straight to the most expensive model.
Estimate only. Model prices are reference figures and change frequently — verify current pricing with each provider.