What prompt caching actually is
Most LLM apps send the same block of text at the start of every request — a long system prompt, a set of tool definitions, a few-shot examples, or a chunk of reference context. Normally you pay full input token price for that block on every single call, even though it never changes.
Prompt caching lets the provider store that stable prefix after the first call and, on later calls that begin with the exact same text, bill it at a small fraction of the normal rate — often around 10% (a ~90% discount). You still pay full price for the part that changes each call (the user's actual question), but the fixed overhead becomes almost free.
Prompt caching savings calculator →How it works across providers
There are two broad flavours, and the difference matters for your bill.
Explicit caching (e.g. Anthropic)
You mark where the cacheable prefix ends. The first call writes the cache and costs a little more than a normal input token — commonly about 1.25× the input rate — because the provider has to store the processed prefix. Every later call that reuses it is a cache read, billed at roughly 0.1× the input rate. The cache has a time-to-live (TTL) — often a few minutes — that refreshes each time it's hit, so a steady stream of traffic keeps it warm.
Automatic caching (e.g. OpenAI and others)
The provider detects repeated prefixes for you and applies a discount with no cache-write premium and no code changes. It's simpler, but you have less control: you can't force a long-lived cache, and the discount and eligibility rules are set by the provider (typically kicking in once a prefix passes a minimum length).
| Explicit cache | Automatic cache | |
|---|---|---|
| Who decides what's cached | You (mark the prefix) | Provider (detects repeats) |
| Cache-write cost | ~1.25× input, once | None |
| Cache-read cost | ~0.1× input | Discounted (provider-set) |
| Control over TTL | Yes | No |
Multipliers are typical published values and vary by model and provider — always confirm on the provider's pricing page.
The write-vs-read trade-off — when it pays off
With an explicit cache you pay a small one-time premium to write, then bank a big discount on every read. So caching pays off once you reuse the prefix enough times to offset the write cost. The break-even is quick: the write premium is only about 0.25× extra input, while each read saves about 0.9× input — so you're ahead after roughly two reuses of the same prefix. After that, every hit is nearly pure savings.
The ingredients that make caching a clear win:
- A large repeated prefix — the more fixed tokens, the bigger the discount in absolute terms.
- High call volume — many requests share the same prefix within the TTL window.
- A stable prefix — the reused block is byte-for-byte identical every call.
Retrieval-augmented apps, coding assistants with big system prompts, and multi-turn chat all fit this shape. See the broader tactics in the guide to cutting your LLM API bill.
A worked example
Say you have a 2,000-token system prompt (instructions + tool definitions + a few examples) that goes out with every request, and you make 100,000 calls. Assume an input price of $3 per million tokens, a cache-read rate of 0.1× ($0.30/M) and a cache-write rate of 1.25× ($3.75/M). We'll ignore the per-call user question and output here, since caching doesn't change those.
Without caching
Every call pays full price for all 2,000 prefix tokens:
100,000 × 2,000 = 200,000,000 tokens × $3/M = $600.
With caching
The prefix is written once and read on the other 99,999 calls (in practice it's re-written whenever the TTL lapses, but with steady traffic that's negligible):
- 1 write: 2,000 tokens × $3.75/M ≈ $0.0075
- 99,999 reads: ~200,000,000 tokens × $0.30/M ≈ $60
Total ≈ $60 versus $600 — a ~90% cut on the repeated portion of the bill, for the price of one line marking where the prefix ends. On real workloads the prefix is often larger and calls more frequent, so the absolute saving is bigger still.
Token cost calculator →Estimate your savings →Pitfalls to watch
- Everything before the changing part must be identical. Caching matches on an exact prefix. A timestamp, a user name, or a shuffled example order near the top invalidates the match and you silently pay full price.
- TTL expiry. If traffic is sparse and the gap between calls exceeds the cache's lifetime, the prefix expires and the next call pays the write cost again. Bursty, low-volume workloads benefit least.
- Small prefixes aren't worth it. Below a provider's minimum cacheable length — or when the fixed block is only a few hundred tokens — the savings are tiny and the write premium can leave you slightly worse off.
- Put the stable stuff first. Structure prompts so the unchanging system prompt, tools and context come before the volatile user input; caching can only cover the leading run of identical tokens.
Frequently asked questions
How much does prompt caching save?
On a cache read, most providers bill the cached tokens at roughly 10% of the normal input rate — a 90% discount on the repeated part of your prompt. The exact figure varies by provider, but a large, stable prefix reused across many calls is where the savings are biggest.
Does prompt caching cost extra to set up?
With explicit caches (such as Anthropic's), the first call that writes the cache costs slightly more than a normal input token — commonly about 1.25× the input rate. You recover that write premium as soon as you reuse the prefix a couple of times, after which every hit is billed at the deep-discount read rate.
When is prompt caching not worth it?
Caching only helps when a large prefix is identical across calls and reused before the cache expires. A short prefix, a prompt that changes every call, or low call volume means the write cost is never recovered — in those cases caching adds overhead instead of saving money.
Educational reference only — cache rates, TTLs and minimum lengths are estimates; confirm current terms on each provider's pricing page.