What prompt caching actually does

When a model processes your input, most of the work happens on the input tokens before it generates a single word of output. Prompt caching saves the processed state of a prefix of your prompt so that identical prefix does not have to be processed again on the next call.

The key word is prefix. Caching works on the beginning of your prompt, up to the point where content starts to differ. If the first 5,000 tokens of every request are identical (say, a fixed instruction block plus a reference document) those tokens can be cached, while the unique user question at the end is processed normally.

Why it saves money

Providers charge much less for tokens read from cache than for tokens processed fresh. A common structure is that cached input tokens cost a small fraction of the normal input rate, often around one tenth, though the exact discount varies by provider.

Here is an illustrative example (invented rates for clarity). Suppose input normally costs $3 per million tokens, and cached reads cost $0.30 per million. If you send a 10,000-token fixed prompt on 1,000 calls, processing it fresh every time costs 10,000 x 1,000 = 10 million tokens x $3 = $30. With caching, the first call pays full price and the rest read from cache: roughly 10,000 x $3/million + 9.99 million x $0.30/million, close to $3. That is the scale of saving repetitive prefixes make possible.

There is usually a write cost and a time limit

Caching is not purely free. Most providers charge a small premium to write content into the cache the first time, sometimes around 25% above the normal input rate for those tokens. You recover that on subsequent reads, so caching pays off only when the same prefix is reused enough times.

Caches also expire. A typical time-to-live is a few minutes of inactivity, with some providers offering longer retention for an additional cost. If your calls are spread far apart in time, the cache may expire between them and you lose the benefit. Caching rewards bursty, high-frequency reuse of the same content.

When caching is worth it

Prompt caching shines in specific patterns:

• Long, fixed system prompts reused across many users or requests.
• Document Q&A where one large document is queried many times in a session.
• Few-shot prompting with a big block of examples that never changes.
• Chatbots where the early conversation turns stay constant while new turns are appended.

It is not worth it when every request is unique, when your fixed prefix is small (a few hundred tokens), or when calls are rare and far apart so the cache keeps expiring.

How to structure prompts for caching

The golden rule is: put the stable content first, the variable content last. Order your prompt so the unchanging system instructions, reference material, and examples sit at the top, and the user's specific question goes at the bottom. Because caching works on the shared prefix, any variation near the start breaks the cache for everything after it.

Avoid sprinkling dynamic values (timestamps, user names, random IDs) into the early part of the prompt. Even a single changed character early on can invalidate the cache. Keep those dynamic bits in the tail of the request where they belong.

Estimating the payoff

To decide whether caching helps, compare two numbers: your fixed-prefix size in tokens, and how many times you reuse it within the cache window. The more tokens in the prefix and the more times you reuse it, the bigger the win. A tiny prefix reused twice saves almost nothing; a 20,000-token prefix reused hundreds of times per hour is where caching transforms your bill.

You can model both scenarios in the LLM cost calculator by comparing a run priced entirely at the standard input rate against one where most input tokens are priced at the cached rate. Seeing the two totals side by side makes the decision obvious, and the model comparison pages show which providers publish cached-token pricing.

Continue learning

Related tools

Frequently asked questions

Does prompt caching change the model's output?

No. Caching only reuses the processed input state to save cost and latency. The model produces the same output it would have without caching, because the underlying tokens are identical.

How long does a cached prompt stay valid?

It varies by provider, but a common default is a few minutes of inactivity before the cache expires. Some providers offer extended retention for an extra fee. Frequent reuse keeps the cache warm.

Is there any risk to caching sensitive data?

Cached content is tied to your account and used only to serve your own repeated requests, but you should still follow your provider's data policies and avoid caching anything you are not permitted to store.