LLM API usage
Cache cost
| Metric | Without cache | With cache |
|---|
How semantic caching works
Every incoming query is converted to an embedding vector. The cache checks if a similar vector exists (cosine similarity ≥ threshold, typically 0.92–0.97). If yes, the stored response is returned instantly — zero LLM tokens consumed. If no, the query goes to the LLM and the result is stored for future hits.
Where hit rate comes from: repetitive queries (FAQ bots, customer support) see 30–60%. Diverse open-ended queries (coding assistants, creative writing) see 5–15%. You can increase hit rate by lowering the similarity threshold, but this risks serving slightly-wrong cached answers.
Embedding cost: Every query (hit or miss) costs an embedding lookup. At $0.02/1M tokens, embedding a 300-token query costs $0.000006 — negligible unless you have millions of queries per day.
See also: Prompt caching savings · RAG cost calculator · Vector DB cost
Frequently asked questions
What is a semantic cache for LLMs?
A semantic cache stores previous LLM responses and uses embedding similarity to detect queries that are semantically similar. When similarity is above threshold, the cached answer is returned without calling the LLM, saving the full token cost for that query.
What cache hit rate is realistic?
Customer support bots with repetitive questions often see 30–60% hit rates. Open-ended chatbots or coding assistants may see 5–15%. A good similarity threshold (0.92–0.97 cosine) balances hit rate against quality degradation.
What is the break-even hit rate?
Break-even = cache cost ÷ (queries/month × tokens × LLM price/token). If your LLM costs $0.003/query and the cache costs $50/month with 100,000 queries/month, you need only 1.7% hit rate to break even.
How does semantic caching differ from prompt caching?
Prompt caching (Anthropic cache_control, OpenAI automatic caching) reuses KV computation for identical token prefixes. Semantic caching is at the application level — it serves full cached responses when queries are semantically similar. They are complementary.
What tools provide semantic caching for LLMs?
Popular options: GPTCache (open source), Momento Vector Index (managed), Zep (memory layer), LangChain CacheBackedEmbeddings, Redis with vector search. Managed options cost $10–$200/month; self-hosted Redis costs $5–$30/month on a small VM.