Published 2026-07-08 · reference numbers, verify before budgeting
When you build a chatbot with any LLM API, every message you send includes the full conversation history as input. Not just the latest user question — all previous turns, every assistant reply, everything. That's how LLMs maintain context. It's also how your costs quietly compound into something surprising.
By message 30 in a single conversation, the input token count has grown from roughly 1,150 tokens to over 11,000. At $5/1M input tokens (GPT-4o pricing), that one response costs you nearly 7× more than the first one did.
Say your chatbot has a 1,000-token system prompt, users send messages averaging 150 tokens, and the model replies with about 200 tokens each time. On turn k:
input(k) = 1,000 (system) + (k−1) × (150 + 200) (prior turns) + 150 (current message)
That simplifies to 800 + 350k. The cost climbs linearly with every turn — because each turn adds 350 tokens to every future call forever.
| Turn | Input tokens | Input cost ($5/1M) | Total turn cost |
|---|---|---|---|
| 1 | 1,150 | $0.0058 | $0.009 |
| 5 | 2,550 | $0.0128 | $0.016 |
| 10 | 4,300 | $0.0215 | $0.025 |
| 20 | 7,800 | $0.039 | $0.042 |
| 30 | 11,300 | $0.0565 | $0.060 |
Prices are reference estimates, July 2026. Report outdated price →
Output: 200 tokens × $15/1M = $0.003 per turn. Model: GPT-4o ($5/$15 reference pricing).
Turn 30 costs $0.060, turn 1 costs $0.009 — a 6.8× difference for the same 150-token user question and 200-token reply.
Adding up all 30 turns: the total input across the conversation is 186,750 tokens. Total cost:
If the API were stateless and each call only sent the current turn (1,150 tokens in), the same 30 turns would cost 30 × $0.009 = $0.27. The conversation history adds $0.75 in extra input cost — money spent re-reading what the model already processed.
10,000 users per day, each averaging 15 turns:
| Daily cost | Monthly cost | |
|---|---|---|
| 15-turn conversations (with history) | $1,840 | $55,200 |
| Stateless (hypothetical, no history) | $810 | $24,300 |
| History overhead | $1,030 | $30,900 |
The history overhead isn't a billing bug. It's the structural cost of building conversational AI on an API that prices every token on every call. Most teams discover it when the month-end invoice arrives.
1. Prune old messages. Keep only the last N turns (e.g., 5–8). Works well for factual Q&A assistants where old context rarely matters. Cheapest fix, implemented in an afternoon. Trade-off: the model forgets early context.
2. Compress with a summary. After every 5–10 turns, call the model once to summarize the conversation so far. Replace the raw history with the summary. Cost: one extra call per N turns. Benefit: history overhead stays nearly flat regardless of conversation length.
3. Sliding window with semantic filtering. Keep recent turns always, and selectively include only semantically relevant older turns using embeddings. More complex but preserves key context without full history cost. Worth implementing at scale for support or research assistants.
4. Prompt caching. On Claude and Gemini, marking the static system prompt as cacheable gives 50–90% off that portion. Doesn't help with the growing history (which is always unique), but reduces the fixed overhead. At $0.0050 from a 1,000-token system prompt per call, caching it saves ~$0.0045 per turn — meaningful across millions of calls. See the prompt caching savings calculator.
Most production chatbots should be using a rolling window of 6–10 turns plus a compression pass at some threshold. "Send all history forever" is a reasonable default for prototypes. It's an expensive default for production.
Put your own numbers in the conversation history cost calculator to see exactly where your cost goes across turns, compare models, and estimate compression savings. The difference between 30-turn history and a 5-turn window is often 60–70% lower cost for the same product experience.
Every API call re-sends the full conversation history as input tokens. Each turn adds the user message and assistant reply to everything that follows — so input size grows linearly with the number of turns.
The most practical fix is a rolling window: keep only the last 5–10 turns instead of all history. For assistants where context matters, periodic summarization lets you compress old turns into a short paragraph, keeping cost flat. Use the conversation history cost calculator to model the savings.
Only for the static portion (system prompt, tool definitions). The growing history is unique per conversation, so it can't be cached. Caching helps reduce the fixed overhead — the variable history part you pay for regardless.
With GPT-4o at $5/$15 per 1M tokens, a 1,000-token system prompt, 150-token user turns, and 200-token assistant replies: about $1.02 total. The first turn costs $0.009, the last costs $0.060 — nearly 7× more.
Reference numbers based on GPT-4o pricing, June 2026 — verify current rates on the provider's pricing page.