How Agent Context Accumulation Works

LLMs have no memory — your loop has to supply it

A model call is stateless: the API has no idea what you asked five minutes ago unless you send it again. A simple chatbot handles this cheaply because the conversation grows slowly, a few hundred tokens per turn. An autonomous agent — a loop that plans, calls tools, reads results and decides the next step — grows its context far faster, because every tool result (a file read, a search result, a command's stdout) gets appended to the transcript the model has to see again on the very next step. The context isn't optional overhead; it é the agent's memory, and it is billed as input tokens on every single step that follows.

The arithmetic: why N steps cost more than N × one step

Say a step starts with a base context B (system prompt + tool schemas) and each step appends G tokens of new tool output before the next step runs. The input bill at step i é B + (i−1)·G — step 1 pays just the base, step 2 pays the base plus one tool result, step 10 pays the base plus nine tool results, and so on. Summed over N steps, total input tokens are:

Total input tokens = N·B + G·N·(N−1) / 2

That second term is the one that matters: it scales with N², not N. Double the number of steps and the output-side cost doubles, but the accumulated-context cost roughly quadruples. A naive estimate — "one step costs $X, so N steps cost N·$X" — only ever sees the linear part and misses the part that actually drives the bill on longer runs.

Worked example, in real dollars

Base context B = 2,000 tokens, tool-result growth G = 3,000 tokens/step, output O = 300 tokens/step (the model's next decision), priced on GPT-5 ($1.25 / $10 per 1M tokens, input/output). The "naive" column is what you'd estimate by pricing one step and multiplying by the step count — the number most back-of-envelope agent cost estimates stop at.

The gap doesn't creep up — it accelerates. At 10 steps the real cost is 4× the naive guess; at 100 steps it's 35×. At 1,000 concurrent runs a day for a month, that 30-step row alone is the difference between budgeting $4,950/month and actually being billed roughly $53,880/month. Nobody is overcharging you — the naive estimate was simply pricing the wrong shape of curve.

The fix: compaction turns the curve back toward linear

You can't avoid re-sending context — the model needs it to keep going — but you can cap how big it's allowed to grow before you reset it. Context compaction periodically feeds the whole accumulated context back to the model, asks for a compressed summary, and continues from that summary instead of the full transcript. It isn't free (summarizing a large context still bills it once as input, plus a small output for the summary) but it stops the quadratic term from compounding indefinitely.

Same 100-step run as above, now compacting every time context crosses 20,000 tokens and resetting to a 2,000-token summary (16 compactions fire over the run):

Compaction overhead itself is about 24.6% of that $1.95 — compaction isn't free, it's just dramatically cheaper than the alternative of re-reading an ever-larger transcript on every remaining step. Compact too early and you fire many small, wasteful summarizations; compact too late and each step still re-reads a near-maximal context before the reset lands. There's a real minimum in the middle, worth sweeping for your own step count and growth rate rather than guessing.

Three other levers, besides compaction

• Cap max steps. Because cost grows quadratically, a runaway loop (stuck re-trying, no progress) doesn't just waste time — it is disproportionately the most expensive failure mode you have. A hard step ceiling bounds the worst case directly.
• Prompt-cache the stable prefix. The base context and tool schemas don't change step to step; caching lets you re-read that unchanging prefix at a steep discount instead of full input price, which softens (but doesn't eliminate) the growth — the part that keeps growing, the tool-result tail, still isn't cacheable.
• Trim tool outputs before they join context. A tool that returns a 10,000-token raw API response when 200 tokens of summary would do is adding to G directly — the single biggest lever is usually not calling the model less, it's returning less per call.

Erros comuns

• Pricing an agent from one step's token count. A single step's cost tells you almost nothing about a 50-step run; the accumulated-context term dominates well before then.
• Budgeting on the average run length. Step-count variance (see the loop budget calculator above) and context accumulation compound each other — a run that's merely 2× longer than typical can cost 4× as much, not 2×.
• Treating compaction as pure overhead to avoid. Skipping it doesn't remove the cost, it defers it — and the deferred version grows quadratically instead of being capped.
• Letting raw tool output join context unfiltered. Every byte a tool returns gets re-billed on every subsequent step for the rest of the run, not just once.

Continuar aprendendo

Perguntas frequentes

Why do agent loops get so much more expensive than a single API call?

An LLM is stateless — it has no memory between calls, so every step of an agent loop must re-send the entire conversation so far: system prompt, tool schemas, and every prior tool result. Context grows by roughly a fixed amount each step, but it gets re-billed as input on every subsequent step too. Summed over N steps, total input tokens grow with N squared, not N.

How much does a long agent run really cost compared to a naive estimate?

With a 2,000-token base context, 3,000 tokens of tool output added per step and GPT-5 pricing ($1.25/$10 per 1M tokens), a 100-step run costs about $19.11 in practice versus $0.55 if you naively multiply one step's cost by 100 steps — a 35× gap, entirely from re-reading accumulated context on every step.

How does context compaction fix the quadratic growth?

Compaction periodically summarizes the full context and resets it to a short summary, capping how large the re-read gets before it grows again. That turns the cost curve back toward linear: the same 100-step run drops from about $19.11 to about $1.95 when compacted every time context crosses 20,000 tokens, roughly a 90% saving, even though the compaction events themselves add their own one-time cost.

Is context accumulation the same thing reasoning-token pricing causes?

No, they stack but come from different places. Reasoning tokens are hidden output tokens billed once per call. Context accumulation is an input-side effect across many calls in a loop — the growing transcript you resend, not the thinking a single call does. An agent using a reasoning model pays both taxes at once.