Tokens vs Requests Explained

A request and a token are not the same thing

An API request is one call to the endpoint. Tokens are the units of text inside that call. One request can carry a handful of tokens or hundreds of thousands. LLM providers bill by tokens; many other APIs (payments, maps, SMS) bill per request or per action. Mixing the two up is the most common budgeting mistake.

Where each model applies

How to estimate before you build

For an LLM feature, estimate: tokens per call × calls per user × users. A chat reply might be ~500 input + ~300 output tokens; multiply by monthly conversations to get token volume, then price it. This is far more accurate than guessing "per request", because token count per request varies wildly. Keep input and output separate all the way through — providers price them differently, often 3–5× apart, so a blended "average token" estimate can be off by a large margin.

The same split shows up again in rate limits

Most LLM providers cap usage on both axes at once: a requests-per-minute (RPM) ceiling and a tokens-per-minute (TPM) ceiling. Which one you hit first depends entirely on how many tokens your average call carries — the exact number this page is about.

• Short calls, high frequency (e.g. a classifier firing on every keystroke) tend to hit the RPM cap first — each call is cheap in tokens but there are a lot of them.
• Long calls, lower frequency (e.g. a document-summarization endpoint with a 20k-token context) tend to hit the TPM cap first — a handful of calls can already saturate the token budget.

Knowing which limit you'll hit changes the fix: an RPM-bound workload benefits from batching several small calls into one larger request (fewer requests, same total tokens), while a TPM-bound workload benefits from trimming context or output length rather than reducing call frequency. See API Rate Limits Explained for the concrete RPM/TPM tiers by provider.

Worked example: sizing a chatbot feature

Say you're planning a support chatbot for 5,000 monthly active users, each averaging 2 sessions/month of 3 turns apiece. Every turn carries roughly 900 input tokens (system prompt + recent conversation history + the user's message) and 250 output tokens.

• Tokens per turn: 900 + 250 = 1,150
• Turns per user/month: 3 × 2 = 6
• Tokens per user/month: 1,150 × 6 = 6,900 (≈4,900 input / 1,500 output)
• Total monthly tokens: 6,900 × 5,000 = 34.5M (≈24.5M input / ~10M output)

Run the input and output totals through a token cost calculator separately for your chosen model — because output is priced higher, the 10M output tokens can cost as much as the 24.5M input tokens, even though it's a third of the volume.

Common mistakes that blow up an estimate

• Blending input and output into one price. Output tokens are usually 3–5× the input rate — estimating both at the input price understates the bill.
• Forgetting resent history. Multi-turn chat that resends prior messages each call means token count per turn grows through a session, not just per-message.
• Estimating by characters, not tokens. Token count per character varies by language and content — code, JSON and non-English text typically use more tokens per character than plain English.
• Ignoring retries. A failed call that gets retried still consumes request quota, and often tokens too, so error-prone integrations cost more than the happy-path math suggests.
• Skipping the rest of the pipeline. A RAG or agent feature usually adds embedding requests, vector-search calls and sometimes a smaller routing model — each billed on its own, separately from the main model's tokens.

Continue learning

Frequently asked questions

What is the difference between a token and an API request?

An API request is a single call to the endpoint; tokens are the units of text inside it. One request can contain very few or hundreds of thousands of tokens. LLMs bill per token, while many other APIs bill per request.

How many tokens is a typical chatbot message?

A short user message plus a system prompt is often a few hundred input tokens, and a reply a few hundred output tokens — roughly 500–1,000 tokens per exchange, though it varies with prompt length and answer verbosity.

How do I estimate my monthly token usage?

Multiply tokens per call by calls per user by number of users. Estimate input and output tokens separately since they're priced differently, then run the total through a token cost calculator for your chosen model.

Does a token cost the same whether it's input or output?

No — output tokens are typically priced higher than input tokens for the same model, often 3–5× as much. Keeping input and output totals separate, rather than blending them into one average, gives a much more accurate cost estimate.

What else besides the main model's tokens should I budget for?

In a RAG or agent pipeline, budget for embedding requests, vector database queries, and any smaller routing or classification model calls — each is billed separately from the main model's tokens and can add up across a high-volume feature.

Do rate limits count requests or tokens?

Usually both at once — providers set a separate requests-per-minute (RPM) cap and tokens-per-minute (TPM) cap, and you hit whichever one your traffic pattern saturates first. High-frequency short calls tend to hit RPM first; low-frequency long-context calls tend to hit TPM first.