API Rate Limits & Throughput Explained

What a rate limit actually is

A rate limit is a cap on how fast you can use an API. Providers impose them to keep the service fair and stable: without them, a single buggy loop or a traffic spike from one customer could starve everyone else, so limits protect capacity, enforce plan tiers, and blunt abuse and denial-of-service attacks. When you cross a limit, the API stops serving you for a moment and returns an HTTP 429 "Too Many Requests" response instead of doing the work.

Limits come in a few common units, and you can hit any of them first:

On LLM APIs the two that bite most often are RPM and TPM, and they are independent. Fifty tiny classification calls can blow past RPM while barely touching TPM; one 100K-token document summarisation can blow past TPM in a single request. Design for whichever ceiling your workload reaches first.

Turning a TPM limit into "how many users can I serve?"

Capacity planning is just arithmetic. Start from the tokens one user consumes per minute, then divide your limit by it.

• Tokens per request = input tokens (prompt + system message + context) plus output tokens the model generates. Both count against TPM.
• Requests per user per minute = how chatty an active user is.
• Tokens per user per minute = tokens per request × requests per user per minute.

Example: each chat turn uses ~1,500 input + ~500 output = 2,000 tokens, and an active user sends ~1.5 turns per minute → 3,000 tokens/user/minute. With a 300,000 TPM limit you can serve roughly 300,000 ÷ 3,000 = 100 concurrent active users. Do the same division against your RPM limit and take the smaller of the two answers — that is your real ceiling. Note "active" means actively sending; a product with 100 concurrent users usually has thousands of signed-in accounts.

Tiers: spending more raises your limits

Most providers run usage tiers. New accounts start with low RPM/TPM/RPD; as you spend more and your account ages, you are automatically promoted to higher tiers with far bigger ceilings — sometimes 10× or 100× the starting numbers. If you are hitting limits, the first questions are: which tier am I on, and do I qualify for the next one? Enterprise and committed-spend agreements can lift limits further or add dedicated capacity.

Techniques that stretch your effective throughput

• Batching — combine many items into fewer, larger requests to ease RPM pressure. Providers' dedicated batch APIs also run at a discount for non-urgent work.
• Streaming — streaming tokens back doesn't raise your token budget, but it shortens perceived latency and frees client resources sooner, so overlapping requests clear faster.
• Backoff and pacing — spreading calls evenly across the minute instead of firing them all at once keeps you under bursty ceilings you'd otherwise trip.
• Trimming tokens — shorter prompts, prompt caching and tight output limits directly increase how many requests fit inside a fixed TPM budget.

Handling 429s: backoff, retries and queueing

Hitting a limit is normal — the question is whether your app recovers gracefully. The standard pattern is exponential backoff with jitter: on a 429, wait a short delay, then retry; if it fails again, roughly double the delay each time (e.g. 1s, 2s, 4s, 8s) and add a small random offset so many clients don't retry in lockstep. Always respect a Retry-After header if the provider sends one — it tells you exactly how long to wait.

Hard limit vs soft / burst limit

A hard limit is an absolute ceiling — exceed it and every request is rejected until the window resets. A soft or burst limit allows short spikes above your steady rate (often via a token-bucket that refills over time), so brief bursts pass but sustained overload still gets throttled. Knowing which you face changes your strategy: burst limits reward smoothing traffic over the window; hard limits require a real queue that meters requests out at a safe rate.

• Queue non-urgent work so it drains at a controlled pace instead of hammering the API.
• Spread load evenly rather than bursting at the top of each minute.
• Cap total retries so a doomed request fails cleanly instead of looping forever.

The cost angle: limits shape architecture, not the bill

Rate limits don't cost money directly — you are never charged for a 429. But they heavily influence how you build, and those choices do have a cost. When a single key can't supply the throughput you need, teams commonly reach for a fallback chain (fail over to a second model or provider when the primary throttles), multiple keys or providers to pool capacity, and a batch API to push non-urgent jobs onto cheaper, higher-limit lanes. Each adds resilience and headroom, but also integration complexity and sometimes higher per-token pricing on the fallback. The healthiest approach is to raise your tier and trim tokens first, then add redundancy only where a hard limit genuinely blocks you.

Frequently asked questions

What is the difference between RPM and TPM?

RPM (requests per minute) caps how many API calls you can make each minute, while TPM (tokens per minute) caps how much text those calls can move. You can hit either ceiling first — many small calls exhaust RPM, while a few large-context calls exhaust TPM.

How do I turn a TPM limit into the number of users I can serve?

Estimate the tokens one user consumes per minute (tokens per request times requests per user per minute, counting both input and output), then divide your TPM limit by that figure. If a user needs 3,000 tokens/minute and your limit is 300,000 TPM, you can serve about 100 concurrent active users.

How should I handle a 429 Too Many Requests error?

Retry with exponential backoff and jitter — wait progressively longer between attempts and respect any Retry-After header the provider returns. Queue non-urgent work, spread load evenly instead of bursting, and cap total retries so a request eventually fails cleanly rather than looping forever.