AI & API Cost Glossary

How to use this glossary

API and AI pricing pages are full of jargon that hides what you'll actually pay. Below are 40 of the most common terms, grouped into LLM pricing & tokens, infrastructure & self-hosting, and billing models. Each definition is vendor-neutral and focused on the cost angle. New to LLM billing? Start with our how LLM API pricing works guide, then estimate a real number with the token cost calculator.

LLM pricing & tokens

Token

The small unit of text an LLM reads and writes — in English about ¾ of a word, or roughly 4 characters. Billing is per token, so token count (not the number of requests) drives your LLM bill.

Input tokens

The tokens you send to the model: your prompt, system instructions, conversation history and any attached context. They are usually the cheaper of the two rates but multiply fast when you resend context on every call.

Output tokens

The tokens the model generates in its reply. Output is almost always priced higher than input — commonly 3–5× — because generating text costs more compute than reading it.

Context window

The maximum number of tokens a model can consider at once (e.g. 128K or 1M). It is a capacity ceiling, not free space: every token you place in the window is billed on each call.

Cost per million tokens

The standard unit LLM prices are quoted in ($/1M tokens), listed separately for input and output. Multiplying your monthly token volume by these rates is the fastest way to estimate spend.

Prompt caching

Storing a repeated prompt prefix — a long system message or document — so it is not reprocessed each call. Cached input is billed at a steep discount (often 75–90% off), lowering cost for repetitive workloads.

Reasoning tokens

The hidden "thinking" tokens some models generate before their visible answer. You don't see them, but they are billed as output — a major and easily overlooked cost driver on reasoning models.

Multimodal tokens

Images, audio and video are converted into token-equivalents for billing. A single high-resolution image can cost hundreds or thousands of input tokens, so multimodal prompts add up faster than text alone.

Blended rate

A single average price per token that combines input and output rates according to your typical mix. Useful for quick estimates, though it hides that output is the expensive half of the bill.

Batch API

An asynchronous endpoint that processes large jobs within a delay window (often up to 24 hours) in exchange for a discount, commonly around 50%. Ideal for non-urgent work like bulk classification or embeddings.

Embeddings

Numeric vectors that represent the meaning of text, images or other data so they can be searched by similarity. They are cheap to generate per token and underpin semantic search and RAG.

Fine-tuning

Further training a base model on your own examples to specialize its behavior. It adds an upfront training cost and, on hosted models, often a higher per-token inference rate than the base model.

RAG (Retrieval-Augmented Generation)

A pattern that fetches only the relevant snippets from a knowledge base and adds them to the prompt, instead of stuffing whole documents into context. It trims input tokens while keeping answers grounded in your data.

Model tier (flagship / mid / nano)

Providers offer families of models at different capability and price levels — flagship (most capable, most expensive), mid-range (balanced), and nano/small (cheapest, fastest). Matching the tier to task difficulty is the biggest lever on cost.

Temperature

A setting from 0 to about 2 that controls randomness in the output. It affects answer style, not price directly, but higher values can produce longer or more varied responses that shift output-token cost.

Function / tool calling

Letting the model return a structured request to run one of your defined tools or APIs. The tool definitions are sent as input tokens on each call, and multi-step tool loops multiply total token usage.

Streaming

Delivering the model's output token by token as it is generated rather than in one block. It improves perceived speed and lowers time-to-first-token but does not change the total token price.

Distillation

Training a smaller "student" model to mimic a larger "teacher" model. The result runs far cheaper per token while retaining much of the quality on a targeted set of tasks.

Quantization

Reducing the numerical precision of a model's weights (e.g. from 16-bit to 4-bit) so it needs less memory and runs faster. For self-hosters this cuts GPU cost, usually with a small quality trade-off.

Inference

Running a trained model to produce an output — the act you pay for on every API call. Inference cost scales with tokens on hosted APIs and with compute time on self-hosted GPUs.

Infrastructure & self-hosting

Rate limit (RPM / TPM)

The ceilings a provider enforces per account: RPM is requests per minute, TPM is tokens per minute. Exceeding either returns a 429 error, so scale requires batching, queuing or a higher tier.

Concurrency limit

The maximum number of requests allowed in flight at the same time. It caps how many parallel jobs you can run and, on self-hosted or provisioned setups, directly sizes the hardware you must pay for.

TTFT / latency

TTFT (time-to-first-token) is how long before the first output token arrives; latency is total response time. They shape user experience and, on rented GPUs, longer latency means more billed compute per request.

Throughput

How many tokens or requests a system processes per second. Higher throughput serves more users on the same hardware, lowering the cost per request when you self-host or reserve capacity.

Vector database

A store optimized for holding embeddings and searching them by similarity. It powers RAG and semantic search, and is usually priced by stored vectors, dimensions and queries rather than by token.

GPU / VRAM

The graphics processor and its video memory that run model inference. A model must fit in VRAM to load, so larger models demand pricier, higher-memory GPUs — the core cost of self-hosting.

Self-hosting

Running an open-weight model on your own or rented hardware instead of calling a hosted API. It removes per-token fees but adds fixed GPU, engineering and idle-capacity costs that only pay off at high volume.

Egress / bandwidth

The charge for data leaving a cloud provider's network. Easy to forget in AI budgets, egress fees can be significant when moving large datasets, media or model weights between services or regions.

Cold start (serverless)

The delay when a serverless function or model container spins up from idle before it can respond. It adds latency and, for large models with long load times, can push you toward always-warm capacity that costs more.

Provisioned throughput

Reserving a fixed, guaranteed amount of model capacity for a flat hourly or monthly fee instead of paying per token. It stabilizes cost and latency for steady high-volume traffic but wastes money when idle.

Spot vs on-demand

On-demand instances are available anytime at full price; spot (preemptible) instances use spare capacity at a deep discount but can be reclaimed with little notice. Spot suits interruptible batch jobs, not latency-critical serving.

Committed-use / reserved capacity

A discount in exchange for committing to a minimum spend or specific hardware over one to three years. It lowers the effective rate for predictable workloads but locks you in.

SLA (uptime)

A service-level agreement is the provider's contractual promise of availability (e.g. 99.9%) and the credits owed if they miss it. Higher SLA tiers usually cost more but matter for production reliability.

Billing & pricing models

Per-seat vs usage-based pricing

Per-seat charges a flat fee for each user regardless of activity; usage-based charges for what you actually consume (tokens, calls, compute). Seats are predictable; usage scales with demand and can spike unexpectedly.

Credits

Prepaid balance you buy and draw down as you use a service. Credits smooth billing but can expire, and their per-unit value sometimes obscures the real per-token or per-call price.

Overage

Usage beyond your plan's included allowance, billed at a separate — often higher — rate. Overage charges are a common source of surprise bills when traffic exceeds the tier you signed up for.

Free tier

A no-cost allowance (a monthly credit, limited calls or a trial window) to let you evaluate a service. It's useful for testing but rarely enough for production, and limits or expiry dates apply.

TCO (total cost of ownership)

The full cost of running a solution — not just API or GPU fees but engineering, monitoring, storage, egress and overhead. Comparing hosted vs self-hosted only makes sense on TCO, not headline rates.

Cost per user

Your total AI or API spend divided by active users. It turns raw usage into a unit economic you can compare against revenue or subscription price to check whether a feature is profitable.

Gateway / router markup

The fee an AI gateway or model router adds on top of the underlying provider's price for routing, fallback, analytics or unified billing. Convenient, but the markup can meaningfully raise your effective per-token cost.

Frequently asked questions

What is a token?

A token is the small chunk of text an LLM reads and generates — in English roughly ¾ of a word, or about 4 characters. Providers bill per token for both the text you send (input) and the text the model produces (output), which is why token count, not request count, drives an LLM bill.

What is prompt caching?

Prompt caching lets a provider store a repeated prefix — such as a long system prompt or document — so it is not reprocessed on every call. Cached input tokens are billed at a large discount (often 75–90% off), cutting cost for workloads that resend the same context.

What do TPM and RPM mean in a rate limit?

RPM is requests per minute and TPM is tokens per minute — the two ceilings a provider enforces on your account. Hitting either returns a 429 error, so high-volume apps must batch, queue, or request a higher tier rather than assume unlimited throughput.