Weights are the easy part. Fitting the model is a one-line calculation and it tells you almost nothing. What decides whether you can serve eight users or eighty is the KV cache — memory that scales with context length and with every concurrent sequence in flight. This tool sizes both, then converts the resulting capacity into a cost per million tokens you can hold against an API bill.
Concurrency and unit cost by context length
Same model, same GPU, same money — only the context window each request carries changes. This is the table that turns "we'll just raise max_model_len" into a capacity decision.
| Context | KV / request | Concurrent | Tokens/sec est. | $ / 1M tokens |
|---|
The utilisation cliff
A GPU bills by the hour, not by the token. Your true unit cost is the headline figure divided by your duty cycle — this is where most self-hosting business cases quietly fall apart.
| Average utilisation | Effective tokens/sec | $ / 1M tokens | vs API |
|---|
Fitting the model is not the same as serving it
The advice you find everywhere is that a 70B model at INT4 needs about 35GB, so it fits on a single 80GB card with room to spare. That is true and it is nearly useless, because it describes a model sitting idle. The moment requests arrive, every sequence in flight allocates its own KV cache: two tensors per layer, sized by KV heads times head dimension, held for every token in that sequence's context. At 80 layers, 8 KV heads, 128 dimensions and FP16 that is 327,680 bytes per token — a third of a megabyte — which at an 8K context is about 2.7GB per concurrent request. The 45GB of headroom that looked so comfortable is roughly a dozen users, and the thirteenth gets queued. Capacity planning for inference is KV cache planning; the weights just set the entry fee.
The KV head count is the number that actually moves
Grouped-query attention is the reason modern open-weight models are servable at all, and it is routinely missed by anyone sizing hardware from a parameter count. A model with 64 attention heads and 8 KV heads stores an eighth of the cache of an equivalent multi-head design, which converts almost linearly into eight times the concurrency on the same card. Try the 13B multi-head preset above against the 70B preset: the smaller model is often the one that runs out of memory first, which is not intuitive until you see where the bytes go. Halving the KV precision to FP8 doubles concurrency again for a modest accuracy cost. These two switches move capacity far more than buying a bigger card does, and they cost nothing.
Utilisation decides the economics, not the hardware
The final table is the one that ends most self-hosting projects honestly. A rented H100 costs the same $2.50 an hour at 3am with no traffic as it does at peak, so your real cost per million tokens is the headline number divided by your average duty cycle. Serve at 20% utilisation and you are paying five times what the calculator's optimistic figure suggests — and against commodity API rates for a small model, that is not close. Self-hosting wins on sustained, saturated, predictable volume, on workloads with data-residency or latency constraints an API cannot meet, and on models nobody sells per-token. It loses on bursty traffic, and it loses quietly, because the invoice is flat and the wasted capacity never appears as a line item. Cross-check the conclusion with the self-hosted LLM vs API calculator, price the hardware itself with the GPU inference cost calculator, and if the answer is marginal, look at a distilled smaller model or prompt caching before you buy a rack.