How this calculator works
The GPU Inference Cost Calculator converts three inputs — your GPU price per hour, model throughput in tokens per second, and a realistic utilisation percentage — into the true cost per million tokens for a self-hosted model. The core idea is simple: an hour of GPU time buys a fixed number of tokens, so dividing hourly cost by the tokens actually produced gives the per-token price. Throughput sets how many tokens the hardware generates each second, while utilisation discounts that figure to reflect idle time, batching gaps, and traffic that rarely keeps the GPU fully loaded.
The key trade-off is utilisation. Benchmark throughput assumes a constantly busy GPU, but real workloads sit idle between requests, so a rented GPU billed hourly can cost far more per token than the headline number suggests. Before committing to self-hosting, estimate your true utilisation honestly and compare the result against per-token API pricing — low, spiky traffic often favors an API, while steady high volume rewards owning the hardware.
Frequently asked questions
How do I calculate cost per token on my own GPU?
Take the GPU hourly price and divide by how many tokens it actually serves per hour. That is throughput (tokens/sec) times 3600, scaled by your real utilisation. A GPU billed continuously but half-idle serves half the tokens, so its cost per token doubles versus the datasheet throughput.
Why is my self-hosted cost higher than expected?
Almost always utilisation and batching. Benchmark throughput assumes full, batched load; real traffic is bursty, so the GPU sits idle between requests while the meter runs. Continuous batching, right-sizing the GPU, and consolidating traffic onto fewer, busier GPUs are the main ways to bring the per-token cost down.