Your hardware
Estimated speed by model
Tokens/sec, sorted fastest first. Only models that run at all are shown.
| Model | Params | Fit | Memory | Est. tok/s | License |
|---|
How this calculator works
Required memory is three terms: model weights (parameters × bytes per weight at your quantization), the KV cache for your context length (2 × layers × KV heads × head dimension × bytes, scaled by context — DeepSeek's MLA attention uses a different, much smaller, compressed-latent formula instead), and about 0.6GB per GPU of runtime overhead. If that fits inside your VRAM, you get the fast path; if it only fits once system RAM is added, llama.cpp-style layer offloading kicks in and speed drops because the CPU side is bandwidth-starved compared to the GPU. Speed itself is a roofline estimate: bytes read per generated token (the active parameters, not total — this is why mixture-of-experts models punch above their size) divided by effective memory bandwidth, plus a small fixed per-token cost. See the VRAM & concurrency calculator for the server-side version of this math (sizing for many simultaneous users instead of one).
The CPU matters more than people assume once any layer runs off RAM — either because you picked "CPU only" or because the model spilled past your VRAM. RAM bandwidth is a theoretical ceiling; a weak or old CPU can't issue enough parallel memory reads to actually reach it, so the "CPU" selector above derates your RAM bandwidth to a realistic figure: ~85% for a modern 8+ core chip with AVX2/AVX-512, ~60% for a typical 4-8 core laptop CPU, ~35% for older or budget chips. If you only have a CPU and no graphics card, pick "CPU only" at the top of the GPU list — the whole model then runs out of system RAM, same math, no VRAM term at all.
Quantization: the one setting that matters most
Quantization shrinks a model's weights from their training precision down to fewer bits each, trading a little quality for a lot less memory. Going from 16-bit to 4-bit roughly quarters the VRAM a model needs — which is usually the difference between "doesn't fit" and "fits with room to spare" on consumer hardware.
| Precision | Bytes/param | Quality vs full | Typical use |
|---|---|---|---|
| 4-bit (Q4_K_M) | 0.5 | Small, usually unnoticeable in chat | Default for most local setups |
| 5/6-bit (Q5/Q6_K) | 0.6–0.75 | Very close to full precision | When you have VRAM to spare |
| 8-bit (Q8_0) | 1 | Negligible | Quality-sensitive tasks, coding |
| 16-bit (full) | 2 | Reference / training precision | Fine-tuning, research |
Why bandwidth — not VRAM — sets your speed
Once a model fits, every token it generates means re-reading its active weights from memory — so tokens/sec is bounded by how fast your GPU can move bytes, not by how much it can store. This is why two cards with the same VRAM can run the same model at very different speeds:
| Card | VRAM | Bandwidth |
|---|---|---|
| RTX 4060 Ti 16GB | 16 GB | 288 GB/s |
| RTX 4090 | 24 GB | 1,008 GB/s |
| RTX 5090 | 32 GB | 1,792 GB/s |
| H100 SXM | 80 GB | 3,350 GB/s |
Full list of 40+ GPUs with verified bandwidth is built into the calculator above — pick yours from the dropdown to see the real number, not a guess.
📚 New here? Learn how to choose an LLM →
Hardware too small? Rent cloud GPU instead
FAQ
How do you calculate whether a model fits on my GPU?
Required memory = model weights (parameters × bytes per weight at your chosen quantization) + the KV cache for your context length (computed from the model's real architecture — layers, KV heads, head dimension — not guessed from parameter count) + about 0.6GB per GPU of runtime overhead. If that fits in your VRAM alone it's fast; if it only fits once you add system RAM, llama.cpp-style layers spill to the CPU and speed drops a lot; past that, it simply doesn't run.
Why does the same model show different tokens/sec on different GPUs with the same VRAM?
Token generation is memory-bandwidth bound, not compute bound, for single-sequence local inference. An RTX 4090 (1,008 GB/s) and an RTX 5080 (960 GB/s) have similar VRAM but very different bandwidth, and bandwidth — not VRAM size — sets your tokens/sec ceiling once the model fits.
Why is a 671B mixture-of-experts model sometimes faster than a 70B dense model?
A MoE model only reads its active parameters for each token, not the full parameter count — DeepSeek-V3/R1 has 671B total parameters but only 37B active, so it reads roughly half the bytes per token that a 70B dense model does. Total size sets whether it fits; active size sets how fast it generates.
Are these tokens/sec numbers exact?
No — they are a bandwidth-based roofline estimate, not a calibrated benchmark. Real llama.cpp/vLLM throughput depends on your exact backend, batch size and driver version too. Treat the number as a reasonable estimate for comparing models and hardware, not a guaranteed speed.
Does my CPU matter if I have a good GPU?
Yes, whenever any part of the model spills into system RAM — either because you picked "CPU only" with no dedicated GPU, or because the model didn't fully fit in VRAM. RAM bandwidth is a theoretical ceiling; a weak CPU can't issue enough parallel memory reads to reach it, so a modern 8+ core chip gets noticeably more real tokens/sec from the same RAM than a 4-core laptop CPU does. If the whole model fits in VRAM alone, the CPU barely matters.