Your hardware

Chat
Coding
Reasoning
✓
—
models fit in VRAM
◐
—
fit with RAM offload
🏆
—
biggest model that fits

Estimated speed by model

Tokens/sec, sorted fastest first. Only models that run at all are shown.

Fits in VRAMNeeds RAM offload (slower)
WeightsKV cache + overhead
ModelParamsFitMemoryEst. tok/sLicense
⚠️ Tokens/sec is a memory-bandwidth roofline estimate (not a calibrated benchmark) — real llama.cpp/vLLM throughput varies with backend, batch size and driver. VRAM figures use decimal GB (1 GB = 10⁹ bytes). Apple Silicon and other unified-memory platforms share one memory pool between CPU and GPU — the GPU VRAM figure already represents your whole machine's memory, so the system-RAM field is disabled for those. Architecture data (layers, KV heads, head dimension) is read from each model's published config.json, not estimated from parameter count — that shortcut is exactly what produces wrong answers elsewhere.

How this calculator works

Required memory is three terms: model weights (parameters × bytes per weight at your quantization), the KV cache for your context length (2 × layers × KV heads × head dimension × bytes, scaled by context — DeepSeek's MLA attention uses a different, much smaller, compressed-latent formula instead), and about 0.6GB per GPU of runtime overhead. If that fits inside your VRAM, you get the fast path; if it only fits once system RAM is added, llama.cpp-style layer offloading kicks in and speed drops because the CPU side is bandwidth-starved compared to the GPU. Speed itself is a roofline estimate: bytes read per generated token (the active parameters, not total — this is why mixture-of-experts models punch above their size) divided by effective memory bandwidth, plus a small fixed per-token cost. See the VRAM & concurrency calculator for the server-side version of this math (sizing for many simultaneous users instead of one).

The CPU matters more than people assume once any layer runs off RAM — either because you picked "CPU only" or because the model spilled past your VRAM. RAM bandwidth is a theoretical ceiling; a weak or old CPU can't issue enough parallel memory reads to actually reach it, so the "CPU" selector above derates your RAM bandwidth to a realistic figure: ~85% for a modern 8+ core chip with AVX2/AVX-512, ~60% for a typical 4-8 core laptop CPU, ~35% for older or budget chips. If you only have a CPU and no graphics card, pick "CPU only" at the top of the GPU list — the whole model then runs out of system RAM, same math, no VRAM term at all.

Quantization: the one setting that matters most

Quantization shrinks a model's weights from their training precision down to fewer bits each, trading a little quality for a lot less memory. Going from 16-bit to 4-bit roughly quarters the VRAM a model needs — which is usually the difference between "doesn't fit" and "fits with room to spare" on consumer hardware.

PrecisionBytes/paramQuality vs fullTypical use
4-bit (Q4_K_M)0.5Small, usually unnoticeable in chatDefault for most local setups
5/6-bit (Q5/Q6_K)0.6–0.75Very close to full precisionWhen you have VRAM to spare
8-bit (Q8_0)1NegligibleQuality-sensitive tasks, coding
16-bit (full)2Reference / training precisionFine-tuning, research

Why bandwidth — not VRAM — sets your speed

Once a model fits, every token it generates means re-reading its active weights from memory — so tokens/sec is bounded by how fast your GPU can move bytes, not by how much it can store. This is why two cards with the same VRAM can run the same model at very different speeds:

CardVRAMBandwidth
RTX 4060 Ti 16GB16 GB288 GB/s
RTX 409024 GB1,008 GB/s
RTX 509032 GB1,792 GB/s
H100 SXM80 GB3,350 GB/s

Full list of 40+ GPUs with verified bandwidth is built into the calculator above — pick yours from the dropdown to see the real number, not a guess.

Learn & compare
VRAM & concurrency calculator →Self-hosted vs API →Compare 400+ models →
Share: 𝕏 Post Reddit

Hardware too small? Rent cloud GPU instead

DigitalOcean $200Hostinger

FAQ

How do you calculate whether a model fits on my GPU?

Required memory = model weights (parameters × bytes per weight at your chosen quantization) + the KV cache for your context length (computed from the model's real architecture — layers, KV heads, head dimension — not guessed from parameter count) + about 0.6GB per GPU of runtime overhead. If that fits in your VRAM alone it's fast; if it only fits once you add system RAM, llama.cpp-style layers spill to the CPU and speed drops a lot; past that, it simply doesn't run.

Why does the same model show different tokens/sec on different GPUs with the same VRAM?

Token generation is memory-bandwidth bound, not compute bound, for single-sequence local inference. An RTX 4090 (1,008 GB/s) and an RTX 5080 (960 GB/s) have similar VRAM but very different bandwidth, and bandwidth — not VRAM size — sets your tokens/sec ceiling once the model fits.

Why is a 671B mixture-of-experts model sometimes faster than a 70B dense model?

A MoE model only reads its active parameters for each token, not the full parameter count — DeepSeek-V3/R1 has 671B total parameters but only 37B active, so it reads roughly half the bytes per token that a 70B dense model does. Total size sets whether it fits; active size sets how fast it generates.

Are these tokens/sec numbers exact?

No — they are a bandwidth-based roofline estimate, not a calibrated benchmark. Real llama.cpp/vLLM throughput depends on your exact backend, batch size and driver version too. Treat the number as a reasonable estimate for comparing models and hardware, not a guaranteed speed.

Does my CPU matter if I have a good GPU?

Yes, whenever any part of the model spills into system RAM — either because you picked "CPU only" with no dedicated GPU, or because the model didn't fully fit in VRAM. RAM bandwidth is a theoretical ceiling; a weak CPU can't issue enough parallel memory reads to reach it, so a modern 8+ core chip gets noticeably more real tokens/sec from the same RAM than a 4-core laptop CPU does. If the whole model fits in VRAM alone, the CPU barely matters.