How Quantization Affects LLM Hosting Cost

It's just a smaller number format for the same weights

A trained model's weights are just numbers — one float per parameter. Quantização stores those same numbers in a smaller format: fewer bits per parameter, at some cost in precision. It doesn't change the model's architecture or what it learned; it changes how many bytes each learned number takes up in GPU memory. That's the whole mechanism, and it's why the memory savings are exact arithmetic, not an estimate: weight memory = parameters × bytes per parameter.

The common formats, in bytes per parameter: FP32 (4 bytes, rarely used for inference), FP16 / BF16 (2 bytes, the common default), INT8 / FP8 (1 byte), and INT4 (0.5 bytes, via calibrated methods like GPTQ or AWQ). Each step down halves the weight footprint of the exact same model.

The GPU math: a 70B model at four precisions

Take a 70-billion-parameter model. Weight memory alone, before any KV cache or runtime overhead, at each precision:

That "fits on one card" column is where the real money is. At $1.89/hr for a rented A100 80GB, running the FP16 version means renting two cards around the clock: 2 × $1.89 × 730 hours/month ≈ $2,760/month. Quantize to INT8 and the same model fits on a single card: $1.89 × 730 ≈ $1,380/month — half the bill, from a format change alone, with no change to the model's actual knowledge. Drop to INT4 and you're still on one card, but now with 45GB free instead of 10GB — room that goes straight into KV cache, i.e. how many concurrent requests that one card can serve before queuing starts (see the VRAM & concurrency calculator for that second half of the math).

Quantization has two separate targets: weights and KV cache

It's easy to think of quantization as one setting, but it's really two independent levers that both show up in a serving stack:

• Weight quantization shrinks the fixed cost of loading the model at all — the number above that decides how many GPUs you need just to hold it.
• KV cache quantization (usually FP8 instead of FP16 for the attention cache) shrinks the per-request, per-token memory that scales with concurrent traffic and context length — it doesn't change how many GPUs you need to start, but it changes how many simultaneous users one GPU can serve once requests arrive.

Quantizing only the weights still leaves you memory-constrained on concurrency the moment real traffic shows up, because the cache grows per in-flight request regardless of weight format. Quantizing only the cache doesn't help if the weights themselves don't fit on the card to begin with. Most deployments running close to a memory ceiling end up quantizing both — which is exactly what the VRAM & concurrency calculator's separate "weight precision" and "KV cache precision" fields are modeling.

The quality trade-off is real, but not uniform

None of this is free. INT8 is close enough to lossless on most models that the accuracy difference rarely matters in production. INT4 is where the trade-off becomes real: calibrated post-training methods like GPTQ e AWQ — which pick scaling factors based on a sample of real activations rather than rounding naively — keep most models close to their FP16 benchmark scores, but "close" is not "identical," and the gap is wider on tasks that lean on precise numerical or multi-step reasoning than on general chat. Naive round-to-nearest INT4, done without that calibration step, degrades noticeably more, especially on outlier weight channels that calibrated methods are specifically designed to protect. Below INT4 — INT3, INT2 — the quality cost climbs fast enough that it's rarely worth it outside of research settings, even though the memory math looks tempting on paper.

The practical takeaway: INT8 is close to a free lunch. INT4 via GPTQ/AWQ is a real, usually-worth-it trade for most production chat and RAG workloads, but you should benchmark your own task before committing to it, not just the memory savings. Going further than INT4 is a research trade-off, not an operations one.

Where this actually moves your bill

• Crossing a GPU-count threshold. The biggest single jump isn't gradual — it's whichever quantization step takes weight memory from "needs N+1 cards" to "needs N cards." For a 70B model that threshold sits right between FP16 and INT8.
• Fine-tuning, not just serving. QLoRA — the quantized variant of LoRA fine-tuning — quantizes the frozen base model to 4-bit specifically so the training job fits on one rented GPU instead of a multi-GPU cluster. See the LoRA/QLoRA vs GPU rental calculator for that side of the math.
• Concurrency headroom, which is a cost multiplier in disguise. Freeing GPU memory via quantization doesn't just let the model fit — it raises how many concurrent requests one card serves before you need a second one, which is the real driver of $/million-tokens at any real traffic volume.

Erros comuns

• Treating "quantized" as one setting. Weight precision and KV cache precision are independent choices that both need to be set — quantizing one and forgetting the other leaves half the possible savings on the table.
• Comparing a quantized self-hosted cost to an unquantized API baseline, or vice versa. Most commercial API providers already serve at a quantized or otherwise optimized precision internally — the per-token price you're billed already reflects that. A fair self-host-vs-API comparison should quantize the self-hosted side realistically rather than assuming it has to run at FP16; see self-hosting vs API for the full comparison.
• Assuming INT4 quality loss is zero because GPTQ/AWQ "work well." They substantially close the gap to FP16, not eliminate it — validate on your own eval before shipping, especially for precise or multi-step tasks.
• Sizing GPU count off weight memory alone. The weights are the fixed entry fee; the KV cache is the term that actually determines how many concurrent users a given card serves, and it's sized separately.

Continuar aprendendo

Perguntas frequentes

How much VRAM does quantization actually save?

Weight memory is parameters times bytes per parameter: FP32 uses 4 bytes, FP16/BF16 uses 2, INT8/FP8 uses 1, and INT4 (GPTQ, AWQ) uses 0.5. A 70B model is about 280GB at FP32, 140GB at FP16, 70GB at INT8, and 35GB at INT4 — the same weights, a 4x spread in footprint, purely from the number format.

Does quantization hurt model quality?

INT8 is close to lossless for most models — the accuracy drop is small enough to ignore in most production use. INT4 with a calibrated method like GPTQ or AWQ keeps quality close to the FP16 original on most benchmarks, but it is a real, measurable degradation, not zero — naive round-to-nearest INT4 without calibration degrades noticeably more. Below INT4 (INT3, INT2) the quality cost usually stops being worth the memory savings for most workloads.

Why does the same model sometimes need one GPU and sometimes two?

Because GPU memory comes in fixed sizes — 24GB, 48GB, 80GB cards — and weight memory either fits under that ceiling or it doesn't. A 70B model at FP16 (140GB) can't fit on one 80GB card, so it needs two. The same model at INT8 (70GB) fits on one. Quantization doesn't just save money gradually — it can drop you below a GPU-count threshold entirely, which is where the biggest cost jumps happen.

Should I quantize the weights, the KV cache, or both?

They're two separate levers that stack. Weight quantization shrinks the fixed cost of loading the model at all. KV cache quantization (commonly FP8 instead of FP16) shrinks the per-request, per-token memory that scales with concurrent traffic and context length. Quantizing only the weights still leaves you memory-constrained on concurrency under load; quantizing only the KV cache doesn't help if the weights alone don't fit. Most self-hosted deployments running near a GPU memory ceiling quantize both.