How Mixture-of-Experts (MoE) Affects LLM Cost

A dense model uses every parameter, every token. MoE doesn't.

In a standard ("dense") transformer, every parameter takes part in computing every token — a 70B dense model does roughly 70B parameters' worth of math for each token it generates, no matter how simple or complex that token is. A Mixture-of-Experts model replaces each feed-forward block with a bank of smaller sub-networks ("experts") plus a small router network. For every token, the router picks a handful of experts — commonly 2 out of 8, or 8 out of 128 or 256 — and only those selected experts do any computation. The rest of the experts sit in memory, untouched, for that token.

That gives every MoE model two different parameter counts that matter for cost: total parameters (everything that has to be loaded into memory, because any expert might get picked) and active parameters (what actually gets computed for a given token, because only the router's chosen experts run). The gap between the two is the entire reason MoE exists: it lets a model carry far more learned knowledge in its total weights while keeping the compute cost of each token pinned to the much smaller active count.

The real numbers: three production MoE models

These aren't estimates — all three are documented in their technical reports or model cards:

The pattern holds across all three: as total parameter count grows, the active share shrinks even further. DeepSeek-V3 carries 14x more total weight than Mixtral 8x7B, but its active compute per token is only about 3x higher — a direct result of having far more experts to choose from, and choosing a smaller fraction of them.

Why this makes MoE cheap to compute but still expensive to host

Inference compute (FLOPs) per token scales with aktiv parameters — roughly 2 × active params per token for a forward pass, the same approximation used for dense models, just applied to the smaller active subset. That's the number that drives GPU-hour cost per request: DeepSeek-V3's 37B active parameters mean each token costs roughly what a ~37B dense model's token would cost to compute, not what a 671B dense model would cost.

GPU memory, on the other hand, scales with total parameters — every one of the 671B weights has to sit resident in GPU memory at all times, because the router can send any token to any expert and the system can't predict in advance which ones. There's no way to page experts in and out fast enough to keep up with per-token routing decisions in practice. So a provider serving DeepSeek-V3 needs enough combined GPU memory for the full 671B at whatever precision they serve it at (see how quantization affects that number) — the same multi-GPU memory bill a 671B dense model would carry — while paying the much smaller compute bill of a 37B active model.

That split is the whole MoE cost story in one line: cheap to compute, expensive to hold in memory. It's the same compute-vs-memory duality that shows up in VRAM and concurrency math for dense models — MoE just moves the two numbers further apart.

Why this still ends up cheap per token on an API

A large, mostly-idle memory footprint is a fixed cost: it exists whether one request or ten thousand are in flight. The way to make that fixed cost cheap pro Token is to spread it across as many tokens as possible — which is exactly what high-volume API providers do by batching large numbers of concurrent requests onto the same loaded cluster. The GPU-hours spent just holding 671B parameters in memory get divided across every token every customer sends that hour, not just one customer's traffic. That's a big part of why a 671B-parameter model can be priced at a fraction of a cent per thousand tokens on a busy provider's API, despite needing a GPU memory footprint that would be unaffordable for a single low-traffic self-hosted deployment to run economically — see self-hosting vs API for that trade-off in full.

It also explains why MoE architectures only make economic sense at scale. A provider with little traffic gets none of the batching benefit, so they're stuck paying the full memory cost of a 671B-class cluster to serve a trickle of 37B-active-equivalent compute — the worst of both numbers. That's a specific reason self-hosting a large MoE model rarely pencils out below very high request volume, independent of the general self-host-vs-API calculation.

Häufige Fehler

• Reading "671B parameters" as the compute cost. It's the memory cost. The compute cost — the number that scales with GPU-hours per request — is the active parameter count, which can be an order of magnitude smaller.
• Assuming a smaller active count means the model is comparable to a same-sized dense model in capability. It isn't a fair comparison in the model's favor, either — MoE models are generally more capable than a dense model with the same active parameter count, not less, because the router draws on a much larger pool of specialized total knowledge. The active count is a compute cost figure, not a capability ceiling.
• Expecting MoE to reduce GPU memory requirements. It doesn't — a 671B MoE model needs the same memory as a 671B dense model would. MoE's savings are entirely on the compute side.
• Comparing API prices across providers as if the architecture were irrelevant. Two models with similar benchmark scores can have very different active-parameter counts, and that's a big driver of why one is priced several times higher than the other per token — check the model's technical report, not just its total parameter headline, before assuming a price gap reflects a quality gap.

Lernen Sie weiter

Häufig gestellte Fragen

Does a bigger MoE model cost more per token than a smaller dense model?

Not necessarily. Compute cost per token scales with active parameters, not total parameters. DeepSeek-V3 has 671B total parameters but only 37B active per token — less active compute than a 70B dense model, even though it is roughly 10x larger in total size. That is why MoE models can be priced cheaply per token despite being the largest models by parameter count.

If only a fraction of the model is used, why does MoE still need huge GPU memory?

Because which experts activate changes with every token, and the router can pick any of them. A provider can't predict in advance which experts the next token will need, so all experts for every layer have to stay loaded in GPU memory at all times, ready to be selected. Memory footprint scales with total parameters, exactly like a dense model of that total size — only the compute (FLOPs) scales down with active parameters.

Why do MoE models need high request volume to be cost-efficient to host?

The GPU memory holding all the experts is a large fixed cost that doesn't change whether one user or one thousand are being served. Batching many concurrent requests together spreads that fixed memory and GPU-hour cost across far more tokens, which is why MoE serving economics favor high-volume providers batching thousands of requests over a single low-traffic self-hosted deployment.

Is a 37B-active MoE model as good as a 37B dense model?

No — it's generally better, which is the entire point of the architecture. The router can draw on hundreds of billions of total parameters' worth of specialized knowledge, picking a different subset of experts for different tokens, while paying the compute cost of only the smaller active subset. A 37B dense model has no such reservoir — its 37B parameters are all it has, for every token, with nothing to route between.