Mixture of Experts (MoE) Explained: How Sparse LLMs Work, Why They Spread and What They Cost to Run
An accessible guide to Mixture of Experts, the sparse architecture behind DeepSeek-V3, Llama 4, Mixtral, gpt-oss and Gemini 1.5: how routing works, total versus active parameters, load balancing, the papers that shaped it, common myths and practical trade-offs for serving and fine-tuning.
Mixture of Experts (MoE) has become one of the most important ideas in large language models. Instead of running every parameter for every word, an MoE model activates only a small part of itself for each token. That lets developers build models with far more total parameters without a matching rise in compute per token. Many prominent open-weight models now use it, including DeepSeek-V3, Meta’s Llama 4, Mistral’s Mixtral and OpenAI’s gpt-oss, and Google has said its Gemini 1.5 models are MoE. Most closed labs do not disclose their architectures, so claims that a particular proprietary model uses a specific number of experts are usually speculation.
How MoE works
A standard transformer layer contains an attention block and a feed-forward network. In an MoE model, the single feed-forward network is replaced by several parallel “experts”, each a smaller feed-forward network. A small router (or gating network) looks at each token and picks which experts should process it, typically the top one or two out of eight, or a handful out of dozens or more in newer designs. The outputs of the chosen experts are combined, weighted by the router’s scores.
The result is a sparse model: only a fraction of its parameters do work for any given token. A dense model, by contrast, uses all of its parameters every time.
Notable MoE models
| Model | Total parameters | Active per token | Experts |
|---|---|---|---|
| Mixtral 8x7B (Mistral AI) | About 47B | About 13B | 8 per layer, 2 active |
| DeepSeek-V3 | 671B | About 37B | Fine-grained routed experts plus shared experts |
| Llama 4 Scout (Meta) | 109B | 17B | 16 |
| Llama 4 Maverick (Meta) | 400B | 17B | 128, alternating dense and MoE layers |
| gpt-oss-120b (OpenAI) | 117B | 5.1B | MoE, fits on a single 80GB GPU |
Sources: the Mixtral paper, DeepSeek-V3 technical report, Meta’s Llama 4 announcement and the gpt-oss-120b model card. Google’s Gemini 1.5 technical report describes Gemini 1.5 Pro as a mixture-of-experts model without listing expert counts.
Why MoE spread
The appeal is efficiency. Because compute per token depends on active rather than total parameters, an MoE model can hold much more knowledge for a similar training and inference cost per token. The Switch Transformer paper reported up to a sevenfold increase in pre-training speed over comparable dense T5 models using the same computational resources. DeepSeek-V3 showed that a 671B-parameter model with about 37B active parameters could compete with leading models, and its technical report attributes much of its training efficiency to the MoE design.
The trade-off is memory. Every expert must be loaded even if only a few are used per token, so an MoE model needs memory for all its parameters. It also adds engineering complexity in routing, communication between GPUs and training stability.
A common myth: the “maths expert”
MoE is often explained with an analogy: a company routing legal questions to the legal team and maths questions to the maths team. It is a useful picture of sparse routing, but it overstates how experts actually specialise. When Mistral’s researchers analysed Mixtral’s routing, they did not observe obvious patterns of experts being assigned by topic, such as biology versus philosophy. The routing appeared to follow syntax and token-level patterns more than subject matter. Experts do specialise, but usually not in human-readable domains.
The routing problem
The central engineering challenge is load balancing. If the router keeps sending most tokens to a few popular experts, the others are underused, capacity is wasted and some GPUs become bottlenecks. In extreme cases, a few experts end up doing almost all the work, a failure known as expert collapse.
Early systems such as the Switch Transformer added an auxiliary loss that penalises imbalanced routing. Too much balancing pressure, however, can blur the specialisation that makes MoE useful. DeepSeek-V3 introduced an auxiliary-loss-free strategy that adjusts a bias term for each expert to keep load balanced without that extra loss, and it combined many small “fine-grained” experts with a few always-active “shared” experts that capture common knowledge.
Key papers behind the architecture
The modern MoE story can be traced through a handful of primary sources:
- Shazeer et al., 2017 — “Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer” (arXiv:1701.06538). Introduced a trainable gating network that activates only a few experts per token, showing that model capacity can grow far faster than compute per token.
- Lepikhin et al., 2020 — GShard (arXiv:2006.16668). Scaled MoE Transformers across thousands of accelerators for machine translation and introduced practical sharding techniques.
- Fedus, Zoph and Shazeer, 2021 — Switch Transformers (arXiv:2101.03961). Simplified routing to a single expert per token (top-1), with an auxiliary load-balancing loss, and reported large pre-training speedups over dense baselines at equal compute.
- Jiang et al., 2024 — Mixtral of Experts (arXiv:2401.04088). Mistral AI’s open-weight Mixtral 8x7B uses 8 experts per layer with 2 active per token: about 47B total parameters but roughly 13B active per token.
- DeepSeek-V3 Technical Report (arXiv:2412.19437). Describes a 671B-parameter MoE model with about 37B parameters activated per token, using fine-grained and shared experts plus an auxiliary-loss-free load-balancing strategy.
Total vs. active parameters: how to read model cards
MoE model cards usually list two numbers. Total parameters determine how much memory you need to hold the model, because every expert must be loaded. Active parameters determine compute per token and therefore much of the inference cost and speed. A model with 47B total and 13B active parameters runs with roughly the per-token compute of a 13B dense model but still needs memory for all 47B. That is why MoE models are attractive for high-throughput serving but can be awkward on memory-constrained hardware.
Total versus active parameters: how to read model cards
MoE model cards usually list two numbers. Total parameters determine how much memory you need, because every expert must be loaded. Active parameters determine compute per token and therefore much of the inference speed and cost. A model with 47B total and 13B active parameters runs with roughly the per-token compute of a 13B dense model but still needs memory for all 47B. That is why MoE suits high-throughput serving on large GPUs but can be awkward on memory-constrained hardware such as laptops.
| Question | Look at | Why |
|---|---|---|
| Will it fit on my GPU? | Total parameters and quantisation level | All experts must be in memory |
| How fast and cheap per token? | Active parameters | Compute scales with active weights |
| How good is it? | Benchmarks and your own tests | Neither number alone predicts quality |
Practical trade-offs for teams
- Serving: MoE benefits from expert parallelism across GPUs. Frameworks such as vLLM and SGLang support popular open MoE models, but throughput depends heavily on batch size and communication between devices.
- Fine-tuning: Load imbalance and expert collapse can reappear during fine-tuning, so many teams use parameter-efficient methods and monitor how often each expert is used.
- Quantisation: Because memory is the bottleneck, MoE models are frequently quantised for local use, at some cost in quality.
- Latency: With small batches, MoE models may not deliver the full speed advantage their active parameter count suggests, because loading many experts’ weights still takes memory bandwidth.
Where research is heading
Researchers are exploring several directions. Adaptive computation lets a model vary how many experts it uses depending on how hard a token or task is. Finer-grained experts, as in DeepSeek’s designs, trade a few large experts for many small ones to improve specialisation. Soft or differentiable routing blends expert outputs rather than choosing discrete experts, avoiding some load-balancing problems. And MoE is being combined with other efficiency techniques, such as attention variants that reduce memory use for long contexts. Which ideas reach production models depends on whether they hold up at scale and on serving hardware.
Our analysis
Mixture of Experts is best understood as a way to decouple a model’s knowledge capacity from its per-token compute cost. It has become standard in many leading open-weight models because it makes very large models trainable and affordable to serve, but it shifts the burden to memory, routing and engineering. For teams choosing or deploying models, the key habit is reading total and active parameters separately and treating claims about undisclosed proprietary architectures with scepticism.