SEC.03 / LLM Research
Mixture of Experts (MoE) Explained: How Sparse LLMs Work, Why They Spread and What They Cost to Run
An accessible guide to Mixture of Experts, the sparse architecture behind DeepSeek-V3, Llama 4, Mixtral, gpt-oss and Gemini 1.5: how routing works, total versus active parameters, load balancing, the papers that shaped it, common myths and practical trade-offs for serving and fine-tuning.