> ## Content Index > Fetch the complete content index at: https://bullquill.com/llms.txt > Use this file to discover other available public pages before exploring further. # What is mixture of experts? Total vs active parameters explained - URL: https://bullquill.com/mixture-of-experts-moe-explained/ - Published: 2026-10-27T10:15:22.000Z - Updated: 2026-10-27T10:25:36.000Z - Description: A learned router sends every token to only a few experts, which is why MoE models quote two sizes. See how routing, load balancing and memory work, and when a dense model is the better choice for local inference. - Author: Rowan Ashby - Tags: AI, Explainer, Glossary, Mixture of Experts, Mixtral, DeepSeek, Transformer, #lang-en, #evergreen, #tr-mixture-of-experts-moe-explained ## Key takeaways - A mixture-of-experts (MoE) model splits part of each Transformer layer into many smaller "expert" networks, and a learned router sends every token to only a few of them. - That is why MoE models quote two sizes: DeepSeek-V3 stores 671 billion parameters but activates 37 billion per token, according to its technical report. - MoE cuts the compute needed per token, but not the memory: every expert still has to be loaded, so a 46.7B-parameter Mixtral needs the memory of a 46.7B model. - Training an MoE requires load balancing so the router does not overload a few favourite experts while others sit idle. - For local inference, MoE models run fast for their size if you have the RAM or VRAM; with limited memory, a smaller dense model is often the more practical choice. What is mixture of experts? It is a neural network design in which a router sends each token to only a few expert networks. Mistral AI's Mixtral 8x7B holds 46.7 billion parameters yet uses about 12.9 billion per token, the company says. The idea explains why most headline open-weight models now advertise a "total" and an "active" parameter count, why they can be cheap to run per token, and why they still demand so much memory. This explainer walks through how MoE works, what the numbers mean, and what changes when you try to run one on your own hardware. ## What is mixture of experts (MoE)? In a standard ("dense") Transformer, every token passes through every weight in every layer. Each layer has an attention block, which lets tokens look at each other, and a feed-forward network (FFN), which processes each token on its own. In most large models the FFN blocks hold the majority of the parameters. An MoE model replaces some or all of those FFN blocks with a set of parallel FFNs called **experts**, plus a small **router** (also called a gate or gating network). For each token, the router scores all experts, keeps the top few, and combines their outputs using the router's scores as weights. The other experts are skipped entirely for that token. The concept is old. Hugging Face's [MoE primer](https://huggingface.co/blog/moe?ref=bullquill.com) traces it to a 1991 paper, "Adaptive Mixture of Local Experts." The modern version for language models took shape in 2017, when researchers including Noam Shazeer, Geoffrey Hinton and Jeff Dean published "[Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer](https://arxiv.org/abs/1701.06538?ref=bullquill.com)." That paper inserted a layer of up to thousands of experts between LSTM layers, reached up to 137 billion parameters, and reported more than 1,000x gains in model capacity with only minor losses in computational efficiency. ## How the router chooses experts: top-k gating The router is a small learned layer trained together with the rest of the network. For each token it produces one score per expert. The model then keeps only the *k* highest scores (top-k routing) and sends the token to those experts. - **Top-1:** The Switch Transformer used a single expert per token. According to Hugging Face, it was released with 1.6 trillion parameters and 2,048 experts and reported a 4x pre-training speed-up over T5-XXL. - **Top-2:** Mixtral 8x7B sends each token to 2 of its 8 experts per layer, per the [Mixtral paper](https://arxiv.org/abs/2401.04088?ref=bullquill.com). - **Many small experts:** DeepSeek-V3 has 1 shared expert plus 256 routed experts in each MoE layer and activates 8 routed experts per token, according to its [technical report](https://arxiv.org/abs/2412.19437?ref=bullquill.com). Qwen3's two MoE models each have 128 experts with 8 active, per the Qwen team. - **Shared expert:** Meta's Llama 4 Maverick sends every token to one shared expert and to one of 128 routed experts, and alternates dense and MoE layers, Meta said. Routing happens per token and per layer. A single word can pass through different experts in layer 5 and layer 30, so "experts" are not neat subject specialists such as a "maths expert" or a "French expert." They are learned partitions of the network's capacity. ## Total vs active parameters: what the numbers mean **Total parameters** is everything stored in the model file: attention weights, embeddings, the router, and every expert. **Active parameters** is the subset that actually multiplies against one token: the shared layers plus the few experts the router selected. Mixtral shows why the total is not simply "8 x 7B = 56B." Only the FFN blocks are duplicated into experts; attention and other layers are shared, Hugging Face explains, so the real total is about 47 billion. With two experts per token, Hugging Face says inference compute is comparable to a 12B dense model. | Model | Released | Total parameters | Active per token | Active share | Expert setup | | ------------------------- | -------- | ---------------- | ---------------- | ------------ | ----------------------------------------- | | Mixtral 8x7B (Mistral AI) | Dec 2023 | 46.7B | 12.9B | 27.6% | 8 experts, 2 per token | | DeepSeek-V3 | Dec 2024 | 671B | 37B | 5.5% | 1 shared + 256 routed, 8 routed per token | | Llama 4 Scout (Meta) | Apr 2025 | 109B | 17B | 15.6% | 16 experts | | Llama 4 Maverick (Meta) | Apr 2025 | 400B | 17B | 4.3% | 1 shared + 128 routed, 1 routed per token | | Qwen3-30B-A3B | Apr 2025 | 30B | 3B | 10.0% | 128 experts, 8 active | | Qwen3-235B-A22B | Apr 2025 | 235B | 22B | 9.4% | 128 experts, 8 active | | gpt-oss-20b (OpenAI) | Aug 2025 | 21B | 3.6B | 17.1% | MoE, details in model card | | gpt-oss-120b (OpenAI) | Aug 2025 | 117B | 5.1B | 4.4% | MoE, details in model card | *Figures from each developer's own publication: Mistral AI announcement, DeepSeek-V3 technical report (arXiv 2412.19437), Meta Llama 4 blog post, Qwen3 blog post, and OpenAI's gpt-oss-120b model card on Hugging Face. "Active share" is Bullquill's calculation (active / total). Data as of 10:30 UTC, October 27, 2026.* The trend in the table is clear: the largest recent models push the active share to around 4-6% by using many small experts, so a very large store of weights costs only a small slice of compute per token. Some model names now spell the split out directly: Qwen3-30B-A3B means 30B total and 3B active. ## Why MoE cuts compute but not memory The cost of generating a token scales mainly with the active parameters, because those are the only weights multiplied for that token. That is the payoff: a model with the knowledge capacity of hundreds of billions of parameters can generate text with the arithmetic budget of a model a fraction of its size. Mistral said Mixtral offered 6x faster inference than Llama 2 70B while outperforming it on most benchmarks. Memory is a different story. The router can choose any expert for the next token, so all experts must be resident in GPU or system memory. Hugging Face's primer states plainly that all parameters need to be loaded in RAM, which makes memory requirements high. Meta makes the same point for Llama 4: all parameters are stored in memory, but only a subset is activated during serving. A rough rule: weight memory is about total parameters multiplied by bytes per parameter. At 16-bit precision (2 bytes), Mixtral's 46.7 billion parameters work out to roughly 93 GB before any cache or overhead; that is Bullquill's arithmetic, not a vendor figure. This is why [quantization](https://bullquill.com/what-is-llm-quantization-gguf/) matters so much for MoE: Ollama's [mixtral:8x7b](https://ollama.com/library/mixtral?ref=bullquill.com) download is listed at 26 GB, and `mixtral:8x22b` at 80 GB. OpenAI says gpt-oss-120b fits on a single 80 GB GPU partly because its MoE weights were post-trained with MXFP4 quantization, while gpt-oss-20b runs within 16 GB of memory. Long prompts add more memory on top of the weights through the key-value cache, which grows with the [context window](https://bullquill.com/what-is-a-context-window-llm/) you use. DeepSeek-V3 pairs MoE with Multi-head Latent Attention, an attention design its report says was adopted for efficient inference. ## Load balancing: keeping every expert busy Left alone, a router tends to favour a handful of experts early in training. Those experts improve faster, get chosen even more, and the rest are wasted. In distributed training, overloaded experts also become bottlenecks because each expert typically lives on a particular device. Classic MoE designs counter this in two ways, per Hugging Face: - **Auxiliary loss:** an extra training term that rewards spreading tokens evenly across experts. - **Expert capacity:** a cap on how many tokens one expert can take per batch, set as (tokens per batch / number of experts) x a capacity factor. Switch Transformers performed well with factors of 1 to 1.25; higher values tolerate imbalance but raise communication costs. DeepSeek-V3 took a different route. Its report describes an auxiliary-loss-free strategy: each routed expert gets a bias that is added to its score only when choosing the top experts. After every training step, the bias of overloaded experts is lowered and that of underloaded experts raised by a small fixed step, while the weights used to mix expert outputs still come from the original scores. DeepSeek also limits each token to at most 4 nodes to contain cross-machine traffic. The approach has a cost profile that drew wide attention: DeepSeek reported 2.788 million H800 GPU hours for the full training run, or $5.576 million at an assumed rental price of $2 per GPU hour, excluding earlier research and ablation experiments. ## MoE vs dense models: trade-offs for local inference | Question | MoE model | Dense model | | ------------------ | ------------------------------------------------------- | ------------------------------------------------------ | | Speed per token | Set by active parameters, so fast for its total size | Set by all parameters | | Memory needed | Set by total parameters; all experts must be loaded | Set by all parameters, usually smaller for equal speed | | Best fit | Plenty of RAM/VRAM, high-throughput serving, many users | Tight memory budget, single GPU, low throughput | | Fine-tuning | More prone to overfitting; historically harder | Mature, well-understood recipes | | Serving complexity | Expert parallelism and routing overhead at scale | Simpler to shard and batch | *Based on Hugging Face's MoE primer and the developer publications cited above.* Hugging Face sums up the rule of thumb: sparse MoE models suit high-throughput settings with many machines, while dense models suit low-throughput use with limited VRAM. It also warns that total parameter counts of sparse and dense models should not be compared directly. In practice, that gives three situations for home or small-team setups: 1. **You have lots of system RAM but a modest GPU.** An MoE model with a small active count, such as a 30B-total, 3B-active design, can be attractive, because each token touches relatively few weights. Tools covered in our [Ollama guide](https://bullquill.com/how-to-run-local-llm-with-ollama/) can split layers between GPU and CPU. 2. **You have one GPU and little else.** The total size must fit after quantization. If it does not, a dense model of similar quality that fits is usually faster than an MoE that spills heavily to system memory. 3. **You serve many users.** Batching many requests is where MoE shines, but it needs a serving engine that handles expert routing well; see our guide to [serving an LLM with vLLM](https://bullquill.com/how-to-serve-llm-with-vllm/). ## Fine-tuning MoE models Hugging Face notes that MoE models overfit more easily than dense ones, so stronger regularisation inside experts can help. It also reports that freezing the MoE parameters and updating only the rest worked almost as well as updating everything, which speeds up training and saves memory. Instruction tuning looks promising: Flan-MoE gained more from it than Flan-T5 did over T5\. For most readers, parameter-efficient methods remain the starting point; our [LoRA fine-tuning tutorial](https://bullquill.com/how-to-fine-tune-llm-with-lora/) covers the basics. ## What it means MoE has become the default way to scale open-weight language models because it decouples how much a model knows from how much it costs to run per token. Every model in the table above uses it, from Mixtral in 2023 to OpenAI's gpt-oss in 2025. For anyone choosing a model, the practical lesson is to read both numbers. The active count predicts speed and per-token cost; the total count predicts how much memory you need. A "120B" MoE can generate text quicker than a 30B dense model and still refuse to load on a laptop. Match the total size, after quantization, to your hardware first, then use the active size to estimate speed. ## Frequently asked questions ### What is mixture of experts in simple terms? It is a model design in which parts of each layer are split into many smaller expert networks, and a learned router sends each token to only a few of them. The model can store many parameters while using only a fraction of them for each token. ### What is the difference between total and active parameters? Total parameters are all the weights stored in the model, including every expert. Active parameters are the weights actually used for one token. DeepSeek-V3, for example, has 671 billion total parameters and activates 37 billion per token, according to its technical report. ### Does a mixture-of-experts model need less memory? No. MoE reduces the computation per token, but all experts must be loaded because the router may pick any of them for the next token. Memory needs follow the total parameter count, which is why quantization is widely used to run MoE models locally. ### Is Mixtral 8x7B a 56B model? No. Only the feed-forward blocks are duplicated into experts, while attention and other layers are shared, so Mixtral 8x7B has about 46.7 billion parameters in total and uses about 12.9 billion per token, according to Mistral AI.