AI_SLANG_ENTRY

Mixture of Experts (MoE) Meaning

A neural network architecture that splits computation across many specialized sub-models (experts) and routes each input to only a subset of them, so total parameter count can grow without making every forward pass equally expensive.

AI_TASTE=3/5 ███░░ TREND=HIGH

What does Mixture of Experts (MoE) mean?

Instead of one giant network doing all the work, the model is split into many smaller specialist networks, and only a few of them run on any given input. That lets the total model be much larger while keeping the per-step compute cost closer to a smaller model.

Origin and usage

Grew out of the original Jacobs, Jordan, Nowlan and Hinton adaptive mixtures of local experts idea and was scaled into modern language models by Shazeer's sparsely-gated MoE work and Google's Switch Transformer. Today the pattern sits behind several well-known frontier and open-weight models.

Source type: paper. Last checked: 2026-07-25.

Architectural pattern with multiple stable paper references. Google's Switch Transformer (Fedus, Zoph, Shazeer) is the clearest canonical citation for modern sparsely-gated MoE language models; Shazeer's sparsely-gated mixture-of-experts layer paper is the earlier conceptual reference.

Primary reference

How a mixture of experts works

The model has many expert sub-networks, but only a few are activated for any given token.
A learned router scores each token and decides which experts should handle it.
Active experts produce the output, and a combine step merges their predictions.
Training has to balance load across experts so the router does not collapse onto a few favorites.

Why MoE matters for builders

MoE is one of the main reasons a modern frontier model can advertise a very large parameter count while still running at usable speeds. The hardware cost is closer to the active parameter count than the total parameter count, even though the full model is what gets loaded into memory.

For builders, the practical question is what the model exposes: the hosted API usually only shows total parameter count, active parameter count, and context window, not the routing behavior. Open-weight MoE releases also require enough VRAM to hold every expert even if only a few run per token.

MoE versus a dense model

A dense model uses every parameter on every token. A mixture of experts keeps a large parameter pool in memory but only spends compute on a small subset per token. The trade is memory footprint versus per-step cost, and the result usually shows up as higher total parameter counts with similar or faster inference latency.

It is not a free lunch: MoE models are harder to fine-tune, can be less stable during training, and may behave unevenly on niche tasks if some experts are underused.

Examples

The new flagship is a mixture of experts with eight active experts per token.
MoE lets a model have hundreds of billions of parameters without paying for all of them on every step.

FAQ

What is a Mixture of Experts model?

A Mixture of Experts (MoE) model is a neural network that splits its capacity into many specialist sub-models and routes each input to a small subset of them, so the total parameter count can be very large without paying for every parameter on every token.

Why do modern LLMs use mixture of experts?

MoE lets a model scale parameter count without scaling per-token compute cost the same way, which is why several frontier and open-weight LLMs use the pattern to balance quality with inference cost.

Is MoE the same as an ensemble?

No. Ensembles usually run several full models and average their outputs. MoE routes each input through only a few expert sub-networks inside one trained model, and the routing is learned rather than fixed.

Does MoE save memory?

Not necessarily. MoE saves per-token compute, but all experts still need to be loaded, so total memory footprint can be much higher than for a similarly priced dense model.