learn
Mixture of Experts
What Mixture of Experts means, how it works, and which tools and models on this site relate to it.
Definition
A mixture-of-experts (MoE) model routes each token to a subset of expert networks so total parameters can be large while compute per token stays smaller.
How it works
A gating network chooses experts. Serving MoE models has different memory and routing costs than dense models of similar quality.