Mixture of Experts

A mixture-of-experts (MoE) model routes each token to a subset of expert networks so total parameters can be large while compute per token stays smaller.

A gating network chooses experts. Serving MoE models has different memory and routing costs than dense models of similar quality.