4.2. Sparse and Mixture-of-Experts (MoE) Models

As model size grows (billions → trillions of parameters), we hit a wall:

• Compute cost grows linearly with active parameters.
• Memory and latency become unmanageable.

Mixture-of-Experts (MoE) solves this by sparsely activating parameters: not all neurons work at once — only a small subset (experts) does.

That means the total model capacity is huge (say, 1 trillion parameters), but the active compute per token is small (say, 20 billion).

So you get the best of both worlds: 🧠 High capacity, ⚡ Low cost per token.

The Switch Transformer (Google, 2021) made MoE practical and efficient.

• The model contains multiple feed-forward expert layers per Transformer block.
• For each token, a router computes scores for all experts and picks the top one (top-1 routing).
• Only that expert processes the token — the rest stay inactive.

This drastically cuts computation. Each token takes a specialized route through the network, allowing conditional computation.

Key innovation: The “Switch” mechanism replaced complicated multi-expert routing with a simple, scalable approach — one token → one expert.

Equation (conceptually):

Where $G(x)$ is the router score for each expert.

Result: Switch Transformers achieved 7x faster training and comparable or better quality than dense models of equal compute.

GLaM (Generalist Language Model), also by Google, refined this further. Instead of using only one expert per token, it used top-2 routing: each token is processed by two experts, and their outputs are combined via weighted averaging.

This increased representational power while keeping compute efficient.

GLaM’s architecture:

• 64 experts per MoE layer.
• Only 2 experts active per token.
• Total parameters: 1.2 trillion.
• Active parameters per token: only ~97 billion.

So GLaM had 1.2T knowledge capacity but 90% compute savings.

Mixtral (2023) combined the power of MoE with modern efficient attention.

• It uses 8 experts, activating only 2 per token (top-2 routing).
• Each expert is itself a small Transformer block.
• Thanks to parallel expert computation, it achieves higher throughput than dense models.

Mixtral’s claim to fame: It achieves GPT-3.5-level performance while being open-source and significantly cheaper to run — because only a fraction of parameters are active per forward pass.

In other words:

More brains, less burn. 🧠🔥

Routing is handled by a gating network — a small module that decides which experts to activate.

For each input token $x$:

1. Compute scores for all experts: $g_i = W_g \cdot x$
2. Pick the top-$k$ experts (usually $k=1$ or $2$).
3. Normalize their scores using softmax to create routing probabilities.
4. Forward the token to those experts.

Mathematically:

where $p_i = \text{softmax}(g_i)$ are the routing weights.

This mechanism ensures tokens take different routes depending on their content — some go to “math experts,” others to “grammar experts,” etc.

A major challenge in MoE training: router collapse. If left unchecked, the router may send most tokens to the same expert, underutilizing others.

Solutions include:

1. Load Balancing Loss: Add a penalty if some experts are overused while others idle.

   $$L_{balance} = \lambda \cdot \text{Var}(n_i)$$

   where $n_i$ is the number of tokens per expert.

2. Expert Dropout: Randomly disable experts during training → encourages robustness and fair routing.

3. Noisy Gating: Add Gaussian noise to router logits → promotes exploration of different experts early in training.

Together, these make MoE models stable, fair, and scalable.

Dense models apply the same computations to every token — like asking the whole company to attend every meeting.

MoE models apply conditional computation — only relevant submodules engage. This improves:

• Efficiency: Fewer active parameters per step.
• Specialization: Experts develop niche skills.
• Scalability: Add more experts without linearly increasing compute.

Mixture-of-Experts represents a paradigm shift: Instead of making one monolithic brain, we’re building modular intelligence systems — where components learn specialized behaviors and coordinate dynamically.

This design philosophy echoes biological intelligence: the brain also activates only specific regions for specific tasks.

• $p_i$: routing probability for expert $i$.
• $\text{Expert}_i(x)$: output from expert $i$.
• Only the top-$k$ experts (usually 1 or 2) are active per token.

• “MoE just adds experts — it’s bigger, not faster.” Incorrect — the total model is larger, but per-token compute is much smaller.
• “Routing means each token goes through all experts.” No — typically, only 1–2 experts are used per token.
• “Experts are like layers.” They’re parallel modules within a layer, not sequential ones.