Chapter 5

Mixture of Experts (MoE)

The mixture-of-experts model uses routing mechanisms and sparse activation to enable training and inference of ultra-large-scale models. Models such as GPT-4 and Mixtral all adopt the MoE architecture.

Principles of the MoE architecture

MoE achieves model scaling by decomposing the model into multiple expert networks and activating only a subset of experts for each input.

Core concepts

1
Expert network (Experts): multiple independent neural networks, each expert specializing in different patterns
2
Router: Decide which experts each input should activate
3
Sparse activation: activate only a subset of experts at a time (for example, 2–4), rather than all of them
4
Weighted combination: Weight and combine the outputs of activated experts to obtain the final result

Workflow

1. Input → Router
2. The router calculates the weight of each expert
3. Select Top-K experts (e.g., Top-2)
4. Activate the selected expert
5. Weighted combination of expert outputs
6. Output results

Advantages of MoE

MoE makes it possible to train ultra-large-scale models while maintaining inference efficiency.

Ultra-large-scale model

  • • Parameter scale: can train trillion-parameter-scale models
  • • Model capacity: Large total parameter count, but few active parameters
  • • Real-world cases: GPT-4 (estimated 1.76T parameters), Mixtral 8x7B
  • • Skill improvement: larger model capacity brings stronger capabilities

Training efficiency

  • • Sparse activation: only train a portion of experts each time
  • • Savings calculation: training compute is far less than dense models
  • • Parallel training: different experts can train in parallel
  • • Cost control: training costs are relatively manageable

Specialization

  • • Specialized division of labor: different experts learn different patterns
  • • Domain specialization: Experts can focus on specific domains
  • • Knowledge separation: Different knowledge is stored in different experts
  • • Explainability: can analyze which experts were activated

Inference efficiency

  • • Few activation parameters: Only some parameters are used during inference
  • • Speed improvement：Inference speed is close to that of small models
  • • Cost reduction: Reasoning cost is far lower than dense models
  • • Practical example: Mixtral 8x7B inference speed is close to that of 7B models

Challenges of MoE

MoE is powerful, but it also brings new challenges and complexity.

Routing complexity

The router needs to learn how to correctly assign input to experts:

  • • Routing Study: the router needs to be trained jointly with the expert network
  • • Load balancing: It is necessary to ensure that all experts are fully utilized
  • • Expert collapse: Some experts may not be used (expert crash issue)
  • • Training instability: Routing training may be unstable

Load balancing

  • • Evenly distribute: need to ensure inputs are evenly distributed to each expert
  • • Load balancing loss: Encourage load balancing through the loss function
  • • Real challenges: some inputs may always be routed to the same expert
  • • Solution: load balancing regularization, auxiliary loss functions

Communication costs

  • • Distributed training：Experts may be distributed across different GPUs
  • • Communication overhead: Data needs to be transferred between GPUs
  • • Bandwidth limit：Communication bandwidth may become a bottleneck
  • • Optimization: communication patterns and data transfer need to be optimized

Practical application cases

Many well-known models have adopted the MoE architecture, proving its effectiveness.

GPT-4 (speculated)

Although OpenAI has not 공개 detailed architecture, it is speculated that GPT-4 uses MoE:

  • • Parameter scale: estimated at about 1.76 trillion parameters
  • • Number of experts: may require 16 experts
  • • Activation strategy: activate 2–4 experts each time
  • • Effect: Powerful capabilities, controllable inference cost

Mixtral 8x7B

Mistral AI's open-source MoE model demonstrates the real-world effects of MoE:

  • • Architecture: 8 experts, each with 7B parameters
  • • Activate: Activate 2 experts each time (Top-2)
  • • Total parameters: 47B parameters, but about 13B active parameters
  • • Performance: Performance is close to a 70B dense model, and inference speed is close to a 7B model

Switch Transformer

Google's MoE model demonstrates the scalability of MoE:

  • • Scale: from 1.6T to 1.6T parameters
  • • Innovation: simplify the routing mechanism and use Top-1 routing
  • • Effect: Demonstrated the scalability of MoE
  • • Impact: inspired the development of subsequent MoE models

Summary of MoE advantages

  • • Scaling expansion: can train trillion-parameter-scale models
  • • Cost control: Training and inference costs are relatively manageable
  • • Performance improvement: larger model capacity brings stronger capabilities
  • • Practical value: Improve model capability while maintaining inference efficiency

Learning outcomes

After completing this chapter, you will:

  • 1Understand the architectural principles of MoE (expert networks, routers, sparse activation, weighted combination)
  • 2Learn how sparse activation improves efficiency and understand MoE applications in ultra-large-scale models
  • 3Understand the challenges of MoE (routing complexity, load balancing, communication cost) and the solutions
  • 4Understand the practical effects and application value of MoE through real-world cases such as GPT-4 and Mixtral
Previous chapter: Mamba / State Space ModelsNext chapter: RAG retrieval-augmented generation