Chapter 2

The core mechanism of Transformers

Gain a deep understanding of the Attention mechanism, Encoder-Decoder architecture, feed-forward networks, and residual connections, and master how Transformers work.

Attention mechanism: the soul of Transformers

The Attention mechanism is the core of the Transformer, enabling the model to "attend" to different parts of the input sequence and understand the relationships between them.

Self-Attention: self-attention

Self-Attention allows each position in a sequence to directly attend to all other positions:

1
Query (Q): what information do you want to search for at the current location
2
Key (K): What information is provided elsewhere
3
Value (V): actual information content
4
Computation process：Attention(Q, K, V) = softmax(QK^T / √d_k) V

Multi-Head Attention: Multi-Head Attention

Multiple attention heads work in parallel to understand information from different angles:

  • • Parallel Computing: multiple heads compute simultaneously without interfering with each other
  • • Different perspectives: each head focuses on different relations (syntax, semantics, position, etc.)
  • • Information fusion: Concatenates the outputs of multiple heads and then applies a linear transformation
  • • Expressiveness: The multi-head mechanism greatly enhances the model's expressive power

Positional Encoding

The Attention mechanism itself does not contain positional information and needs to be injected through positional encoding:

  • • Sinusoidal positional encoding: Use sin/cos functions to generate positional encodings
  • • Learnable positional encoding: Models such as BERT use learnable embeddings
  • • Relative position encoding: focus on relative position rather than absolute position
  • • Limitations: Fixed positional encoding struggles with very long sequences (e.g., 1M+ tokens)

Highlighter theory: an intuitive way to understand Attention

The metaphor of marking key points with a highlighter helps explain how the Attention mechanism works.

Explain by analogy

Imagine you’re reading an article and highlighting the important parts with a highlighter:

1
Reading process: you read word by word and understand the meaning of each word
2
Mark key points: when you encounter important information, mark it with a highlighter
3
Related understanding: Marked words help you understand contextual relationships
4
The role of Attention：Attention is the model’s "highlighter", automatically marking and linking important information

Practical example

Sentence: "The cat sat on the mat"
When the model processes "sat", the Attention mechanism will:
• High attention weight → "cat" (subject)
• Middle attention weights → "on", "mat" (positional information)
• Low attention weight → "The" (article)

Encoder-Decoder architecture

The original Transformer architecture includes an Encoder and a Decoder, but many variants have emerged in practical applications.

Encoder

Understand the input sequence and generate an intermediate representation:

  • • Self-Attention: Understand the internal relationships within the input sequence
  • • Feed-Forward: nonlinear transformation
  • • Output: encoded representation of each position
  • • App: understanding tasks such as BERT, RoBERTa, etc.

Decoder

Generate the target sequence based on the Encoder output:

  • • Masked Self-Attention: only the generated portion is visible
  • • Cross-Attention: Focus on the Encoder output
  • • Generate: generate the target sequence token by token
  • • App: generative tasks such as GPT, T5

Practical application variants

GPT（Decoder Only）

Architecture: Decoder only, remove the Cross-Attention layer

Features: autoregressive generation, generating text from left to right

App: Text generation, conversation, code generation

BERT（Encoder Only）

Architecture: Encoder only, bidirectional understanding

Features: See the entire input sequence at once, with bidirectional context

App: Text classification, question answering, named entity recognition

T5（Encoder-Decoder）

Architecture: Complete Encoder-Decoder structure

Features: A unified framework where all NLP tasks are converted to text-to-text

App: Various tasks such as translation, summarization, and Q&A

Feedforward networks and residual connections

Feed-Forward Network, Residual Connection, and Layer Normalization are key components for stable Transformer training.

Feed-Forward Network (FFN)

The feedforward network performs nonlinear transformations independently at each position:

  • • Structure: A two-layer fully connected network with an activation function in between (usually ReLU or GELU)
  • • Role: enhance the model's nonlinear expressiveness
  • • Features: Each position is processed independently, allowing parallel computation
  • • Parameter: FFN usually accounts for most of the model's parameters (e.g., 66% in GPT-3)

Residual Connection (residual connection)

Add the input directly to the output, solving the vanishing gradient problem in deep networks:

  • • Formula：output = input + sublayer(input)
  • • Role: Provide a "highway" for gradient propagation
  • • Effect: enables the model to train deeper networks (e.g., GPT-3 has 96 layers)
  • • Source: Learn from the success of ResNet

Layer Normalization

Normalize the output of each layer to stabilize the training process:

  • • Position: after Attention and FFN, before the residual connection
  • • Role: Stable activation-value distributions accelerate training convergence
  • • Effect: allows the use of a larger learning rate, making training more stable
  • • Variant: Pre-LN (normalization before the sublayer) vs Post-LN (normalization after the sublayer)

Complete structure of the Transformer Block

Transformer Block:
1. Multi-Head Self-Attention
2. Add & Norm (residual connection + layer normalization)
3. Feed-Forward Network
4. Add & Norm (residual connection + layer normalization)

Multiple Transformer Blocks are stacked to form a deep network, and each Block contains the structure above

Learning outcomes

After completing this chapter, you will:

  • 1Gain a deep understanding of the mathematical principles of the Attention mechanism (Query, Key, Value) and its computation process
  • 2Learn how Multi-Head Attention understands information from multiple perspectives, and the role of positional encoding
  • 3Understand the architectural differences and use cases of GPT (Decoder only) and BERT (Encoder only)
  • 4Understand the roles of Feed-Forward Network, Residual Connection, and Layer Normalization
Previous chapter: What is a Transformer?Next chapter: Strengths and limitations of Transformer