Chapter 1

What is a Transformer? Why does it dominate AI?

In 2017, a paper titled "Attention is All You Need" changed the AI field. The Transformer architecture not only solved the sequence processing problems of RNN/LSTM, but also became the foundation of modern AI. From GPT to BERT, from text to images, Transformers are everywhere.

The birth of Transformer

In June 2017, the Google Brain team published the paper “Attention is All You Need” and introduced the Transformer architecture. This paper had only 6 authors, yet it changed the development trajectory of the entire AI field.

Core paper information

1
Paper title：Attention is All You Need
2
Publication time: June 2017 (NIPS 2017)
3
Author Team: Google Brain (Vaswani et al.)
4
Core contributions：Fully based on the Attention mechanism, no recurrence or convolution required

Historical background

Before Transformers, sequence modeling mainly relied on RNNs and LSTMs:

  • • The dilemma of RNNs: vanishing gradient problem, difficult to handle long sequences
  • • Improvements to LSTM: Mitigates vanishing gradients through a gating mechanism, but computation is still sequential
  • • CNN attempts: used for sequence modeling, but the receptive field is limited
  • • The breakthrough of Transformer: fully parallel, global attention, fundamentally transformed sequence modeling

Why have Transformers become mainstream?

Transformer was not the first architecture to use Attention, but by completely abandoning recurrence and convolution, it achieved true parallel computation and global dependencies, which was the key to its success.

vs RNN

  • • Parallelism: RNNs must process sequentially, while Transformers can process in parallel
  • • Long-range dependencies：RNN gradient vanishing, Transformer global attention
  • • Training speed: Transformer training is 10-100x faster
  • • Memory efficiency: Transformer memory usage is more stable

vs LSTM

  • • Complexity: LSTM has a complex gating mechanism, while the Transformer architecture is simpler
  • • Scalability: LSTM is difficult to scale to very large sizes, while Transformer can
  • • Expressiveness: Transformer attention mechanisms are more flexible
  • • Hardware-friendly: Transformers are better suited for GPU parallel computation

vs CNN

  • • Receptive field: CNNs require multiple layers to expand the receptive field, while Transformers achieve it in one step
  • • Location information: CNN uses position encoding of convolution kernels, while Transformer uses positional encoding
  • • Long sequence: CNNs require many layers to process long sequences, while Transformers are more efficient
  • • Generality: Transformers use a unified architecture, while CNNs need to be designed for different tasks

Key breakthrough points

1
Fully parallel: compute all positions simultaneously, fully leveraging GPU parallelism
2
Global Attention: any two positions can interact directly without multi-level propagation
3
Scalability: From 100 million parameters to 1 trillion parameters, following the Scaling Law
4
Unified architecture : Text, images, and audio can all be processed with Transformers

Core component: Encoder-Decoder architecture

The core of Transformer is the Encoder-Decoder architecture, but in practical applications many variants have emerged.

Standard Transformer architecture

The complete architecture in the original paper includes two parts: Encoder and Decoder:

Encoder: Understand the input sequence and encode the input into an intermediate representation
Decoder: Based on the Encoder output and the generated partial results, generate the target sequence
Application scenarios: sequence-to-sequence tasks such as machine translation and text summarization

Architecture variants

GPT（Decoder Only）

Decoder only, autoregressive generation, suitable for text generation tasks

BERT（Encoder Only）

Encoder only, bidirectional understanding, suitable for understanding tasks (classification, question answering)

Key innovation points

  • • Self-Attention: Let the model focus on different parts of the input sequence
  • • Multi-Head Attention: understand information from multiple perspectives
  • • Positional Encoding: inject positional information to compensate for the lack of sequence order in Attention
  • • Feed-Forward Network: nonlinear transformation, enhancing expressive power
  • • Residual Connection: Solve the vanishing gradient problem in deep networks
  • • Layer Normalization: Stable training process

Historical significance: a paradigm shift in the AI field

Transformer is not just an architecture, but a paradigm shift in AI, ushering in the era of large-scale pre-trained models.

Impact scope

Natural language processing

  • • GPT series: GPT-1/2/3/4, ChatGPT, transformed text generation
  • • The BERT family: BERT, RoBERTa, ALBERT, transformed text understanding
  • • T5, BART: unified the framework for text tasks

Computer Vision

  • • Vision Transformer (ViT): apply Transformers to images
  • • DETR: a Transformer approach for object detection
  • • CLIP: multimodal Transformer model

Speech Processing

  • • Whisper: OpenAI's speech recognition model
  • • AudioLM: a Transformer model for audio generation

Multimodal

  • • GPT-4V: visual understanding capability
  • • Gemini: native multimodal architecture
  • • Unified architecture processes text, images, and audio

Paradigm shift

1
From task-specific models to general models: Transformers ushered in the era of large-scale pretraining + fine-tuning
2
From handcrafted features to end-to-end learning: Transformer achieved true end-to-end learning
3
From single modality to unified multimodality: Transformer unified the way different modalities are handled
4
From model design to data scale: Transformers demonstrated the importance of data scale (Scaling Law)

Learning outcomes

After completing this chapter, you will:

  • 1Understand the historical background behind the birth of Transformers and the core contributions of the 2017 paper
  • 2Master the core advantages of Transformers over RNNs/LSTMs/CNNs (parallelism, global attention, scalability)
  • 3Understand the basic principles and key innovations of the Encoder-Decoder architecture
  • 4Understand the significance and scope of the paradigm shift brought by Transformer to the AI field
Previous chapter: Architecture overviewNext chapter: The core mechanisms of Transformer