Strengths and limitations of Transformers
Analyze in depth the core strengths of Transformer (parallel computation, long-range dependencies, scalability, versatility) and its limitations (computational complexity, memory consumption, inference efficiency), and understand the impact of Scaling Law on model development.
Transformer's strengths
Transformer’s success comes from its unique advantages, which have made it the foundational architecture of modern AI.
Advantages of parallel computing
Compared with RNNs, which must process sequentially, Transformers can process all positions in parallel:
- • Training speed: 10–100 times faster than RNN
- • GPU-friendly：Make full use of GPU parallel computing capabilities
- • Scalability: Easily scalable to a larger batch size
- • Actual impact: this makes it possible to train large models like GPT-3
Long-range dependencies
Global attention mechanism, any two positions can interact directly:
- • One-stop: no multi-stage propagation needed, directly establish long-distance connections
- • Understanding ability: able to understand document-level semantic relationships
- • Application scenarios: long-document comprehension, code analysis, conversational systems
- • Comparison: RNNs require O(n) steps, while Transformers need only O(1) steps
Scalability (Scaling Law)
Transformers follow the scaling law; the larger the model, the better the performance:
- • Parameter scale: From 100 million (BERT) to 1 trillion (GPT-4) parameters
- • Performance improvement: When parameters increase 10x, performance improves by about 2x
- • Data requirements: training data that matches the parameter scale is required
- • Cost: training costs grow exponentially with parameter scale
Generality
An integrated architecture that handles multiple modalities and tasks:
- • Text: GPT, BERT, T5, etc.
- • Image：ViT、DETR、CLIP
- • Audio：Whisper、AudioLM
- • Multimodal: Process GPT-4V and Gemini uniformly
Scaling Law: the relationship between parameter scale and performance
Scaling Law reveals the relationship between model size, data volume, computing resources, and performance, and is key to understanding the development of large models.
Core principles
Parameter scale
As the number of model parameters increases, performance (usually) improves. However, there are diminishing returns, so it needs to match the amount of data.
Data scale
The amount of training data needs to match the model scale. A dataset that is too small cannot fully utilize the capacity of a large model.
Computing resources
Training compute (FLOPs) increases with model size. GPT-3 training requires about 3.14×10²³ FLOPs.
Actual data
Note: The increase in parameter scale has brought significant performance improvements, but it has also caused a sharp rise in training and inference costs
Limitations of Transformer
Although Transformers have been very successful, they also have some fundamental limitations, which have driven research into new architectures.
Computational complexity O(n²)
The Attention mechanism needs to compute relationships between all position pairs, resulting in quadratic complexity:
- • Source: QK^T matrix computation requires O(n²) time and space
- • Impact: when sequence length doubles, computation increases by 4x
- • Practical limitations: Difficult to handle extremely long sequences (e.g. 1 million tokens)
- • Solution: Sparse Attention, Linear Attention, chunked processing
Memory consumption
The attention matrix needs to store the relationships between all pairs of positions:
- • Memory requirements: sequence length n, attention matrix size is n×n
- • Practical example: Processing 32K tokens requires about 4GB of memory (attention matrix only)
- • Limitations: limited the sequence length that can be processed
- • Optimization: Techniques such as Flash Attention and gradient checkpointing
Inference efficiency
Autoregressive generation must generate tokens one by one and cannot be parallelized:
- • Sequential generation: each token depends on all previous tokens
- • Latency issues: Generating long text requires multiple forward passes
- • KV Cache: computed KVs are cached, but still need to be generated one by one
- • Comparison: Training can be parallelized, but inference must be serial
Limitations of positional encoding
Fixed positional encoding struggles to handle very long sequences:
- • Fixed encoding: The sequence length seen during training is limited
- • Extrapolation problem: difficult to handle sequences longer than those during training
- • Relative position: Relative positional encoding has improved, but there are still limitations
- • Solution: New technologies such as RoPE (Rotary Positional Encoding) and ALiBi
Training data requirements
- • Data scale: Large models require massive training data (GPT-3 used 570GB of text)
- • Data quality: Data quality is just as important; high-quality, diverse data is required
- • Cost: Data collection, cleaning, and labeling are costly
- • Bottleneck: Acquiring high-quality data may become a bottleneck in model development
Actual impact analysis
The impact and trade-offs of these characteristics in real-world applications.
Opportunities brought by advantages
- • Large-scale pretraining: can train ultra-large-scale models
- • General-purpose capabilities: One model handles multiple tasks
- • Rapid iteration: Accelerate development with parallel training
- • Unified architecture: Simplifies model design and deployment
Challenges posed by limitations
- • Cost issues: Training and inference costs are high
- • Long sequence limitation: Difficult to handle very long documents
- • Real-time capability: inference latency affects real-time applications
- • Resource requirements: Requires a large amount of GPU and memory
Trade-off thinking
In practical applications, it is necessary to weigh the advantages and limitations of Transformer according to the scenario:
- • Short text tasks: Transformers have a clear advantage, and O(n²) complexity is acceptable
- • Long document tasks: Need to consider linear-complexity architectures (such as Mamba)
- • Real-time applications: You need to optimize inference efficiency or use a faster architecture
- • Resource-constrained: It may be necessary to use smaller models or a hybrid architecture
Learning outcomes
After completing this chapter, you will:
- 1Understand the core advantages of Transformers (parallel computation, long-range dependencies, scalability, versatility)
- 2Master the core principles of Scaling Law and understand the relationship between parameter scale, data volume, and performance
- 3Deeply understand the limitations of Transformers (O(n²) complexity, memory consumption, inference efficiency, positional encoding)
- 4Able to evaluate the suitability of Transformers in different scenarios and make sensible architecture choices