Chapter 6

RAG retrieval-augmented generation

Retrieval-Augmented Generation (RAG) solves the knowledge-update, explainability, and cost issues of large models through external knowledge bases and context injection, and is the mainstream architecture for enterprise applications.

How RAG Works

RAG retrieves external knowledge bases and injects relevant information into the generation process, enabling knowledge-enhanced generation.

Complete workflow

1
Query understanding: Understand the intent and needs behind user queries
2
Vectorization: Convert queries into vector representations (Embedding)
3
Search: Retrieve similar documents in a vector database
4
Context injection: inject retrieved documents into the Prompt as context
5
Generate: Generate answers based on enhanced context

Core Components

  • • Vector database: Store vector representations of documents (e.g., Pinecone, Weaviate, Milvus)
  • • Embedding model: Convert text into vectors (e.g. OpenAI Embeddings, BGE)
  • • Retriever: perform similarity search (vector search, keyword search, hybrid search)
  • • Generative model: Generate answers based on retrieval results (e.g., GPT-4, Claude)

Advantages of RAG

RAG solves a key problem of large models, making them better suited for enterprise applications.

Knowledge updates

  • • Real-time updates: The knowledge base can be updated at any time without retraining the model
  • • Domain knowledge: Can inject domain-specific expertise
  • • Timeliness: Can access the latest information
  • • Flexibility: different queries can use different knowledge bases

Explainability

  • • Source tracing: can trace back to the source document of the answer
  • • Transparency: Users can view the retrieved documents
  • • Confidence: You can evaluate the credibility of information sources
  • • Audit：Convenient for auditing and compliance

Cost-effectiveness

  • • No fine-tuning required: no need to fine-tune models for each domain
  • • Knowledge base reuse: The same knowledge base can serve multiple applications
  • • Cost control: lower cost compared to fine-tuning
  • • Fast deployment: can quickly deploy new applications

Reduce hallucinations

  • • Factual basis: Generate based on retrieved documents to reduce hallucinations
  • • Accuracy: Improve the accuracy of answers
  • • Reliability: Better suited for scenarios that require accurate information
  • • Quality control: Answer quality can be controlled through knowledge base quality control

Challenges of RAG

RAG is powerful, but it also faces some challenges that need to be addressed.

Depends on retrieval quality

The effectiveness of RAG depends heavily on retrieval quality:

  • • Retrieval accuracy: If relevant documents cannot be retrieved, it will lead to incorrect answers
  • • Retrieval relevance: Retrieving irrelevant documents affects generation quality
  • • Documentation quality: the quality of the knowledge base directly affects RAG results
  • • Optimization directions: Improve retrieval algorithms, optimize document chunking, and use hybrid retrieval

Latency issues

  • • Retrieval latency: Vector search takes time
  • • Generation latency: generative models need to handle longer contexts
  • • Overall latency: The overall latency of RAG is higher than direct generation
  • • Optimization: caching, asynchronous retrieval, optimization of retrieval algorithms

Context limit

  • • Context window: models have context length limits
  • • Document selection: Need to select the most relevant from a large number of search results
  • • Information Compression: Retrieved documents may need to be compressed or summarized
  • • Solution: reranking, document summarization, hierarchical retrieval

Best practices for RAG

Key practices for building high-quality RAG systems.

Vector database selection

Pinecone

Managed service, easy to use, suitable for rapid prototyping

Weaviate

Open source, feature-rich, supports hybrid search

Milvus

High performance, suitable for large-scale deployment

Chroma

Lightweight and easy to integrate

Retrieval strategy

  • • Vector search: Based on semantic similarity, suitable for semantic search
  • • Keyword search: Based on keyword matching, suitable for exact matching
  • • Hybrid retrieval: Combining vector search and keyword search yields better results
  • • Reordering: Use a reranking model to optimize search results

Document Processing

  • • Chunking strategy: Split appropriately and preserve semantic integrity
  • • Metadata: add metadata (source, time, type, etc.)
  • • Preprocessing: clean up, standardize, deduplicate
  • • Update mechanism: Establish a mechanism for document updates and version management

Prompt engineering

  • • Context organization: Organize retrieved documents appropriately
  • • Clear instructions: Clearly instruct the model how to use the retrieved information
  • • Citation requirements: Require the model to cite sources
  • • Quality control: Add quality control instructions (e.g., "If information is insufficient, please say so")

RAG vs fine-tuning: how to choose?

Understand the applicable scenarios for RAG and fine-tuning, and make the right choice.

Scenarios for choosing RAG

  • • Knowledge updates are frequent: Need to update knowledge frequently
  • • Applications across multiple domains: Need to handle multiple domains
  • • Interpretability requirements: Need to trace the source of the answer
  • • Fast deployment: Need to quickly deploy new applications
  • • Cost control: need to control costs

Scenarios for choosing fine-tuning

  • • Task-specific: need to learn specific task patterns
  • • Style adaptation: Need to adapt to a specific output style
  • • Performance optimization: performance needs to be optimized for specific tasks
  • • Knowledge consolidation: Knowledge is relatively stable and does not require frequent updates
  • • Latency-sensitive: extremely high latency requirements

Hybrid approach

In real applications, RAG and fine-tuning can be used together: use fine-tuning to optimize core capabilities, and use RAG to inject domain knowledge.

Learning outcomes

After completing this chapter, you will:

  • 1Understand the complete RAG workflow (retrieval→augmentation→generation) and core components
  • 2Understand the advantages of RAG (knowledge updates, interpretability, cost-effectiveness) and the challenges (retrieval quality, latency, context limitations)
  • 3Understand RAG best practices (vector database selection, retrieval strategies, document processing, prompt engineering)
  • 4Able to evaluate the choice between RAG and fine-tuning, and make the right architecture decisions based on the situation
Previous chapter: Mixture of Experts (MoE)Next chapter: other emerging architectures