Advanced RAG Patterns for Enterprise

Beyond Basic RAG

The basic RAGRetrieval-Augmented Generation (RAG)A pattern that combines document retrieval with LLM generation. The system searches a knowledge base for relevant context, then feeds it to the model to produce grounded, accurate answers.See glossary pattern -- embed a question, find similar document chunks, pass them to the LLM -- works well for straightforward use cases. But banking demands more. When a compliance officer asks about anti-money laundering requirements for correspondent banking, the system needs to retrieve the right passages from the right documents with high precision. A wrong or incomplete retrieval does not just produce a bad answer -- it produces a confidently wrong answer that could inform a compliance decision.

This unit covers the advanced patterns that elevate RAG from a prototype to a production-grade system suitable for banking workloads.

Pattern 1: Hybrid Search

Hybrid search, now offered by nearly every serious vector database, is one of the most impactful upgrades to basic RAG.

The Problem

Pure semantic searchSemantic SearchSearch that understands meaning rather than just matching keywords. Uses embeddings to find conceptually similar documents even when they use different terminology.See glossary finds documents that are conceptually related to the query. But banking queries often include specific regulatory terms, statute references, or account numbers that must be matched exactly. Semantic search might understand the concept behind "12 CFR 1026.35" but miss a document that specifically references this regulation.

The Solution

Hybrid search runs both keyword-based search (BM25 or similar algorithms) and embeddingEmbeddingsNumerical representations (vectors) of text that capture semantic meaning. Similar concepts produce vectors that are close together, enabling machines to understand relationships between words, sentences, or documents.See glossary-based semantic search simultaneously, then combines the results using a weighted scoring function. The weight balance can be tuned: for regulatory queries heavy in technical terms, weight keyword search higher; for natural-language questions, weight semantic search higher.

Most implementations combine the two result lists with Reciprocal Rank Fusion (RRF), so documents near the top of both lists score highest.

Pattern 2: Reranking

The Problem

Initial retrieval from a vector databaseVector DatabaseA specialized database optimized for storing and querying high-dimensional vectors (embeddings). Enables fast similarity search across millions of documents for RAG and recommendation systems.See glossary uses fast but approximate similarity matching. The top 20 results are "probably relevant," but their ranking may not reflect true relevance to the specific question. Document chunk number 15 might actually be more relevant than chunk number 3.

The Solution

After initial retrieval, pass the results through a reranking model -- a specialized model trained specifically to assess the relevance of a document to a query. Reranking models are slower than vector search (they process each result individually), but they are far more accurate at judging true relevance.

The typical pattern:

1. Retrieve the top 20-50 results from the vector database (fast, approximate)
2. Pass all results through the reranker with the original query (slower, precise)
3. Use the reranker's scores to select the top 3-5 most relevant results
4. Send only these highly relevant results to the LLM for answer generation

Banking Impact

Reranking reduces the risk of the LLM receiving marginally relevant context, a primary cause of hallucinatedHallucinationWhen an AI model generates plausible-sounding but factually incorrect information. A critical risk in banking where inaccurate outputs could lead to regulatory violations or financial losses.See glossary or incorrect answers. In banking, where answer accuracy directly affects compliance and risk decisions, the precision improvement from reranking justifies the additional latency.

Reranking is now usually a configuration choice rather than a separate build: current rerankers handle long and multilingual documents, and some are hosted inside the databases themselves -- Pinecone hosts Cohere Rerank, for example, and Elastic hosts Jina rerankers.

Pattern 3: Query Decomposition

The Problem

Complex banking questions often contain multiple sub-questions: "Compare our current CRE concentration against the OCC's guidance and explain what actions we should take if we exceed the soft limit." This single question requires: (1) retrieving the bank's current CRE exposure data, (2) retrieving OCC guidance on CRE concentration, and (3) retrieving the bank's internal escalation procedures.

The Solution

Query decomposition uses an LLM to break complex questions into simpler sub-questions, routes each to the right index, and then synthesizes the results. This ensures that each aspect of the question gets focused retrieval attention rather than hoping a single query will surface all relevant information.

Pattern 4: Multi-Index Strategy

The Problem

Different document collections have different characteristics and serve different retrieval needs. Your compliance policy manual (structured, versioned, authoritative) is fundamentally different from your customer complaint logs (unstructured, high-volume, time-sensitive). Using a single index and retrieval strategy for both produces suboptimal results.

The Solution

Maintain separate indexes for different document collections, each optimized for its specific characteristics:

• Compliance policies: Section-aware hybrid index with version metadata
• Regulatory guidance: Hybrid search index (keyword + semantic)
• Credit memos: Vector index with department and risk-rating metadata
• Customer communications: Vector index with date-range filtering

A router determines which index or indexes to query based on the nature of the question. Some questions may query multiple indexes simultaneously.

Pattern 5: RAG Evaluation

Why Evaluation Matters for Banking

In banking, you cannot deploy what you cannot demonstrate is accurate. RAG evaluation is the mechanism that gives compliance officers, risk managers, and regulators confidence that the system produces reliable outputs.

Key Metrics

Retrieval metrics:

• Recall: Does the system retrieve all relevant documents? (Missing a critical regulation is unacceptable)
• Precision: Are the retrieved documents actually relevant? (Noise degrades answer quality)
• Mean Reciprocal Rank: Are the most relevant documents ranked highest?

Generation metrics:

• Faithfulness: Is the generated answer supported by the retrieved documents? (Does the LLM hallucinate beyond what the context provides?)
• Relevance: Does the answer actually address the question asked?
• Completeness: Does the answer cover all aspects of the question?

Building an Evaluation Framework

For banking RAG deployments, establish:

1. Ground truth datasets: Subject matter experts create question-answer pairs with identified source documents
2. Automated evaluation: Run new questions through the pipeline and compare results against ground truth
3. Human evaluation: Compliance and business experts regularly review system outputs for accuracy
4. Regression testing: When you change the pipeline (new chunking strategy, different embedding model, updated documents), verify that previously correct answers remain correct
5. Production monitoring: Track retrieval and generation metrics on live queries to detect degradation

Pattern 6: Agentic Retrieval

This is the biggest change in RAG practice over the past year. Instead of a fixed retrieve-then-answer pipeline, an AI agentAgentsAI systems that can autonomously plan and execute multi-step tasks by calling tools, querying data sources, and making decisions without human intervention at each step -- typically within defined permissions and human approval checkpoints.See glossary plans its own searches: it queries several sources, checks whether what it found is sufficient, and searches again if not. It generalises the query decomposition and router patterns above. It is being productised: Azure AI Search offers knowledge bases (generally available for extractive retrieval since April 2026) with agentic query planning still in preview, and Weaviate has its Query Agent.

The banking caveats: more retrieval steps to log and audit, a higher cost per question, and the user's permissions must apply to every search the agent runs, not just the first one.

Pattern 7: GraphRAG

Some questions are about relationships rather than passages: beneficial ownership chains, or aggregating exposure to a counterparty across subsidiaries. GraphRAG combines a knowledge graph of entities and relationships with vector search so the system can follow these multi-hop connections (see the Knowledge-Graph RAG unit). It costs more to build and depends on the quality of entity extraction, so reserve it for questions that genuinely need it.

Pattern 8: Permission-Aware Retrieval

A RAG system that can retrieve every document in its index will happily show a branch employee a board paper, or one business line another's client files -- the risk OWASP calls "Vector and Embedding Weaknesses" (LLM08:2025). The fix, sometimes called security trimming, is to filter results by the user's entitlements at query time, using the bank's identity system, so documents the user may not see are removed before the LLM receives anything. Search services and vector databases increasingly support document-level security filters, but the bank must keep the entitlements in the index in sync with the source systems.

Putting It All Together

A production banking RAG system typically combines multiple advanced patterns:

1. User asks a question
2. Query decomposition breaks complex questions into sub-questions
3. Router determines which indexes to query for each sub-question
4. Permission-aware hybrid search retrieves from the appropriate indexes using both keyword and semantic matching, limited to what the user is entitled to see
5. Reranker re-scores results for true relevance
6. LLM generates an answer from the highest-ranked results with citations
7. Evaluation pipeline monitors retrieval and generation quality continuously

In agentic systems, an agent runs steps 2-5 itself, searching again until it has enough. Very large context windows complement this pipeline rather than replace it: retrieve first, then let a long-context model reason over the shortlist -- accuracy drops and cost rises when whole libraries are stuffed into every prompt.

Each pattern addresses a specific failure mode of basic RAG. Together, they produce a system that banking professionals can trust.