Internal Policy Q&A Assistant

Overview

Every bank maintains hundreds -- often thousands -- of internal policy documents. Compliance manuals, credit policies, operational procedures, HR handbooks, and board-approved risk limits. When an employee needs to find a specific policy, they face a familiar frustration: searching through SharePoint folders, sifting through outdated documents, and hoping the search terms they chose match the language the policy author used.

A Retrieval-Augmented GenerationRetrieval-Augmented Generation (RAG)A pattern that combines document retrieval with LLM generation. The system searches a knowledge base for relevant context, then feeds it to the model to produce grounded, accurate answers.See glossary (RAG) powered Q&A assistant transforms this experience. Instead of keyword-based search, employees ask natural language questions -- "What is our current commercial real estate concentration limit?" or "What are the approval requirements for a new third-party vendor?" -- and receive accurate, sourced answers with citations back to the authoritative document.

This architecture is the most common first AI deployment in banking because it delivers immediate, visible value with manageable risk. The system does not make decisions -- it retrieves and summarizes existing policies. Human judgment remains the final authority.

Increasingly, the largest banks get policy Q&A inside an enterprise assistant they buy rather than a standalone system they build. Barclays' Colleague AI Agent, part of its Microsoft 365 Copilot rollout to 100,000 colleagues, answers policy-compliance and HR questions; Wells Fargo's rollout of Google's agent platform is aimed partly at helping staff find their way through policies and procedures. Either way, the architecture underneath is the same.

Architecture Components

Document Ingestion Pipeline

The ingestion pipeline converts raw policy documents -- PDFs, Word files, SharePoint pages, Confluence articles -- into formats the AI system can search. This involves document parsing (extracting text from various formats), metadata extraction (document title, version, effective date, owner), and format normalization.

Tools to evaluate: Apache Tika for multi-format parsing, LlamaIndex data connectors for SharePoint and Confluence integration, or managed solutions like Amazon Textract for OCR on scanned documents. A newer generation of generative AI document parsers -- Amazon Bedrock Data Automation and Azure Content Understanding -- reads complex layouts such as tables and forms and feeds the results straight into a RAG pipeline.

Chunking Engine

Raw documents must be split into smaller segments before embeddingEmbeddingsNumerical representations (vectors) of text that capture semantic meaning. Similar concepts produce vectors that are close together, enabling machines to understand relationships between words, sentences, or documents.See glossary generation. ChunkingChunkingThe process of splitting large documents into smaller, overlapping segments before generating embeddings. Chunk size and overlap strategy directly affect retrieval quality in RAG systems.See glossary strategy directly affects retrieval quality -- chunks that are too large dilute relevance; chunks that are too small lose context.

For banking policy documents, semantic chunking (splitting on section boundaries and natural paragraph breaks) outperforms fixed-size chunking. Policy documents have clear hierarchical structures -- chapters, sections, subsections -- and preserving these boundaries produces more coherent retrieval results.

Vector Store

The vector storeVector DatabaseA specialized database optimized for storing and querying high-dimensional vectors (embeddings). Enables fast similarity search across millions of documents for RAG and recommendation systems.See glossary holds the embedded chunks and enables fast similarity searchSemantic SearchSearch that understands meaning rather than just matching keywords. Uses embeddings to find conceptually similar documents even when they use different terminology.See glossary. For banking deployments, the key evaluation criteria are: data residency (where vectors are stored), access control (can you restrict which users see which document vectors?), and scale (how many documents and how fast?).

Options range from managed cloud services (Pinecone, Weaviate Cloud) to self-hosted solutions (pgvector on your existing PostgreSQL, Milvus) to platform-integrated options (Snowflake Cortex Search, Bedrock Knowledge Bases).

LLM Generation Layer

The LLM receives the user's question along with retrieved document chunks and generates a synthesized answer. The system prompt instructs the model to base answers strictly on the provided context, cite specific documents, and clearly state when insufficient information is available to answer.

Model selection balances accuracy, latency, and cost. For internal policy Q&A -- where accuracy is paramount but latency tolerance is moderate (2-5 seconds is acceptable) -- a current frontier model (for example from OpenAI, Anthropic or Google) typically outperforms smaller models on complex policy questions.

Citation and Source Tracking

Every answer must include citations to the specific document, section, and version that supports it. This is non-negotiable in banking -- an unsourced policy answer is worse than no answer at all, because it creates false confidence.

The citation system tracks which retrieved chunks contributed to the answer and maps them back to source documents with direct links to the original content.

Evaluation and Monitoring

A sixth component sits alongside the others: a standing "golden set" of SME-written questions with known correct answers and sources. The set is rerun every time a policy document, the model, or the system prompt changes, scored automatically (often by a second model acting as a judge) and sampled by SMEs. Cloud platforms now build this in -- Amazon Bedrock evaluations, for example, score retrieval quality and answer correctness against expected answers. Think of it as back-testing for the Q&A system: you would not change a credit model without re-running its tests, and the same discipline applies here.

Data Flow

1. Document ingestion: Policy documents are loaded from source systems (SharePoint, network drives, compliance databases), parsed into text, and tagged with metadata (document ID, version, effective date, classification level)

2. Chunking and embedding: Documents are split into semantically meaningful chunks (500-1000 tokens each with 100-token overlap), and each chunk is converted into a vector embedding using an embedding model such as OpenAI text-embedding-3 or Cohere Embed

3. Vector indexing: Embeddings are stored in the vector database alongside the original text chunks and metadata, creating a searchable index of the entire policy corpus

4. User query: An employee submits a natural language question through the Q&A interface -- for example, "What are the documentation requirements for a commercial loan over $5M?"

5. Retrieval: The query is embedded using the same embedding model, and the vector store returns the top-k most semantically similar chunks (typically 5-10 chunks)

6. Context assembly: Retrieved chunks are assembled into a context window, ordered by relevance, and combined with the system prompt instructing the model to answer based only on provided context

7. Answer generation: The LLM generates a synthesized answer, citing specific documents and sections, with confidence indicators for each claim

8. Source verification: The system presents the answer alongside clickable links to the source documents, enabling the user to verify every claim against the original policy text

Banking Use Case

Scenario: A compliance officer at a regional bank receives a question from the commercial lending team: "What is our current policy on commercial real estate concentration limits, and has it changed since the last board review?"

Without the Q&A assistant: The compliance officer opens SharePoint, searches for "CRE concentration" across multiple policy folders, finds three versions of the concentration risk policy (two outdated), cross-references with the most recent board minutes, and drafts a response. This takes 45-90 minutes.

With the Q&A assistant: The compliance officer types the question. The system retrieves the current Concentration Risk Policy (v4.2, effective January 2026), the relevant board resolution from the December 2025 meeting, and the related OCC guidance. Within seconds, it presents: "The current CRE concentration limit is 300% of total risk-based capital, as established in Concentration Risk Policy v4.2 (Section 3.2). This represents a reduction from the previous 350% limit, approved by the Board in Resolution 2025-47 on December 15, 2025." Each citation links directly to the source document.

Key Architectural Decisions