Different tools. RAG injects knowledge that changes often and must be cited. Fine-tuning shapes behavior, format, and tone. If the model doesn't know your facts, that's RAG; if it doesn't answer in your style, that's fine-tuning. Many production systems use both.

How to Build a RAG System in 2026: A Practical Implementation Guide

Q: How much does a RAG system cost to run?

Per query: embedding the query, retrieval, reranking, and generation, with generation the main cost driven by context tokens. A well-built system answers most questions for a fraction of a cent to a few cents. The main lever is how much context you send, which is why a tight reranker pays for itself.

Q: Do I need a dedicated vector database?

Not to start. pgvector inside the Postgres you already run is the right default until scale (roughly 10M+ vectors) or very high QPS forces a specialized store.

TL;DR

RAG is still the cheapest, most defensible way to put a language model to work on your own data. In 2026 the question is rarely “RAG or fine-tuning.” It’s “how good does retrieval have to be before the answer is trustworthy.”
The gap between a weekend demo and a production system is almost entirely in the parts nobody screenshots: parsing, chunking, retrieval quality, evaluation, and monitoring. The generation step is the easy 10%.
A production system is two pipelines, an offline indexing pipeline and a per-request query pipeline, that meet at the vector store. Retrieval quality is set upstream, when documents are parsed and chunked.
Start with a boring, strong default stack: Finigami DocumentAI for parsing, pgvector for storage, hybrid retrieval plus a reranker, and Claude for generation. Add complexity only when an eval tells you to.
You cannot improve what you cannot measure. Build the eval harness before you tune anything. Groundedness and retrieval hit-rate are the two numbers that decide whether this ships.
The hard part of getting it approved isn’t technical. It’s a one-page business case that frames RAG as deflected cost or unlocked revenue, with a cost ceiling and a quality floor.

What RAG is in 2026, and what changed

Retrieval-augmented generation is a simple idea with a deceptively deep tail. Instead of relying on a model’s frozen training data, you retrieve relevant passages from your own corpus at query time and put them in the prompt, so the model answers from your documents rather than from memory. The pattern was introduced by Lewis et al. in 2020 (arXiv:2005.11401), and the architecture has barely changed since. What changed is everything around it.

Three shifts matter for how you build in 2026.

Context windows got huge, and it didn’t kill RAG. Models now take hundreds of thousands of tokens. The “just stuff all your documents in the prompt” crowd predicted RAG’s death every year and was wrong every year, for three reasons. Cost scales with tokens, so stuffing 200K tokens into every request is the most expensive way to answer a question that needed two paragraphs. Latency scales with tokens too. And models still attend unevenly across long contexts. The “lost in the middle” effect, where facts buried in the center of a long prompt get ignored, is well documented (Liu et al., 2023). Retrieval is how you put the right two paragraphs near the top. Long context complements RAG rather than replacing it: it lets you retrieve more generously and rerank harder, not skip retrieval.

Retrieval got better, so the bottleneck moved. Hybrid search (combining dense vectors with keyword search), cross-encoder reranking, and techniques like Anthropic’s contextual retrieval (anthropic.com/engineering/contextual-retrieval) cut retrieval-failure rates substantially. The result: in most failed RAG projects the model is fine and the retrieval is bad. The work moved upstream, to indexing. Concretely, hybrid search recovers the exact-match queries pure vectors miss (a part number, an error code, a person’s name), the reranker fixes the ordering that approximate search gets roughly right, and contextual chunking repairs the meaning that splitting a document strips away. None of these touch the model; all of them move the quality needle more than swapping models does.

Parsing stopped being an afterthought. The single biggest determinant of RAG quality in real enterprises is how well you turned a messy PDF, scanned form, or email thread into clean, structured text. Garbage in, confidently-wrong out. This is the part teams under-budget by 10x. Walk into any stalled RAG project and you’ll usually find the demo was built on a folder of clean text files while production is scanned contracts, multi-column reports, and email threads with quoted history. The naive text extraction that worked on the demo silently drops a chunk of the real corpus into garbage, and the team then spends weeks tuning prompts and rerankers to rescue an answer that was already lost at parse time.

The anatomy of a RAG failure

When a RAG answer is wrong, it failed at one of four points. Naming them matters, because the fix is different at each:

Parsing: the fact never made it out of the document intact. A table became gibberish, or a scanned page was skipped. No downstream step can recover what was lost here.
Chunking: the fact survived parsing but got split from the context that makes it findable. Retrieval can’t match a query the chunk no longer resembles.
Retrieval: the right chunk exists and is findable, but ranked below the cutoff. Hybrid weighting and the reranker are the levers.
Generation: the right chunk was retrieved and the model still answered wrong. It ignored the context, over-claimed, or hallucinated despite being grounded.

This guide spends most of its words above the generation step on purpose: three of the four failure points are upstream of the model. Teams instinctively blame the model, since it’s the visible part, and tune prompts for a problem that actually lives in parsing or chunking. Your eval’s entire job is to tell you which of the four it is, so you fix the right box.

When NOT to build RAG

Skip RAG, or descope it, if any of these is true:

The knowledge fits in a system prompt and rarely changes. If it’s 5,000 tokens of policy that updates quarterly, put it in the prompt and move on. RAG is infrastructure; don’t build infrastructure for a constant.
You need exact, structured answers from structured data. “What was Q3 revenue for the EMEA region?” is a SQL question, not a retrieval question. Use a text-to-SQL agent against the warehouse. RAG over a spreadsheet is an anti-pattern.
The task is reasoning, not recall. RAG grounds answers in documents. It doesn’t make a model better at multi-step math or planning.
You can’t articulate the question distribution. If nobody can tell you the ten questions users will actually ask, you can’t build an eval set, which means you can’t tell if retrieval works. Fix that first.

RAG vs the alternatives

Before you build, make sure RAG is the right tool. There are four ways to get knowledge into a model’s answer, and they work as layers rather than rival camps:

Approach	Gets the model…	Best when	Weakness
Prompt-stuffing	a fixed block of context, every call	knowledge is small and static	cost and latency scale with size; lost-in-the-middle
RAG	the relevant slice, per query, cited	corpus is large, volatile, needs provenance	quality is bounded by retrieval
Fine-tuning	new behavior, format, tone	you need style/structure, not fresh facts	doesn’t add citable, changing knowledge
Long-context	a big block of documents in one call	a handful of documents per task	expensive per call; uneven attention
Agentic retrieval	the ability to search in a loop	open-ended research across many sources	slower, costlier, harder to evaluate

Most production systems are RAG, with a fine-tuned generation step for format and a long-context window used to retrieve more generously. These are three layers of one system, not three competing approaches. The mistake is adopting one as ideology. Pick by the question your users actually ask: if it’s “what does our documentation say about X,” that’s RAG; if it’s “answer every email in our house style,” that’s fine-tuning; if it’s “reason across these five contracts I just uploaded,” that’s long-context. For the executive framing of this trade-off, see the AI build-vs-buy decision.

Use cases & where it pays off

RAG earns its keep when three conditions hold: the answer lives in a corpus too large or too volatile for the prompt, the questions are mostly lookup rather than reasoning, and being wrong is expensive enough to justify grounding. The patterns that consistently pay back are these.

Use case	Why RAG fits	What “good” looks like
Internal knowledge assistant (policies, runbooks, wikis)	Corpus is large, changes weekly, answers must cite source	Deflects repetitive questions from senior staff; every answer links to the doc
Customer support copilot	Agents need the right KB article in 2 seconds, grounded	Lower handle time; the agent, not the customer, holds the AI
Contract / document Q&A	Documents are long, parsing-heavy, high-stakes	Correct clause retrieval with citations a lawyer can verify
Field/technical support (manuals, specs)	Answers are in 800-page PDFs nobody reads	Right page, right revision, with the figure
Research / analyst assistant	Synthesis across many sources with provenance	Cited synthesis, not a confident hallucination

The common thread: citations are the product. A RAG answer without a verifiable source is worth less than a search result, because it looks authoritative while being unfalsifiable. If you take one design principle from this guide, take that one.

A worked example, the support copilot. A customer asks, “can I return an opened blender after 40 days?” The agent’s copilot embeds the question, runs hybrid retrieval against the returns-policy corpus filtered to the agent’s region, reranks to the top eight chunks, and prompts the model to answer only from them with citations. Back comes: “Opened items are returnable within 30 days; this order is past that window [returns-policy-2025 p.2]. Manager discretion applies to items under $50 [returns-policy-2025 p.6].” The agent reads it, sees the cited pages, and answers in fifteen seconds instead of three minutes of searching. And if the policy hadn’t covered it, the copilot would have said so rather than inventing a window. Every box in the architecture diagram earned its place in those fifteen seconds.

The reference architecture

A production RAG system is two pipelines, not one. An indexing pipeline runs offline and turns your documents into a searchable index. A query pipeline runs on every request and turns a question into a grounded, cited answer. They meet at the vector store. Most failed RAG projects over-invest in the query side and under-invest in indexing, but retrieval quality is set upstream, when documents are parsed and chunked.

The canonical RAG reference architecture: an offline indexing pipeline and a per-request query pipeline meeting at the vector store, with evaluation and observability across both.

Read the diagram left to right, top then bottom. The top lane runs on a schedule or on document change. The bottom lane runs on every user request, in well under two seconds if you’ve built it right. The two boxes that decide your quality ceiling are Parse & OCR (top) and Rerank (bottom), and both are the ones teams skip in the demo.

Architecture decisions

Six decisions determine 90% of the outcome. Make them deliberately; everything else is tuning.

Build vs. buy the whole thing. Managed “RAG-as-a-service” platforms will get you a demo in an afternoon. They will also own your retrieval quality, your data perimeter, and your cost curve. The rule: buy the components (parsing, embeddings, reranking, the model), build the pipeline. A hosted black box is fine for a prototype and a liability for a system you’ll tune for a year.
Where the documents get parsed. This is the decision that silently sets your quality. Naive text extraction (pdf-to-text) drops tables, scrambles multi-column layouts, and ignores scanned pages entirely. For anything beyond clean HTML, use a real document-understanding API. We default to Finigami DocumentAI: it’s API-first, template-agnostic, and handles scanned and multi-language documents without per-form configuration, which is exactly the failure mode (forms, statements, contracts) where naive parsing falls apart. Alternatives: AWS Textract, Google Document AI, or open-source Unstructured if you want to own the pipeline and have the team to maintain it.
Vector store. Start with pgvector (github.com/pgvector/pgvector) unless you have a reason not to. It keeps your vectors next to your relational data and metadata, supports HNSW indexes (Malkov & Yashunin, 2016) for fast approximate search, and means one fewer system to operate. Reach for a dedicated store (Pinecone, Qdrant, Weaviate) when you cross ~10M vectors, need multi-region replication, or want filtered search at very high QPS.

Option	Use when	Watch out for
pgvector (default)	< ~10M vectors, you already run Postgres	Index build time at scale; tune `hnsw` params
Pinecone	Managed, high QPS, you don’t want to operate it	Cost at scale; data leaves your perimeter
Qdrant / Weaviate	Self-hosted, advanced filtering, hybrid built-in	Another system to run and back up

The operational reality with pgvector that tutorials skip: HNSW build time grows with corpus size, and the m and ef_construction parameters trade build time and memory against recall. Start with defaults, measure recall against your golden set, and tune only when an eval shows misses that look like index-recall problems rather than ranking problems. The underrated payoff of staying in Postgres is that access tags, document metadata, and vectors all live in one place, so “only this user’s region, only current documents” is one SQL WHERE clause, not a cross-system join.

Embedding model. Pick by your actual domain, not the leaderboard headline. Check the MTEB leaderboard (huggingface.co/spaces/mteb/leaderboard) but validate on your queries. Defaults that hold up: Voyage (strong retrieval quality) or OpenAI text-embedding-3-large. Two constraints matter more than the last point of benchmark accuracy: dimensionality (drives storage and search cost) and whether you can re-embed your whole corpus when you upgrade, because you will, and a model swap means a full re-index. Validate two or three candidates on a sample of your own queries before committing; leaderboard rank rarely survives contact with a specific domain’s vocabulary. And size has operational weight: a 1024-dimension model stores and searches faster than a 3072-dimension one, and the quality gap is often inside the noise for a retrieve-then-rerank pipeline, where the reranker cleans up what the embeddings only approximate.
Retrieval strategy. Default to hybrid: dense vector search for semantic matches plus BM25 keyword search for exact terms (names, IDs, error codes), fused with reciprocal rank fusion, then a cross-encoder reranker over the top candidates. Pure vector search misses exact-match queries; pure keyword misses paraphrases. Hybrid is strictly better for the cost of slightly more plumbing.
Generation model. Default to Claude: Opus 4.8 for the hardest synthesis and citation-faithfulness work, Sonnet 4.6 where latency and cost matter more and the retrieval has already done the heavy lifting. The model is the least important decision in the system: once retrieval is good, most capable models produce a good grounded answer. Spend your optimization budget upstream. The one model behavior worth testing for explicitly is citation faithfulness, whether it actually attributes claims to the right source, because that’s the property your whole credibility story rests on.
Index freshness. Decide up front how stale an answer is allowed to be. A policy assistant that cites last quarter’s rules is a liability; a research tool over historical filings can tolerate a nightly rebuild. The answer sets your indexing architecture, event-driven incremental updates for volatile corpora and scheduled rebuilds for stable ones, and it is far cheaper to design in now than to retrofit. Store an indexed_at per chunk so freshness is observable and alertable, not a thing you discover when a user quotes a dead policy back to you.

The reference stack

The boring, strong default. Start here; deviate only when an eval tells you to.

Layer	Default pick	Why	Swap when
Parse / OCR	Finigami DocumentAI	API-first, template-agnostic, scanned + 50+ languages	You have clean HTML only
Chunking	structure-aware (headings/sections)	Preserves semantic boundaries	Highly uniform docs → fixed-size is fine
Embeddings	Voyage / OpenAI `text-embedding-3-large`	Strong retrieval, well-supported	Domain-specific → fine-tune or specialized model
Vector store	pgvector	Co-located with metadata, one less system	> ~10M vectors → Pinecone/Qdrant
Retrieval	hybrid (vector + BM25) + RRF	Catches semantic and exact matches	-
Reranker	cross-encoder (e.g. Cohere Rerank)	Biggest quality-per-dollar lever	Latency-critical → smaller reranker
Generation	Claude Opus 4.8 / Sonnet 4.6	Faithful, good at citations	-
Eval / observability	Langfuse + offline test sets	Traces + scores in one place	Enterprise → Braintrust
Orchestration	thin app code (no heavy framework)	RAG is a pipeline, not an agent	Multi-step → LangGraph

A note on frameworks: you do not need a heavyweight orchestration framework to build RAG. It’s a linear pipeline. A few hundred lines of well-tested application code is more debuggable than a framework you don’t control, and the framework’s abstractions tend to leak exactly where you need to tune. Add orchestration when the workflow becomes genuinely agentic (tool use, loops, branching), not before.

Cost and latency, end to end

Two numbers decide whether a RAG system is viable in production: what each answer costs and how long it takes. Both are dominated by the same lever, how much context you send the model, which is why the reranker earns its keep twice over. A rough per-request profile for a well-built system answering an 8-chunk question:

Stage	Latency (typical)	Cost driver
Embed query	~20–50 ms	a few hundred tokens, negligible
Vector + BM25 search	~10–50 ms	database time
Fuse (RRF)	< 5 ms	free, in-process
Rerank ~50 candidates	~100–300 ms	reranker call, small per query
Generate	~500 ms – 2 s	the driver: input context + output tokens
Total	well under 2 s	generation dominates both

Read it as a budget. Over on latency? The fix is almost always fewer, better chunks (tighter reranking) or a faster model for the easy questions, not a faster database. Over on cost? Same lever: every chunk you send that the answer didn’t need is tokens billed on every request, forever. A system that retrieves 20 chunks “to be safe” pays 2–3× per query for worse answers, because the noise also degrades quality. Precision is cheaper and better at once, one of the rare places those align.

Implementation

The build, step by step, with the decisions that bite at each one.

Step 0: Scope the corpus and the questions

Before any code: pick one document set and write down the ten to twenty questions users will actually ask it. This isn’t busywork. It’s what makes every later step measurable. The questions become your first golden set; the document set bounds your parsing problem; and the pair tells you whether RAG is even the right tool (if the questions are mostly aggregations over structured data, stop and build a text-to-SQL agent instead). Teams that skip this start tuning a system they can’t grade, and discover the mismatch a month in.

Step 1: Parse documents into clean, structured text

Everything downstream inherits the quality of this step. Run source documents through Finigami DocumentAI and keep the structure, not just the text: headings, tables as tables, page numbers, and section hierarchy. You’ll need that structure for chunking and, later, for citations that point a user to the right page.

Two rules. First, preserve provenance: every extracted block should carry its source document ID, page, and section. Citations are the product; you can’t cite what you didn’t track. Second, normalize without mangling: fix encoding and whitespace, but don’t strip tables into prose. A table flattened into a paragraph is a retrieval landmine.

A schematic of the parse step, noting that every block keeps its origin:

# Parse with a document-understanding API; keep structure AND provenance.
# (Calls are schematic. See the Finigami DocumentAI docs for exact endpoints.)
doc = documentai.parse(file="statements/q3-emea.pdf")   # scans, tables, 50+ languages

blocks = []
for page in doc.pages:
    for el in page.elements:                 # headings / paragraphs / tables kept distinct
        blocks.append({
            "text":      el.as_markdown(),    # a table stays a table, not flattened prose
            "type":      el.type,             # "heading" | "paragraph" | "table"
            "source_id": doc.id,
            "page":      page.number,
            "section":   el.section_path,     # e.g. "3 / International orders"
        })

Those metadata fields aren’t optional bookkeeping. source_id, page, and section are what later become the clickable citation; type is what lets the chunker avoid splitting a table from its header. Skip them here and you cannot add them back later, because the structure is gone.

Step 2: Chunk with structure, not a fixed character count

The default advice, “split every 500 characters with 50 overlap,” is where mediocre RAG comes from. It cuts sentences in half and splits tables from their headers. Chunk on semantic boundaries: sections and subsections, with a target size range rather than a hard count. Keep each chunk self-contained enough to answer a question, small enough to be precise.

The biggest upgrade here is contextual chunking: prepend a short, model-generated description of where each chunk sits in its document (“This is from the 2025 refund policy, section 3, on international orders”) before embedding. Anthropic’s contextual retrieval write-up (anthropic.com/engineering/contextual-retrieval) reports large reductions in retrieval failures from exactly this move. It costs a one-time LLM pass over the corpus at index time and it is usually the single best quality investment after parsing.

def chunk(blocks, target_tokens=(500, 1200)):    # a range, not a hard count
    for section in group_by_section(blocks):      # keep each table with its heading
        for piece in split_on_boundaries(section, target_tokens):
            yield piece

def contextualize(piece, doc_summary):
    # one cheap LLM call per chunk, at index time only, never at query time
    prefix = llm(f"In one sentence, situate this excerpt within the document.\n"
                 f"Document: {doc_summary}\nExcerpt: {piece.text}")
    return prefix + "\n\n" + piece.text           # embed THIS; store both versions

Why it works, concretely. A bare chunk reading “Orders over $500 require manager approval” is ambiguous: which policy, which year, which region? The contextualized version, “From the 2025 Refund Policy, section 3 (International orders): orders over $500 require manager approval,” retrieves correctly for the query “what’s the approval threshold for overseas refunds this year.” Same underlying text; far better recall, because the embedding now carries the context the query implies.

Step 3: Embed and index

Embed each chunk with your chosen model and write it to pgvector alongside its metadata (source, page, section, timestamps, access tags). Build an HNSW index for fast approximate nearest-neighbor search. Store the raw chunk text too, since you’ll return it to the model and to the citation.

Index the same chunks into a keyword index (Postgres full-text or a dedicated BM25) for the hybrid retrieval in Step 4. Yes, you store the text twice. Storage is cheap; retrieval misses are not.

The pgvector schema is deliberately boring, vectors sitting next to the metadata that powers filtering and citations:

CREATE EXTENSION IF NOT EXISTS vector;

CREATE TABLE chunks (
    id          bigserial PRIMARY KEY,
    source_id   text         NOT NULL,
    page        int,
    section     text,
    content     text         NOT NULL,        -- returned to the model AND the citation
    embedding   vector(1024),                 -- dimension is fixed by your model
    access_tags text[]       DEFAULT '{}',    -- enforced at retrieval time (Step 4)
    indexed_at  timestamptz  DEFAULT now()
);

-- Approximate nearest-neighbour index (HNSW): fast recall at scale.
CREATE INDEX ON chunks USING hnsw (embedding vector_cosine_ops);

-- Keyword index for the BM25 half of hybrid retrieval.
CREATE INDEX ON chunks USING gin (to_tsvector('english', content));

Two design notes. vector(1024) is pinned to your embedding model: change models and you re-embed and re-index the whole corpus, which is exactly why the embedding decision is sticky. And access_tags lives on the row, so permission filtering happens inside the SQL query rather than as a post-hoc filter the model could be tricked into leaking around.

Step 4: Retrieve, fuse, and rerank

On each query: run dense vector search and BM25 in parallel, fuse the two ranked lists with reciprocal rank fusion, take the top ~50 candidates, and pass them to a cross-encoder reranker that scores each against the query directly. Keep the top 5–10 after reranking. The funnel (cast wide, then narrow with a precise reranker) is what separates production retrieval from a demo.

Cast a wide net with cheap retrieval, then narrow hard with a precise reranker. The reranker is the highest quality-per-dollar lever in the query pipeline.

The whole query path, with access control baked into the SQL so it can’t be bypassed:

def retrieve(query, user, k=200, fuse_to=50, final_k=8):
    q_vec = embed(query)
    # Both halves respect the user's access tags. Filtering happens IN the query.
    dense = sql("""SELECT id, content, source_id, page, section FROM chunks
                   WHERE access_tags && %(tags)s
                   ORDER BY embedding <=> %(qv)s LIMIT %(k)s""",     # <=> = cosine distance
                qv=q_vec, tags=user.tags, k=k)
    bm25 = sql("""SELECT id, content, source_id, page, section FROM chunks
                  WHERE access_tags && %(tags)s
                    AND to_tsvector('english', content) @@ plainto_tsquery(%(q)s)
                  LIMIT %(k)s""", q=query, tags=user.tags, k=k)
    fused = reciprocal_rank_fusion(dense, bm25)[:fuse_to]
    return cross_encoder_rerank(query, fused)[:final_k]            # the precision step

def reciprocal_rank_fusion(*ranked_lists, k=60):
    scores = defaultdict(float)
    for lst in ranked_lists:
        for rank, row in enumerate(lst, start=1):
            scores[row.id] += 1.0 / (k + rank)                     # standard RRF
    return [row for row, _ in sorted(scores.items(), key=lambda x: -x[1])]

The shape that matters: retrieve ~200 candidates per method (cheap), fuse to ~50 (free), rerank to ~8 (the only step that costs real latency, and the one that buys most of the precision). Tune final_k against your eval. More context isn’t better, it’s just more expensive and more room to get lost in the middle.

Step 5: Generate with a grounded, citation-forcing prompt

Assemble the reranked chunks into the prompt with their source metadata, and instruct the model to answer only from the provided context, to cite the source of each claim, and to say “I don’t know” when the context doesn’t contain the answer. That last instruction is not a nicety. A RAG system that confidently answers from missing context is worse than search, because it launders a guess as a citation-backed fact. Put the most relevant chunks near the top of the context to dodge the lost-in-the-middle effect, and return the citations to the UI so the user can verify.

The prompt does three jobs: constrain to context, force citations, and license abstention.

Answer the question using ONLY the context below.
- Cite the source of every claim as [source_id p.PAGE].
- If the context does not contain the answer, reply exactly:
  "I don't know based on the available documents."
- Do not use outside knowledge.

Context:
[1] (refund-policy-2025, p.4, §3) Orders over $500 require manager approval...
[2] (refund-policy-2025, p.5, §3.1) Approved international refunds process in 7 days...

Question: What's the approval threshold for overseas refunds, and how long do they take?

A correct grounded answer then looks like this, every claim traceable to a page a human can open:

Overseas orders over $500 require manager approval [refund-policy-2025 p.4]. Once approved, the refund is processed within 7 days [refund-policy-2025 p.5].

If the context hadn’t contained the threshold, the only acceptable output is the abstention string, not a plausible-sounding number. Wire the citations through to the UI as links to the source page; that link is what converts “an AI said so” into “here’s the policy, verify it yourself.”

Two production details close out the step. Stream the answer so the user sees it forming. For an assistant, perceived latency matters as much as the real number. And resolve citations after generation: have the model emit citation markers, then map them to real source links in your application code, so a malformed citation becomes a caught error instead of a broken link in front of a user.

Verifiability: returning the sources you used

A RAG answer is only as trustworthy as a reader’s ability to check it, which makes verifiability a system requirement rather than a UI nicety. It is decided by something you did back in Step 1: storing source metadata with every chunk.

The mechanism is simple, and it has to be deliberate. Each chunk carries its document_id, page, and section, plus a stable title. Those fields ride along through retrieval, so when the model writes an answer your application can map each cited chunk back to a real, openable location. The model emits a marker like [returns-policy-2025 p.4]; your code resolves that marker to a link to page 4 of that document. The user clicks, lands on the exact page, and reads the source themselves.

Three rules make this hold up:

Return the sources the answer actually used, not everything retrieved. You retrieved eight chunks; the answer may have used three. Cite the three. A citation list padded with chunks the model ignored teaches users to distrust the citations, because half of them won’t contain the claim.
Resolve each citation to the smallest verifiable unit you can. A document-level link (“see the returns policy”) is weak; a page-and-section link is strong; a link that highlights the exact passage is strongest. The granularity of your stored metadata sets the ceiling on how precise the citation can be.
Keep the chunk text you stored, so you can show the exact passage on hover or click. Retrieval returns an ID; the stored content is the evidence you render next to the claim.

This is why the metadata is not optional bookkeeping. You cannot reconstruct a page number or a section path after the fact if you discarded it at parse time. Verifiability is designed in at ingestion, or it is not available at all. It is also what turns an answer a user has to trust into one they can audit, which is the difference between a toy and a tool in any regulated setting.

Chunking strategies, compared

Chunking is the decision that most quietly determines retrieval quality, and the one most teams make by copying a tutorial. Here’s the actual field, worst to best for typical enterprise documents:

Strategy	How it splits	Best for	The catch
Fixed-size	every N tokens, M overlap	uniform, structure-less text	shreds tables and sentences mid-thought
Recursive	on separators (paragraph → sentence)	general prose	still blind to document structure
Semantic	where the topic shifts (embedding deltas)	dense, mixed-topic documents	extra compute at index time
Structure-aware	on headings / sections	anything with real structure	only as good as your parsing
Contextual (+)	structure-aware + an LLM context prefix	high-stakes recall	one LLM pass over the corpus

The decision rule is short. If your corpus is genuinely uniform (log lines, product descriptions of identical shape), fixed-size is fine and fast. For everything else (policies, contracts, manuals, knowledge bases) default to structure-aware chunking, then add the contextual prefix once your eval shows retrieval missing on questions whose answer is in the corpus. That missing-but-present failure is the signal that your chunks have lost the context the query needs, and contextualization is the targeted fix.

Two parameters still matter inside structure-aware chunking. Size: too small and a chunk can’t answer a question on its own; too large and you retrieve noise and pay for it in tokens. A range of roughly 500–1,200 tokens is a sane starting band; then let the eval move it. Overlap: a little (10–15%) hedges against a fact landing exactly on a boundary; a lot wastes storage and returns near-duplicates that crowd out diversity in the top-k. Tune both against the golden set, one at a time.

Metadata: the other half of retrieval

Embeddings find chunks that are semantically similar. Metadata finds chunks that are contextually correct. You need both, and teams consistently under-invest in the second. The failure looks like this: a user in Germany asks about parental leave, retrieval returns a semantically perfect answer from the US HR policy, and the system confidently cites the wrong country’s rules. The embedding did its job. The metadata that would have filtered to Germany was never stored.

The rule: store the metadata your domain uses to disambiguate, and filter on it before or during retrieval. What that metadata is depends entirely on the document type.

Legal contracts need effective_date, parties, jurisdiction, contract_type, and a status (active or superseded). A clause means nothing without knowing which contract, which version, and between whom.
Company policy needs effective_date, valid_until, policy_owner, and region. A policy that expired last quarter should never be retrieved as current, and a valid_until in the past is the cleanest way to exclude it.
Multi-context duplicates are the sharp case. When the same kind of document exists in several variants, an HR policy that differs by country or a price list that differs by segment, the distinguishing field (country, segment) is mandatory. Without it, retrieval cannot tell two near-identical documents apart, and semantic similarity will happily hand back the wrong one.

In practice this means a richer chunk row. Extend the schema from Step 3 with the fields your corpus needs, plus a JSONB column for the long tail:

ALTER TABLE chunks
  ADD COLUMN doc_title      text,
  ADD COLUMN effective_date date,
  ADD COLUMN valid_until    date,
  ADD COLUMN region         text,             -- e.g. 'DE' / 'US' for country-specific policy
  ADD COLUMN meta           jsonb DEFAULT '{}';   -- domain long tail: parties, jurisdiction...

-- Retrieval with hard context filters applied BEFORE the vector search.
SELECT id, content, source_id, page, section
FROM chunks
WHERE region = %(region)s
  AND (valid_until IS NULL OR valid_until > now())
  AND access_tags && %(tags)s
ORDER BY embedding <=> %(qv)s
LIMIT %(k)s;

Two operating notes. Metadata filtering is cheap and high-leverage: a WHERE region = 'DE' AND valid_until > now() clause removes whole classes of wrong-context error before the vector search even runs. And decide which fields are hard filters (region, validity) and which are soft signals (recency as a ranking boost). Hard filters protect correctness; soft signals tune relevance. Confusing the two is how you either leak a wrong-context answer or over-filter yourself into a needless “I don’t know.”

Advanced retrieval: when to add complexity

The hybrid-plus-reranker pipeline in Step 4 handles most cases. When your eval shows a specific failure class it can’t crack, reach for exactly the technique that addresses that class, then re-measure. The catalogue, each gated on a signal:

Query rewriting / expansion: rewrite a terse or pronoun-laden query into a fuller one before retrieval. Reach for it when conversational follow-ups (“and for international?”) retrieve badly because they’re incomplete on their own.
Multi-query: fan one question into several paraphrases, retrieve for each, union the results. Reach for it when the same concept is phrased many ways in your corpus and single-query recall is lumpy.
HyDE (hypothetical document embeddings): ask the model to draft a hypothetical answer, embed that, and retrieve against it. Reach for it when queries and documents use very different vocabulary (a user’s plain-English question vs. dense technical source text).
Metadata pre-filtering: narrow by structured fields (date, product, region, document type) before vector search. Reach for it when the corpus spans versions or tenants and you’re retrieving the right topic from the wrong document.
Parent-document / small-to-big: retrieve on small precise chunks but feed the model their larger parent section. Reach for it when precise chunks retrieve well but lack enough surrounding context to answer fully.
Multi-hop retrieval: retrieve, let the model identify what’s still missing, retrieve again. Reach for it when questions genuinely require synthesizing facts that live in separate documents.

Every one of these adds latency, cost, and a new thing to debug. The discipline from the complexity section applies verbatim: add the one your eval demands, measure the delta, keep it only if it earns its place.

Expanding context at retrieval time

A precise chunk retrieves well and sometimes answers poorly, because the fact it contains needs surrounding context to be useful. Three techniques widen the context you hand the model without widening what you search over. They are not mutually exclusive, and which ones help depends on your documents.

Store a document or chapter summary with each chunk. Alongside each chunk, keep a short summary of its document and section, generated once at index time, and prepend it when you assemble the prompt. Now the model always knows a one-sentence chunk is “from the 2025 EMEA refund policy, section 3,” even when the chunk itself carries no such signal. This is the cheapest way to preserve large-document context, it pairs naturally with the contextual prefixes from the chunking step, and it lives in metadata, so it costs storage rather than query latency.

Parent-document retrieval, or small-to-big. Index and search on small, precise chunks, but feed the model their larger parent section. You get the precision of small chunks for matching and the completeness of large chunks for answering. This helps most when the answer spans more than the matched sentence, which is common in manuals and policies.

Adjacent-chunk retrieval, the previous and next chunk. When a chunk matches, also pull the one immediately before and after it. Whether this helps is entirely a function of document type. For sequential, narrative documents (contracts, procedures, manuals) where meaning flows across boundaries, adjacent chunks recover context that a hard split removed, and they help a lot. For collections of independent units (FAQ entries, product records, unrelated tickets) the neighbours are unrelated, and they only add noise. It is a one-line change to add and a measurable one to keep, so test it against your eval before committing.

The throughline: expand context deliberately, with the smallest addition that fixes the failure your eval shows, never by simply raising top-k, which adds noise and cost without the structure these techniques bring.

Scaling past the prototype

What works at 100K vectors needs rethinking at 10M+. The changes that matter, in the order they bite:

Storage and index. pgvector with a tuned HNSW index carries you a long way, but past roughly 10M vectors, or under high concurrent QPS, move to a store built for it (Qdrant, Pinecone) and tune the ANN parameters (m, ef_construction, ef_search) for your recall-vs-latency trade. Higher ef_search buys recall at the cost of latency; pick the point your eval and your p95 budget agree on.
Re-indexing without downtime. A model upgrade or a chunking change means rebuilding the whole index. Do it blue-green: build the new index alongside the live one, validate it against the golden set, then cut over atomically. Never mutate the index users are querying.
Freshness and incremental indexing. Most corpora change continuously. Index on document-change events, not a nightly full rebuild, and store indexed_at so you can prove (and alert on) staleness. A confident citation of a superseded document is a quiet but real failure.
Caching. Two cheap wins: cache query embeddings for repeated questions, and cache full answers for high-frequency identical queries (with a short TTL and an invalidation hook when the underlying documents change). For a support corpus where the same ten questions are 40% of traffic, an answer cache pays for itself immediately.
Latency budget. Decompose your p95: embedding + dense search + BM25 + fusion + rerank + generation. The two that dominate are rerank (more candidates = more latency) and generation (more context = more latency). When you’re over budget, the fix is almost always fewer, better chunks, which is the reranker’s whole job.

Security, access control & the data perimeter

RAG touches your most sensitive documents, so security is a design constraint from line one, not a phase you bolt on before launch.

Access control belongs in the query. If documents have permissions, retrieval must enforce them. Store each chunk’s access tags on the row and filter inside the SQL (the access_tags && %(tags)s clause in Step 4) so a user can never retrieve, and therefore the model can never surface, a document they couldn’t open directly. Never rely on the model to keep a secret it was handed; a prompt can talk it back out. The only safe place to enforce a permission is before retrieval returns the chunk.

Decide your data perimeter explicitly. Three things leave your environment in a RAG system unless you stop them: document text (to the parser and the model), embeddings, and prompts. For each, decide what’s allowed to go where, and write it down. The options run from fully managed APIs (fastest to ship, data leaves your VPC) to self-hosted models and stores (slowest to stand up, nothing leaves). Most enterprises land in between: a managed model under a zero-retention agreement, with parsing and storage inside their own perimeter. Finigami DocumentAI’s API-first model fits that middle path; so does running pgvector in your own database. For the executive version of this conversation, see MCP and your data perimeter.

Audit what the system did. In regulated settings you will be asked why the assistant answered as it did. Because you already trace every request (query, retrieved chunks, prompt, answer, citations) you can answer that, and retain the evidence. Build retention and redaction into the trace store from the start; adding it after an incident is the expensive path.

PII and the index. Decide whether sensitive fields belong in the index at all. Often the right move is to redact at parse time, so the embedding and the retrievable text never contain the account number the source document did. You cannot leak what you never indexed.

Complexity management

RAG systems rot in a predictable way: every quality problem looks like it needs a new component, and a year later you have a Rube Goldberg pipeline nobody can debug. Resist it.

Defer everything until an eval demands it. Query rewriting, multi-hop retrieval, knowledge graphs, agentic retrieval loops: all real techniques, all premature until your eval shows the simple pipeline failing on a specific class of question. Add the component that fixes that class, measure, keep or revert.
One change at a time. Tuning RAG is empirical. If you change the chunker and the reranker together and the score moves, you’ve learned nothing. Change one variable, re-run the eval, record the delta.
Keep the pipeline linear as long as you can. The moment you add branching (“if the query is type A, do X”), you’ve doubled your test surface. Sometimes necessary; never free.
Version your index. A re-embed or a chunking change produces a different index that will score differently. Treat the index as a versioned artifact tied to the config that built it, so you can roll back a bad change.

A concrete version of how this goes wrong: a team sees the assistant miss on a multi-part question, concludes it needs a knowledge graph, and spends two months building entity extraction and graph retrieval. The eval barely moves, because the real failure was a chunking bug splitting the answer across two chunks, fixable in an afternoon. Now the graph is a permanent maintenance burden, bought to solve a problem it never had. An eval run on day one would have named the actual cause.

The discipline is the same one that keeps any system maintainable: the cost of a component is not building it, it’s owning it forever. (For the executive framing of this trade-off, see the build-vs-buy decision for AI.)

Evaluation & quality

This is the section that decides whether you have a product or a science fair project. Build the eval harness before you tune anything. Without it, you are adjusting parameters and asking the room whether the answers “feel better.” With it, every change has a number attached.

RAG has two failure surfaces, and you must measure both separately, because the fixes live in different parts of the pipeline:

Retrieval quality: did the right chunks come back? If retrieval fails, no model can save the answer. Measure with retrieval hit-rate (is the gold chunk in the top-k?) and, where you have ranked relevance labels, recall@k. When retrieval is the problem, the fix is upstream: parsing, chunking, hybrid weighting, the reranker.
Answer quality: given good retrieval, was the answer correct, grounded, and complete? Measure groundedness/faithfulness (is every claim supported by the retrieved context, or did the model invent?), answer relevance, and citation correctness. When the answer is the problem despite good retrieval, the fix is the prompt or the model.

How to actually run it:

Build a golden test set. 100–200 real questions with known correct answers and the source chunk(s) that contain them. This is the single most valuable asset in the project. Mine it from real user questions, support tickets, and SMEs. Without it you are flying blind; with it you have a regression suite.
Score automatically. Use an eval framework (RAGAS (docs.ragas.io) is the common open-source choice) plus an LLM-as-judge for groundedness and relevance, where a strong model grades each answer against the retrieved context and the gold answer. Calibrate the judge against human labels on a sample so you trust it.
Gate releases on it. Every change to the pipeline runs the eval. A change that improves one metric while quietly regressing another (better relevance, worse groundedness) is exactly what the harness exists to catch.

The harness is small. Each row knows the question, the gold answer, and the chunk that should be retrieved:

def evaluate(test_set, pipeline):
    rows = []
    for ex in test_set:
        retrieved = pipeline.retrieve(ex.question)
        answer    = pipeline.generate(ex.question, retrieved)
        rows.append({
            "hit":          ex.gold_chunk_id in {c.id for c in retrieved},  # retrieval
            "groundedness": judge_grounded(answer, retrieved),             # 0–1, LLM-judge
            "relevance":    judge_relevant(answer, ex.gold_answer),        # 0–1, LLM-judge
            "citations_ok": citations_resolve(answer, retrieved),          # bool
        })
    return aggregate(rows)   # → hit-rate, mean groundedness, mean relevance, citation acc.

The groundedness judge is itself a prompt; keep it blunt: “Score 0–1: is every claim in the ANSWER supported by the CONTEXT? Penalise any claim not traceable to the context. Return only the number.” Calibrate it against human labels on ~50 examples before you trust it.

Read the output as a diagnosis, not a grade. A run that reads hit-rate 0.91, groundedness 0.96, relevance 0.88, citation accuracy 0.94 tells a specific story: retrieval finds the gold chunk 91% of the time (chase the last 9% in chunking, not the model); the model isn’t inventing (0.96); but relevance at 0.88 means some answers are correct-but-incomplete, a prompt problem rather than a retrieval one. Each metric points at a different box in the pipeline. That’s the whole reason you measure them separately.

The four metrics, precisely:

Retrieval hit-rate / context recall: of the chunks needed to answer, how many were retrieved. This bounds everything downstream; if recall is low, no prompt or model can fix it.
Context precision: of the chunks retrieved, how many were actually relevant. Low precision means you’re paying tokens for noise and giving the model room to get lost in the middle.
Faithfulness / groundedness: is every claim in the answer supported by the retrieved context. This is your hallucination meter, and the one to gate releases on hardest.
Answer relevance: does the answer address the question, completely. High groundedness with low relevance is the signature of a correct-but-evasive answer.

Building the golden set is the unglamorous work that decides everything. Mine real questions (from support tickets, search logs, and SME interviews) rather than invented ones, which cluster around what’s easy to imagine and miss how users actually phrase things. For each, record the correct answer and the chunk(s) that contain it. 150 examples that cover your real question distribution beat 1,000 synthetic ones. Then treat it as a living asset: every production failure that gets flagged becomes a new row, so the suite grows toward your actual failure modes instead of your imagined ones.

The closed evaluation loop: offline test set plus live traces drive measured, one-variable-at-a-time changes. Never ship a tuning change that hasn't cleared the gate.

Monitoring & observability

Offline evals tell you the system was good at release. Monitoring tells you it still is. Production RAG degrades quietly: the corpus drifts, users ask new kinds of questions, an upstream document format changes and parsing silently breaks. Instrument for it.

Trace every request end to end with a tool like Langfuse (langfuse.com): the query, the retrieved chunks and their scores, the final prompt, the answer, the citations, latency per stage, and token cost. When an answer is wrong, you need to see which stage failed, retrieval or generation, in one trace, not reconstruct it from logs.

Watch four families of signal:

Signal	What it catches	Example alert
Quality	groundedness/relevance drift, rising “I don’t know” rate	LLM-judge groundedness on sampled traffic drops below floor
Retrieval	corpus drift, parsing breakage	top-1 similarity score trending down week over week
Cost & latency	runaway tokens, slow reranking	p95 latency > 2s; cost-per-query above budget
User feedback	the ground truth automation misses	thumbs-down rate, “this is wrong” flags, escalations

Set the floors from your launch baseline, not from round numbers. Your golden-set groundedness at release is the line; alert when live-sampled groundedness drifts below it by more than a small margin. Do the same for retrieval: capture the top-1 similarity-score distribution at launch and alert when its median trends down week over week, which is the fingerprint of corpus drift or a parsing regression. For cost and latency, set a hard p95 budget (most assistant UIs need an answer starting in under two seconds) and a per-query cost ceiling, and alert before you breach them, not after the invoice.

Two operational habits separate teams that maintain quality from teams that file incidents. First, sample and grade live traffic continuously with the same LLM-judge from your offline harness; that’s your early-warning system. Second, feed thumbs-down traces back into the golden test set: every real failure becomes a permanent regression test, so the system can’t break the same way twice.

When a quality alert does fire, triage in pipeline order. Pull the offending traces and ask, in sequence: did parsing of the relevant document break, or did a source format change upstream? Is the right chunk in the index at all, or is this a freshness/indexing gap? Was it retrieved but ranked below the cutoff? Or retrieved and ignored? The trace (query, retrieved chunks with scores, prompt, answer) answers each question in seconds. Resist the reflex to reach for the prompt first; three of the four likely causes sit upstream of it, and prompt-tuning a parsing failure just buries it.

What to monitor continuously: drift

Point-in-time alerts catch a sudden break. The slower danger is drift, the gradual divergence between the system you launched and the one running today. Four kinds are worth a chart each.

Query drift. The distribution of what users ask moves as new topics appear, phrasing changes, and product launches shift the mix. Track the volume of low-retrieval-score queries and cluster the recent misses; a growing cluster of poorly-served questions is both your next golden-set additions and, often, a content gap to fill.

Corpus drift. Documents change, get added, or go stale. Watch the re-indexing rate and the age distribution of retrieved chunks. If answers increasingly cite old documents, freshness has broken somewhere upstream.

Retrieval-score drift. Track the distribution of top-1 similarity scores over time. A downward trend means queries are matching the corpus less well, the fingerprint of either query drift or a parsing or embedding regression. This is the single most useful drift signal, and it is nearly free to compute.

Quality drift. Run the offline LLM-judge over a sample of live traffic on a schedule, and chart groundedness and relevance week over week against the launch baseline. Offline evals prove the system was good at release; this is what proves it still is.

Set rolling baselines rather than fixed thresholds for these. “Groundedness fell four points below the trailing four-week mean” is a better alert than a hard floor, because it catches the gradual erosion that a static threshold sits above until it becomes a crisis. The weekly review from the team section is what ties it together: a human reads the drift charts and the worst traces and decides whether it’s noise or the start of a regression.

Rollout & the additional work

The model is maybe 30% of the work to ship. Here’s the other 70% that turns a working prototype into something you can put in front of users.

Phase the rollout. Never big-bang it.

Internal alpha: your team, against the real corpus, hammering it for two weeks. Goal: find the failure classes your eval set missed.
Shadow / human-in-the-loop: the system drafts, a human approves. For support, that means the agent sees the answer before the customer does. You get production-quality data with a safety net, and the approvals become training/eval data.
Limited GA: one team, one document set, real users, with an obvious feedback button and a fast path to turn it off.
Broad GA: expand corpus and audience once the metrics hold at each gate.

Each transition is a go/no-go gate, not a calendar date. Alpha → shadow when the team stops finding new failure classes. Shadow → limited GA when the human approves the draft unchanged most of the time. Limited GA → broad when the live metrics hold at golden-set levels for two stable weeks. If a gate doesn’t clear, you don’t expand; you fix and re-measure. Shipping on a date instead of a number is how a quietly-broken system reaches everyone at once.

The work teams forget to budget:

Data & access control. If the corpus has permissions, retrieval must respect them. A RAG system that surfaces a document the user shouldn’t see is a data breach with good UX. Filter by the user’s access tags at retrieval time; never rely on the model to keep a secret.
Security & the data perimeter. Decide what leaves your environment (embeddings, prompts, documents) and what doesn’t, and write it down. For the executive version of this conversation, see MCP and your data perimeter and the AI stack most enterprises should run.
Freshness. How does a changed document get re-indexed, and how fast? A stale index that confidently cites last quarter’s policy is a quiet liability.
The UI. Citations must be clickable and verifiable. “I don’t know” must be a first-class answer, not an error state. A feedback control must be one click.
Compliance & audit. In regulated settings you’ll need to show why the system answered as it did, which is another reason to trace and store retrieved context per request.

Team & skills required

You do not need a research team. You need a small group that can build a pipeline, measure it honestly, and operate it.

Role	What they own	Commitment
ML / AI engineer	The pipeline: parsing, chunking, retrieval, generation, evals	Core, full-time
Data engineer	Ingestion, the index as a versioned artifact, freshness jobs	Core through launch, part-time after
Backend engineer	API, access control, the app around the pipeline	Core
Domain expert / SME	The golden test set, judging “is this answer right”	Part-time, non-negotiable
Product / design	The answer UI, citations, feedback loop	Part-time
DevOps / platform	Deployment, monitoring, cost controls	Shared

The role people skip is the SME, and it’s the one that decides success. Engineers can build retrieval; only a domain expert can tell you whether the answer to “how do we handle a partial refund on an international order” is actually correct. Budget their time explicitly, a few hours a week building and curating the test set, or you’ll ship something that looks right to engineers and wrong to the people who know. For the broader org question, see the AI roles you actually need to hire.

How the team operates matters as much as who’s on it. Run a weekly eval review: the engineer brings the week’s metric deltas, the SME spot-checks a sample of answers the judge scored highly and a sample it scored poorly, and the group commits to the next single change to try. That ritual, small, regular, and evidence-led, is what keeps the system improving instead of drifting. The failure mode is the opposite: a launch, then silence, then a quarter later someone notices quality slipped and nobody can say when or why.

A 30/60/90-day delivery plan

A realistic path from zero to a defensible pilot.

Days 1–30, prove retrieval. Pick one document set and the ten questions that matter most. Stand up parsing (Finigami DocumentAI), structure-aware chunking, pgvector, and hybrid retrieval with a reranker. Build the first 100-question golden set with an SME. The goal by day 30 is a retrieval hit-rate you’d bet on, because the generation step is trivial once retrieval is good, and impossible when it isn’t.

Days 31–60, make it answer and measure it. Add the grounded prompt, the citations, and the abstention path. Wire up the eval harness and per-request tracing. Run the human-in-the-loop phase: the system drafts, your team approves and grades. The goal by day 60 is groundedness above your floor on the golden set, plus a growing backlog of real failures converted into new test rows.

Days 61–90, ship it small. Limited GA to one team, with the feedback button and a kill switch. Watch the live metrics, fold thumbs-down traces back into the set, and tune one variable at a time. The goal by day 90 is a pre-committed success metric, hit, with the evidence in hand to ask for the broad rollout.

The shape is deliberate: you don’t touch the generation step until retrieval is good, and you don’t widen the audience until the metrics hold. Each phase ends at a gate, and the gate is a number, not a vibe.

The business case

Most RAG projects that stall don’t stall on technology. They stall because no one wrote the one page that gets them funded. Here’s that page.

Frame it as deflected cost or unlocked revenue, not “we built AI.” The two framings that get approved:

Cost deflection: “Support agents spend an average of N minutes per ticket searching the knowledge base. A grounded assistant cuts that to under one. At X tickets/month that’s Y hours, ≈ $Z/year.” Make the number conservative and defensible.
Risk / revenue: “Slow or wrong answers cost us churn / SLA penalties / deals. Faster correct answers protect $X.”

Worked example. Support handles 40,000 tickets/month. Agents spend ~4 minutes per ticket searching the knowledge base; a grounded assistant cuts that to ~1. That’s 3 minutes × 40,000 = 2,000 agent-hours/month returned. At a $30/hour loaded cost, ≈ $60,000/month, roughly $720K/year of capacity. Now halve it for ramp and imperfect adoption: still ~$360K/year against a run cost in the low thousands. The payback is weeks. Bring that number, conservatively derived, not “AI will make support better.”

The per-query cost, modeled. Four components: embedding the query (a few hundred tokens, negligible), retrieval (database time, fractions of a cent), reranking (~50 candidates, small), and generation (the driver: input is your ~8 chunks plus the prompt, output is the answer). For an 8-chunk context that’s a few thousand input tokens and a few hundred output tokens: cents per query at frontier-model rates, less at Sonnet rates. The lever is context size: every chunk you fail to rerank away is tokens you pay for on every single request, forever. This is why the reranker is a cost control, not a luxury.

Put a cost ceiling and a quality floor on it. Decision-makers approve bounded bets, not open-ended ones. State the per-query cost target (you can model it: embedding + retrieval + reranker + generation tokens), the monthly ceiling, and the quality bar you’ll hold (e.g., ”≥ 90% groundedness on the test set, or we don’t ship”). The quality floor is what protects the brand; the cost ceiling is what protects the budget.

Pre-empt the three questions you’ll be asked:

Question	Your answer
”Will it make things up?”	Groundedness floor + citations + “I don’t know” as a real answer; we measure it continuously
”What does it cost to run?”	Per-query model + monthly ceiling; we alert before we breach it
”What happens when it’s wrong?”	Feedback loop, human-in-the-loop phase, one-click rollback; failures become regression tests

Tune the framing to who’s in the room. A CFO wants the deflected-cost number and the cost ceiling, conservatively derived. A CISO wants the data-perimeter decision and the access-control story; answer it before you’re asked. An engineering leader wants to know it won’t become an unmaintainable science project, which is exactly what the eval harness and the one-variable-at-a-time discipline are for. Same project, three different opening sentences.

Propose the smallest credible pilot. One team, one document set, an eight-week window, a single success metric agreed in advance. A scoped pilot that hits a pre-committed number is how you get the budget for the broad rollout. For the executive’s view of how to evaluate the return, see the AI ROI question and how to answer it.

Pitfalls & anti-patterns

The failures that recur, in rough order of how much damage they do:

Treating parsing as solved. The demo used clean text; production is scanned PDFs and multi-column layouts. This is the number-one silent killer. Budget for real document understanding.
Fixed-size chunking by default. Splitting on character counts shreds tables and sentences. Chunk on structure.
No eval set. “It feels better” is not a metric. If you skip one thing in this guide, don’t let it be the golden test set.
Pure vector search. It misses exact terms like names, IDs, and error codes. Go hybrid.
No reranker. The cheapest large quality gain in the pipeline, skipped because the demo “worked” without it.
Confident answers from empty context. A system that won’t say “I don’t know” will confidently cite its way into being wrong. Force abstention.
Citations as decoration. If the citation doesn’t point to the exact source a user can verify, it’s worse than nothing: it manufactures false trust.
Ignoring access control. Retrieval that doesn’t respect permissions is a breach waiting to be discovered.
Premature sophistication. Knowledge graphs and agentic retrieval before the simple pipeline is measured and tuned. Earn the complexity.
Evaluating only on easy or synthetic questions. A test set of questions you invented clusters around what’s obvious and hides the messy real ones. Mine the set from real usage, or it will tell you you’re winning while users churn.
Re-indexing in place. Mutating the index users are querying turns a routine upgrade into an outage. Build the new index alongside the old, validate it, then cut over.

FAQ

Is RAG dead now that context windows are huge? No. Long context complements RAG; it doesn’t replace it. Stuffing everything into the prompt is the most expensive, slowest, and least reliable way to answer a question that needed two paragraphs, and models still attend unevenly across very long contexts (Liu et al., 2023). Use long context to retrieve more generously, not to skip retrieval.

RAG or fine-tuning? Different tools. RAG injects knowledge that changes often and must be cited. Fine-tuning shapes behavior, format, and tone. If your problem is “the model doesn’t know our facts,” that’s RAG. If it’s “the model doesn’t answer in our style,” that’s fine-tuning. Many production systems use both.

How accurate can a RAG system get? Accuracy is bounded by retrieval. If the right chunk is never retrieved, the answer can’t be right. That’s why retrieval hit-rate is the first metric to fix, and why parsing and chunking, which set the ceiling, deserve most of your effort.

How much does it cost to run? Per query: embedding the query (cheap), retrieval (cheap), reranking (small), and generation (the main cost, driven by context tokens). A well-built system answers most questions for a fraction of a cent to a few cents. The lever is how much context you send, which is exactly why a tight reranker pays for itself.

How long to a production pilot? With a focused team and a clean-ish corpus, a scoped pilot is a matter of weeks, not quarters. The variable that moves the timeline most is document complexity: a corpus of scanned, multi-language forms takes far longer to parse well than clean HTML, which is the case for getting the parsing decision right early.

Do I need a vector database? Not a dedicated one, to start. pgvector inside the Postgres you already run is the right default until scale (roughly 10M+ vectors) or very high QPS forces a specialized store.

How big should chunks be? Start in the 500–1,200 token range, split on document structure, and let your eval move the number. Too small and a chunk can’t stand on its own; too large and you retrieve noise and pay for it. There’s no universal answer; it depends on your documents, which is exactly what the golden set is for.

Can I use an open-source model for generation? Yes, and many do for cost or data-residency reasons. The caveat: once retrieval is good, generation is forgiving, so the gap between a strong open model and a frontier one is smaller here than on open-ended tasks. Test citation faithfulness specifically. It’s the behavior that varies most between models and matters most for RAG.

How do I handle tables and images in documents? Tables: keep them as tables through parsing and chunk them with their headers, because a flattened table is a retrieval dead zone. Images and charts: extract captions and surrounding text, and for image-heavy corpora consider a multimodal model that can embed the image itself. The silent failure is dropping them at parse time and never knowing the answer lived in a figure.

How do I keep the index fresh? Index on document-change events rather than rebuilding nightly, store an indexed_at per chunk, and alert on staleness. For corpora that change constantly, incremental indexing isn’t optional. A confident citation of a superseded document is a real failure even when the text is perfectly well-formed.

How do I stop it from hallucinating? Three layers, in order of impact. Retrieve the right context (most “hallucinations” are actually retrieval failures, where the model was never given the answer). Force grounding and citations in the prompt, with abstention as a permitted output. Then measure groundedness continuously and gate releases on it. You won’t reach zero, but you can drive it low enough that the citations make every claim checkable, which is the honest goal, not a promise of perfection.

Reference implementation checklist

Ship what’s on this list and you’ll have a system you can defend to a skeptic:

If you’re missing the first eight, you have a demo. With all twelve, you have a system.

Executive briefings (for the people who approve this):

Sibling build guides:

Intelligent Document Processing in 2026: the parsing layer this guide depends on, in depth
AI Coding Agents in 2026
Agentic Workflow Automation in 2026