RAG · PRODUCTION · TRUST
The demo answers.
The system explains why.
After working on document AI products and building one in public, I stopped thinking about RAG as “retrieve chunks and call a model.” In production, the difficult work is preserving a chain of evidence while documents, permissions, models, and failure conditions keep changing.
01 / THE PRODUCTION GAP
The happy path fits
in one arrow.
Uploading a PDF, splitting it, creating embeddings, and getting an answer is an integration test. It does not prove the product can sustain trust.
The first version of almost every RAG diagram fits on one line: document → chunks → index → prompt → answer. The first version of almost every incident lives in what that diagram leaves out: a scanned PDF with no text, a retry that duplicates content, changed permissions, a new embedding version, a slow provider, or a fluent answer built on mediocre evidence.
In my experience, the model is rarely the part that needs the most design. The product becomes serious when every stage has a contract, a visible state, and a way to fail without lying. That moves the conversation from “which vector database?” to “which claim can we defend?”
DATA PLANE · HOW CONTENT BECOMES SEARCHABLE
- 01SOURCE
- 02VALIDATE
- 03EXTRACT
- 04CHUNK
- 05INDEX
QUERY PLANE · HOW EVIDENCE BECOMES AN ANSWER
- 06IDENTITY
- 07RETRIEVE
- 08RANK
- 09GENERATE
- 10VERIFY
The UI should not hide this machine. It should translate it: processing, ready, insufficient evidence, access denied, provider unavailable, and safe to retry are product states, not implementation details.
02 / INGESTION
Answer quality starts
before the index.
Retrieval cannot recover structure that ingestion destroyed. The document pipeline is part of the quality model.
Stable identity
File checksum, source version, and pipeline version. A retry should deliberately resume or replace work, never create a second truth by accident.
Verifiable extraction
Text, page, title, section, tables, and offsets should preserve enough provenance to return to the original. OCR and parsing fail differently; “completed” does not mean “extracted well.”
Explicit states
stored, extracting, extracted, indexing, indexed, and failed say what happened and what can be retried. A ready boolean deletes information exactly when you need it.
Reproducible reindexing
The index is a rebuildable projection. Record which parser, chunking rules, and embedding produced every entry, and make rebuilds possible without losing the source.
WHAT CHANGED MY OWN DESIGNI used to think of the chunk as the primary unit. Now I start with the evidence a person needs to open: document, version, page, section, and passage. A chunk is a retrieval optimization; it should not become the identity of the knowledge.
03 / RETRIEVAL
Measure the search engine
before blaming the model.
A bad final answer combines, at minimum, retrieval errors and synthesis errors. If you only score the prose, you do not know which system to fix.
Did the right evidence appear?
This is the first gate. If the useful passage never reaches context, no prompt can recover it.
Did it appear high enough?
Order matters when the context budget is small and the model uses the supplied evidence unevenly.
How much noise did we send?
More chunks can raise recall while diluting evidence, increasing tokens, and making the answer worse.
Do we detect when evidence is weak?
The suite needs unanswerable cases too. Retrieving something is not the same as retrieving enough.
My implementation order
- 1Versioned question set
Real questions, expected document, relevant passage, and cases that should abstain.
- 2Lexical baseline
BM25 or full-text search: cheap, explainable, and strong for names, codes, dates, and exact terms.
- 3Error analysis
Separate synonyms, bad extraction, broken chunks, incorrect filters, and missing content.
- 4Semantic or hybrid
Add vectors, query expansion, or reranking only where the set demonstrates a gap.
In my public project I started with FTS5/BM25 and Recall@3 and MRR@3 thresholds in CI. Not because lexical search always wins, but because a simple measured baseline turns “we need embeddings” into a testable hypothesis.
04 / GENERATION & CITATIONS
The model writes.
The server answers.
Grounding is not attaching sources below some prose. It is constraining what the system can claim and making every reference resolve to evidence that was present in that request.
- 01
REQUEST-LOCAL EVIDENCE
Give retrieved passages opaque IDs. The model may cite those IDs; it may not invent filenames, page numbers, or URLs.
- 02
STRUCTURED OUTPUT
Validate answer, citations, and abstention state against a schema. A valid parse does not prove truth, but it removes ambiguous states.
- 03
SERVER-RESOLVED CITATIONS
Turn every valid ID into document, version, page, offsets, and link. Reject unknown IDs instead of rendering them.
- 04
ABSTENTION AS SUCCESS
If evidence does not cover the question, “I could not find sufficient support” is correct. Forcing prose is a product failure.
The distinction I find most useful is simple: the model proposes a composition; the application decides which evidence exists, which user may see it, and which structure is accepted.
05 / SECURITY
The index is also
an authorization boundary.
RAG can leak information to people who could never open the source. A prompt is not an access control.
Authorize before ranking
Apply tenant, ACL, classification, and validity filters inside the query. Retrieving everything and filtering later already exposed data to the pipeline and distorted ranking.
Synchronize permissions
Index authorization is a replica of the source. Measure its lag, process revocations, and define behavior while an ACL is stale.
Treat content as untrusted
A document can contain instructions for the model. Delimit data from instructions, reduce capabilities, and never let retrieved context grant permissions or enable tools.
Minimize sensitive telemetry
Queries, chunks, prompts, and responses may hold PII or secrets. Useful logs need not store full content; prefer IDs, hashes, counts, and controlled sampling.
Prompt injection is not solved by another sentence in the system prompt. It is contained by architecture: least privilege, separation of instructions and data, output validation, and no sensitive action without deterministic authorization outside the model.
06 / OPERATIONS
Design for the day
one stage fails.
A RAG request crosses storage, search, and a probabilistic provider. Latency, cost, and availability compose; blind retries do too.
A partial file never appears as a valid source.
The file remains; the whole flow does not restart.
The answer can declare its knowledge boundary.
The model is not called to manufacture confidence.
The user gets an honest state and no work is orphaned.
An unexpected shape never reaches the client.
Ingestion, OCR, bulk embeddings, and reindexing belong in durable jobs once they exceed an HTTP budget. That introduces idempotency, backpressure, dead-letter handling, and persisted progress. Making work asynchronous does not remove state; it makes state unavoidable.
07 / OBSERVABILITY
One total latency
explains nothing.
Instrument the chain as connected stages and separate operational health from quality. A system can be fast and wrong.
documents by state · bytes · pages · extraction time · failures by parser
latency · top-k · filters · scores · zero results · index version
model snapshot · tokens · TTFT · latency · finish reason · estimated cost
citations opened · abstentions · feedback · reformulations · task completed
request_idcorpus_versionretriever_versionprompt_versionmodel_snapshotevidence_idsoutcomeYou do not need to log content to reconstruct a decision. You need correlation and versioning. Sensitive content is captured only under an explicit retention and access policy.
08 / RELEASE GATE
What I review
before calling it production.
A checklist cannot guarantee quality. It can stop an integration that works from being mistaken for a system that operates.
- 01
Does every document have identity, version, checksum, and processing state?
- 02
Can I rebuild the index from source without losing provenance?
- 03
Is there an evaluation set with positive, negative, and unanswerable questions?
- 04
Do retrieval and generation have separate metrics and gates?
- 05
Are citations resolved from server-side evidence in the same request?
- 06
Is authorization enforced during retrieval and are revocations reflected?
- 07
Does each stage have a timeout, bounded retry, idempotency, and visible failure state?
- 08
Can I attribute latency, tokens, and cost per request and tenant?
- 09
Are model, prompt, parser, chunking, embeddings, and index versioned?
- 10
Can the product abstain without treating it as a technical error?
THE IDEA IN ONE LINE
Do not build a chatbot
over documents.
Build a chain of evidence that can ingest, authorize, retrieve, measure, and explain documents. Then let a model write over that chain. That order produces less magic in the demo and much more trust when the system matters.
09 / GO DEEPER
Sources and
verifiable work.
- 01Retrieval-Augmented Generation for Knowledge-Intensive NLP TasksThe original RAG paper · Lewis et al. ↗
- 02RAG in Azure AI SearchPipeline, chunking, hybrid retrieval, and citations · Microsoft ↗
- 03Evaluation best practicesStructured tests for variable systems · OpenAI ↗
- 04Document-level access controlDocument-level authorization during retrieval · Microsoft ↗
- 05LLM01: Prompt InjectionPrompt injection risks · OWASP ↗
- 06AI Knowledge PlatformSource, ADRs, evals, and the decisions discussed here →