AI & Data8 min read

Building production RAG systems: architecture, evaluation and guardrails

Retrieval-augmented generation is easy to demo and hard to run well. This guide covers ingestion, retrieval, evaluation and guardrails for RAG systems that hold up in production.

By INFIDEC AI Engineering Team

Cover: production RAG systems

Why RAG prototypes disappoint in production

Retrieval-augmented generation answers questions by retrieving relevant passages from your own content and passing them to a language model as context. A prototype built over a weekend with a vector database and a few documents often looks impressive. Then real users arrive with ambiguous questions, the corpus grows to tens of thousands of documents with conflicting versions, and answers become confidently wrong.

In our experience, most production problems are retrieval problems rather than model problems. If the right passage is not in the context window, no model will reliably produce the right answer. That is why a production RAG system needs as much care in its data pipeline and evaluation as in its prompt.

Ingestion: the part that decides quality

Ingestion turns messy source content into clean, well-described chunks. Treat it as a data pipeline with tests, not a one-off script.

  • Parse documents with structure intact. Headings, tables and lists carry meaning that plain text extraction often destroys.
  • Chunk along natural boundaries such as sections and paragraphs, with a size tuned by evaluation rather than guesswork. Small overlaps can help when ideas span boundaries.
  • Attach metadata to every chunk: source, title, section path, date, version, product and access permissions.
  • Deduplicate and handle versions, so retrieval does not return an outdated policy alongside the current one.
  • Make ingestion incremental and idempotent, so updates and deletions in the source are reflected quickly.

Retrieval: hybrid search, filters and reranking

Dense vector search is good at matching meaning but can miss exact terms such as product codes, error messages and names. Keyword search with BM25 is the opposite. Hybrid retrieval runs both and merges the results, for example with reciprocal rank fusion, and is a sensible default for most enterprise content.

Apply metadata filters before or during retrieval so results are limited to the right product, region or document type, and enforce access control at this layer. A user should never receive an answer built from a document they are not allowed to open. After retrieving a broader candidate set, a cross-encoder reranker can reorder the top results by relevance, which usually improves precision at modest latency cost.

Choose the embedding model with the same evidence-based approach as everything else. Public leaderboards are a starting point, but your own retrieval test set is what matters, and switching models later means re-embedding the whole corpus, so record which model and version produced each index.

For conversational interfaces, rewrite follow-up questions into standalone queries before retrieval, since a question such as "what about the enterprise plan?" is meaningless without the previous turn.

Evaluation you can trust

Without evaluation, every change to chunking, embeddings or prompts is a guess. Build a test set early: a few hundred realistic questions, ideally drawn from real user queries, with the expected source passages and reference answers reviewed by domain experts.

  • Measure retrieval separately with metrics such as recall at k and mean reciprocal rank: did the right passages come back, and how high?
  • Measure generation for faithfulness to the retrieved context, answer relevance and correctness against the reference.
  • LLM-as-judge scoring scales well, but calibrate it against human ratings on a sample and re-check whenever you change the judge prompt or model.
  • Include questions that should be refused or answered with "I don't know" because the content does not cover them.
  • Run the suite in CI so regressions block a release, just like failing unit tests.

Guardrails for safety and trust

Guardrails keep the system within its intended scope and make failures visible rather than silent.

  • Require citations to retrieved sources, and show them to users so answers can be verified.
  • When retrieval returns nothing relevant, say so instead of letting the model answer from general knowledge.
  • Treat retrieved content as untrusted input. Documents can contain prompt injection, so keep instructions separate from context and never let retrieved text trigger tool calls without validation.
  • Detect and redact personal or sensitive data where policy requires it, in both inputs and outputs.
  • Apply input and output moderation appropriate to your audience and domain.

Operating RAG in production

Log every request with the query, retrieved chunk IDs, prompt version, model version, latency, token usage and any user feedback. These traces are how you debug a bad answer and how you find gaps in the content. Version your index alongside your code so you can reproduce what the system knew on a given day.

Manage cost and latency with caching for repeated queries, smaller models for query rewriting and classification, and limits on how much context is sent per request. Finally, feed production failures back into the evaluation set. A RAG system improves steadily when every reported bad answer becomes a test case.

Strategy

How much does custom software development cost in 2026?

There is no single price for custom software, but there is a reliable way to estimate it. Here are the factors that drive cost, how engagement models differ, and how to reduce budget risk.

7 min read

Have a product to build or a system to fix?

Tell us where you are and what you need. A senior engineer replies within 24 hours with next steps, not a sales script.