RAG in Production: What Actually Breaks at Scale

Every RAG demo looks the same: a clean corpus, three test questions, and a confident answer with a citation. Production is nothing like that. The gap between a working prototype and a reliable product shows up the moment real users ask real questions against a messy, growing knowledge base.

Retrieval quality is where most systems quietly fail first. Naive chunking splits context mid-thought, embeddings drift as your corpus grows, and cosine similarity happily returns confident, plausible, wrong passages. We've had the best results treating retrieval as its own tunable pipeline — hybrid lexical plus vector search, re-ranking, and chunk boundaries that respect document structure rather than a fixed token count.

The second failure mode is silent regression. A prompt change, a model upgrade, or a new document type can quietly tank answer quality with no error, no exception, no red dashboard cell just users who stop trusting the tool. Production RAG needs a real evaluation harness: a growing set of labelled question–answer pairs, automated scoring on every deploy, and human review sampling on a schedule, not just at launch.

Cost and latency are the third axis, and they fight each other. Bigger context windows and re-ranking steps improve accuracy but add both. We typically start with the smallest retrieval pipeline that meets a quality bar, then add re-ranking, query rewriting or multi-hop retrieval only for the query patterns that actually need it — not the whole traffic.

None of this is exotic engineering. It's disciplined engineering: version your retrieval pipeline like code, evaluate every change, and treat 'it answered correctly in the demo' as the starting line, not the finish line.