Why your RAG demo works and your RAG product doesn't
Almost every RAG project I see starts with a demo that works. Someone points a retriever at a folder of clean PDFs, wires it to a good model, asks three questions, and gets three good answers. The room is convinced. Budget gets approved. Then six weeks later the same system is quietly wrong a third of the time on real user questions, and nobody can say why.
The gap between that demo and a product isn't a modeling problem. It's the set of things a demo never has to survive: a messy corpus, users who don't phrase questions the way you did, and cost and latency that only bite at scale. Here are the five that break most often, and how I scope around each before a line of production code gets written.
1. Retrieval quality collapses on a real corpus
A demo retrieves from a few dozen tidy documents. Production retrieves from tens of thousands of them — duplicates, near-duplicates, outdated versions, scanned tables, boilerplate headers, three departments that use different words for the same thing. Top-k similarity that looked precise on the demo set starts returning plausible-but-wrong chunks, and the model faithfully summarizes the wrong chunk.
The fix is rarely a fancier embedding model. It's chunking that respects document structure, metadata filtering, hybrid (keyword + vector) retrieval, and usually a reranker. But you don't know which of those you need until you measure retrieval in isolation — separate from the model — on your documents and your real questions.
2. There are no evals, so "it's worse now" is an argument, not a number
The demo was evaluated by vibes: someone read the answers and nodded. That doesn't scale, and it means every future change — a new model, a prompt tweak, a re-chunk — is a coin flip you can't score. When quality regresses, you find out from an angry user, not a dashboard.
Before I build, I want a labeled eval set of real questions with known-good answers, and separate metrics for retrieval and generation: retrieval recall/precision on one side, answer correctness and faithfulness on the other. This is unglamorous and it is the single highest-leverage thing you can do. It turns "it feels worse" into "retrieval recall dropped from 0.91 to 0.79 when we swapped the chunker," which is a fixable statement.
3. Latency and cost were free in the demo and aren't in production
One user asking one question hides everything. Add a reranker, a large context window, and a frontier API, then multiply by real traffic, and you get p95 latencies that feel broken and a monthly bill that makes someone ask whether this was worth it. I've seen teams overspend 40–60% on inference simply because nobody profiled where the tokens and the milliseconds actually went.
Scoping this up front means deciding what runs on a frontier API versus a cheaper or private model, where caching helps, and what the real cost-per-query and latency budget are — on your task, not a benchmark. Often a fine-tuned open model in your own cloud does the specific job at a fraction of the cost. Sometimes it doesn't, and you should know that before you commit, not after.
4. Guardrails and failure modes don't exist yet
Demo users ask cooperative questions. Real users ask about things not in the corpus, paste in confidential data, try to jailbreak it for fun, and — worst of all — ask perfectly reasonable questions the system answers confidently and wrongly. A demo has no answer for "what should happen when we don't know?" Production needs one.
That means an explicit "I don't have that" path, citations users can check, retrieval-confidence thresholds, and handling for out-of-scope and adversarial input. In regulated contexts — the fintech and compliance-heavy environments I work in most — this isn't polish, it's the difference between shippable and not.
5. Nobody owns it after launch, and it drifts
The corpus changes. Documents get added, policies get updated, the underlying model gets deprecated on ninety days' notice. A demo is frozen in time; a product decays. Without monitoring, the first sign of drift is a complaint.
So I scope ongoing eval monitoring, a re-indexing story, and a clear handoff from day one — because the whole point is that your team owns and runs this after I'm gone, not that you're locked into me to keep it alive.
The pattern
Notice that four of the five are decided before you build, not during. That's the actual lesson: the demo-to-product gap is mostly a scoping problem wearing an engineering costume. The teams that ship measure retrieval and generation separately, know their cost and latency budget, and design for the questions the demo never asked. The teams that stall skip straight to building because the demo already worked.
This is exactly what a short, fixed-scope diagnostic is for — mapping your corpus, defining the eval targets, and sizing cost and latency before anyone commits to a build. If that's where you are, the AI Readiness Sprint is the two-week version of everything above. Or if you'd rather just talk it through, book a call.