The Demo Works. Then the Knowledge Base Moves.
Retrieval-augmented generation solved a real problem: it let language models answer from your data instead of their training memory. But every team that ships RAG discovers the same second act. The demo on a static corpus is impressive; three months later, the system is confidently citing a pricing page that was rewritten last quarter, missing the policy that replaced it, and answering questions about documents nobody has opened since launch. The model did not get dumber — the knowledge system rotted.
The core misconception is treating RAG as a one-time integration: chunk the docs, embed them, index them, ship. In reality, a retrieval system is a data pipeline with a live corpus, and data pipelines have freshness problems, quality problems and evaluation problems. The teams that get this right stop thinking of RAG as an AI feature and start operating it as a knowledge infrastructure project — with owners, SLAs and monitoring.
Here is what actually keeps retrieval systems accurate over time, based on the systems we have built and operated for clients in legal, healthcare, finance and enterprise support.
Chunking Is a Retrieval Decision, Not a Text Processing Step
Most RAG implementations inherit their chunking strategy from a tutorial: split by tokens every 512, overlap 50, move on. That default optimizes for embedding convenience, not answer quality. The right chunk boundary is defined by the question you expect to be asked — a policy document should chunk by clause or section so a retrieved chunk contains a complete rule, not half of one; an API reference should chunk by endpoint; a support corpus by issue-and-resolution pair. A chunk that splits a statement from its conditions produces retrievals that look relevant and answer wrong.
Two refinements matter more than any chunk-size tuning. First, metadata-rich chunks: tag every chunk with its source document, section, effective date and owner, then use metadata filters in retrieval so a question about current policy never retrieves a superseded version. Second, hierarchical retrieval: keep both fine-grained chunks for precise answers and coarse summaries for orientation, and let the pipeline decide which level to pull from. Flat single-level chunking forces every question through the same granularity, which fits none of them well.
Measure chunking quality directly. Build a set of 50-100 real questions with known source locations, retrieve, and check whether the correct chunk ranks in the top results. If your golden questions fail retrieval, no amount of prompt engineering downstream will save the answers — the failure is upstream in how the knowledge was cut up.
Freshness Pipelines: The Part Everyone Forgets to Build
Knowledge rots at three speeds. Fast rot: prices, inventory, status pages — hours to days. Medium rot: policies, procedures, product documentation — weeks. Slow rot: reference material, historical records — months. A system that re-indexes everything nightly wastes compute on slow-rot content while fast-rot content sits stale for 23 hours. The fix is tiered ingestion: subscribe to change events where sources provide them (CMS webhooks, API change feeds), poll fast-rot sources frequently, and full-scan slow-rot sources on a long cycle.
Supersession handling is where naive pipelines fail hardest. When a document is updated, the old chunks must be invalidated, not merely joined by new ones — otherwise retrieval returns both versions and the model blends them into a confident contradiction. Every chunk needs a lifecycle: active, superseded, archived. Retrieval queries active chunks only; audit queries can see the rest. This sounds obvious and is implemented correctly in a minority of production systems we have audited.
Add provenance to every answer. The user (and your evaluators) should see which document, which section, and which version produced each claim. Provenance turns 'the AI said so' into a checkable citation, and it is the single biggest trust multiplier in enterprise deployments. It also makes rot visible: the day a user clicks a citation and finds the source changed, you have caught staleness before it became an incident.
Evaluation Loops: How You Know Accuracy Is Holding
You cannot manage accuracy you do not measure, and RAG accuracy is a composite that degrades in different places: retrieval can fail (wrong chunks surfaced), generation can fail (right chunks, wrong synthesis), or grounding can fail (model ignores the context entirely). A single end-to-end score hides which layer broke. Evaluate each layer separately — retrieval hit-rate against your golden question set, faithfulness of answers against retrieved context, and answer relevance against the question — and track all three over time.
Run the suite continuously, not just at launch. Schedule weekly automated evaluation against the golden set, sample real production conversations daily for human review, and feed corrected examples back into the test set. The golden set must grow with the business: every production miss becomes a regression test, so the same failure mode cannot silently return. Teams that treat the eval set as static are measuring a museum, not a system.
Set thresholds that trigger action, not just observation. Retrieval hit-rate below target triggers a chunking or embedding review; faithfulness drops trigger a prompt or model review; a spike in 'I don't know' responses might mean the corpus lost coverage. When metrics have owners and thresholds, accuracy becomes an operational discipline instead of a hope — which is exactly what a knowledge system deserves when real decisions depend on its answers.