RAG is not magic, it's plumbing
Everyone talks about Retrieval-Augmented Generation as if it's a silver bullet. Here's what I learned building one in production.
What RAG promises
Retrieval-Augmented Generation sounds magical on paper: you connect an LLM to a knowledge base, and it answers precise questions while citing its sources. Hallucination problem solved. Always up-to-date documentation. Zero friction.
In practice, it's plumbing. And like all plumbing, problems are invisible until something leaks.
The real pipeline
A production RAG system looks like this:
Ingestion : extract, chunk, clean source content
Indexing : encode chunks as vector embeddings, store them
Retrieval : for each question, encode the query, find similar chunks
Generation : build a prompt with retrieved chunks, call the LLM
Post-processing : filter, format, cite sources
Each step can silently degrade quality without throwing an error.
The real problems
Chunking is underestimated. If you cut a document at the wrong place, the retrieved chunk lacks context. A 5-step procedure split in the middle of step 3 doesn't answer any question correctly. I spent more time on chunking strategy than on LLM integration.
Retrieval doesn't guarantee relevance. Vector similarity measures semantic proximity, not usefulness for the question asked. A very similar chunk on the surface can be completely off-topic in depth. Score thresholds are empirical and brittle.
Source quality determines everything. Poorly structured, unmaintained, or contradictory documentation -> confusing answers. The LLM doesn't fix knowledge base defects, it amplifies them.
Evaluation is the real unsolved problem. How do you know your system is answering well ? You need a reference question/answer dataset, automatic evaluation logic, and time-based tracking. Without it, you're flying blind.
What I did in practice
On the network watch project at Orange Business Services:
Semantic chunking by document section (not fixed size)
Enriched metadata on each chunk (source, date, document type)
Hybrid filtering: vector similarity + structured filters (document type, relevant equipment)
Manual evaluation on a set of real cases before production deployment
The result : Analysis time has been reduced from several weeks to just a few hours. It's not a matter of a smart LLM, but of plumbing that works.
Conclusion RAG works.
But the magic is in the operational details, not in the API call. If you're planning to build a RAG system, invest 80% of your time on data quality and evaluation. The rest comes after.