A retrieval-augmented generation (RAG) prototype takes an afternoon. Point an embedding model at a folder of PDFs, store the vectors, send the top five chunks to an LLM, and you have an assistant that answers questions about your documents.
Then real users arrive. They ask questions your documents only half answer, they expect citations, and the legal team wants to know who can see what. Here are seven lessons we apply to every RAG system we take to production.
1. Start with an evaluation set, not a model
Before choosing a model or a vector database, sit down with the people who will use the assistant and write 50 to 100 real questions with the answers they expect. Include the awkward ones: questions with no answer in the corpus, questions that need two documents, and questions phrased the way a tired user would phrase them.
This set becomes your benchmark. Every change to chunking, prompts or models gets measured against it. Without it, you are tuning by vibes.
2. Chunking decides most of your quality
Most "the AI is wrong" complaints are really retrieval failures: the right passage never reached the model. Fixed-size chunks of 500 tokens split tables in half and separate headings from the paragraphs they describe.
What works better:
- Split on document structure (headings, sections, list items) first, size second.
- Keep the heading path with every chunk, for example
Policy > Claims > Exclusions. - Store tables as whole units with a short generated summary.
- Overlap chunks slightly so a sentence never loses its context.
3. Hybrid search beats pure vectors
Embeddings are good at meaning and bad at exact terms. A user searching for policy number HX-2291 or a product code needs keyword matching. We combine vector search with BM25 keyword search (in Postgres with pgvector, or Elasticsearch) and re-rank the merged results. The improvement on real queries is usually large and cheap to get.
4. Enforce access control at retrieval time
If a user cannot open a document in your intranet, the assistant must not quote it. Filter by permissions before retrieval, using the same groups and roles as the source system. Filtering the model's answer afterwards is too late, because the content has already been sent to the model.
5. Make the model show its sources
Ask the model to cite the chunk IDs it used, and render those as links to the source document and section. Citations build trust, make errors easy to spot, and give you a feedback loop: when users click a citation and the answer isn't there, you've found a retrieval or prompt bug.
Also give the model explicit permission to say "I couldn't find this in the documents." An honest non-answer is far better than a confident wrong one.
6. Watch cost and latency from day one
Each query can involve an embedding call, a search, a re-ranker and a large prompt. At scale this adds up. Practical controls:
- Cache embeddings and frequent answers.
- Use a smaller, faster model for query rewriting and classification.
- Cap context size and measure whether more chunks actually improve answers.
- Track cost per conversation on a dashboard next to quality metrics.
7. Treat it as a product, not a project
Documents change, models change and user questions drift. Schedule re-indexing, re-run your evaluation set on every release, and review a sample of real conversations each week. The teams that get lasting value from RAG are the ones that keep improving it after launch.
Where to start
Pick one workflow with a clear owner and a measurable outcome, such as support agents answering policy questions or engineers searching runbooks. Build the evaluation set with that team, ship a narrow assistant in a few weeks, and expand from there.
If you'd like a second opinion on a RAG system you're planning or already running, talk to our AI team.