Blog · July 28, 2026 · 8 min read
RAG That Actually Works: A Field Checklist
Most RAG failures are retrieval failures. Chunking, hybrid search, reranking, metadata filters, and evaluation — a practical checklist from building production research pipelines.
Retrieval-augmented generation is the most requested AI feature and the most commonly botched. The failure is almost never the generation step — it's retrieval quietly returning the wrong context, and the model confidently summarizing garbage. This checklist comes from building research and analysis pipelines like Truth Engine, where wrong retrieval means wrong business decisions.
Chunking: respect the document's own structure
Fixed 512-token windows sliced mid-sentence are the default and the mistake. Chunk on semantic boundaries — headings, paragraphs, table rows — and prepend each chunk with its breadcrumb ("Annual Report 2025 > Risk Factors > Currency"). A chunk that can't be understood alone can't be retrieved reliably.
Hybrid search beats embeddings alone
Dense vectors miss exact identifiers — SKUs, names, error codes — that keyword search catches trivially. Run BM25 and vector search in parallel and fuse results (reciprocal rank fusion is fine). This one change fixes a huge share of "the answer was in the corpus but retrieval missed it" tickets.
Rerank before you generate
First-stage retrieval optimizes recall; a cross-encoder reranker optimizes precision on the top 30-50 candidates. Feeding the model 6 highly relevant chunks beats feeding it 20 mediocre ones — better answers *and* lower token spend.
Metadata filters are half the battle
- Store structured fields (date, source, doc type, customer) alongside every chunk.
- Resolve filters *before* semantic search — "latest pricing" should hard-filter on recency, not hope embeddings notice.
- Let the LLM extract the filter from the question with a typed schema, then apply it in the database.
Answer honesty: refuse to guess
The system prompt must make abstention a first-class outcome: if the retrieved context doesn't contain the answer, say so and show what *was* found. Require citations to chunk IDs and render them in the UI. Users forgive "not found"; they don't forgive confident fabrication.
Evaluate retrieval separately from generation
- 01Build a golden set of ~50 real questions with known source passages.
- 02Measure retrieval hit-rate (is the right chunk in the top-k?) — this isolates most failures.
- 03Then grade end-to-end answers for faithfulness and completeness.
- 04Re-run on every change to chunking, embeddings, or prompts.
RAG is a search-quality problem wearing an AI costume. Treat it with the same rigor as the rest of your production agent stack, and it stops being the feature users quietly distrust.
Haider Farooq is an AI engineer and data scientist based in Lahore, Pakistan — core engineer on TryCook.ai, developer at Aligno, and creator of MarkSafe.net. He builds agentic AI systems, RAG pipelines, and automation for teams worldwide. Work with him.
RELATED WORK