Testing RAG: Retrieval vs Generation
RAG feeds the model documents, then asks it to answer from them. It fails in two completely different places — and calling both "the AI is wrong" is the mistake that gets bugs mis-assigned for weeks.
Two stages, two bugs
- Retrieval fetched the wrong documents — the right answer was never in front of the model. Fixing the prompt does nothing; the search is broken.
- Generation got the right documents and still answered wrong — ignored them, mixed them up, or made something up. The search is fine; the model isn't grounded.
Same visible symptom — a wrong answer — opposite fixes. Every RAG test must say which stage failed, or the bug goes to the wrong team.
Score them separately
- Retrieval: did the fetched chunks contain the answer? (Did the known-correct document make the top-k?)
- Faithfulness: is every claim in the answer supported by the retrieved chunks — nothing added? This is the anti-hallucination check.
- Answer relevance: does it actually address the question, not just quote context at the user?
▸ The two-question test
chunks = retrieve(question)
answer = generate(question, chunks)
# stage 1 — retrieval
assert known_answer_doc in chunks # was the answer even available?
# stage 2 — generation, given those chunks
assert judge(answer, chunks,
"every claim is supported by the chunks, nothing invented")Grounded in published RAG evaluation practice (retrieval and faithfulness metrics)