How to Tell Whether a RAG Failure Came from Retrieval or Generation
Use staged checks to find whether the right evidence was missing, buried, ignored, or misrepresented.
Short answer
Check the pipeline in order. If the required evidence was not retrieved, diagnose indexing, query transformation, filters, or ranking. If adequate evidence was present but the answer ignored, contradicted, or overstated it, diagnose generation and prompting.
Key takeaways
- Never debug the final answer without the retrieved passages.
- Test retrieval sufficiency before grading answer faithfulness.
- Preserve hybrid failures where weak evidence and poor generation interact.
Signs this is the problem
Start by confirming the symptom before changing prompts, models, or infrastructure.
- Teams rewrite the prompt when the needed document never appeared.
- Top-k contains the answer but ranking buries it below distracting context.
- The generator invents specifics despite sufficient authoritative evidence.
Step-by-step approach
- 1
Define required evidence
Mark the document or passage needed to answer each test question correctly.
- 2
Evaluate retrieval
Measure whether required evidence appears, where it ranks, and whether filters or freshness removed it.
- 3
Evaluate generation
Given the captured context, check answer correctness, faithfulness, completeness, and appropriate abstention.
- 4
Fix the first broken stage
Change indexing, ranking, context assembly, or generation independently and rerun both layers.
What to measure
| Metric | What it measures | How to use it |
|---|---|---|
| Evidence recall | Cases where required evidence appears within the usable context window. | Fix retrieval when this misses regardless of answer quality. |
| Context precision | Retrieved content that is relevant enough to support the answer. | Reduce distractors when the evidence is present but diluted. |
| Conditional faithfulness | Answer grounding among cases with sufficient retrieved evidence. | Use this to isolate generation behavior. |
Common mistakes
- Using answer correctness as a proxy for retrieval quality.
- Changing chunking and prompts in the same experiment.
- Forcing an answer when the correct result is insufficient evidence.
Practical checklist
- Define required evidence: Mark the document or passage needed to answer each test question correctly.
- Evaluate retrieval: Measure whether required evidence appears, where it ranks, and whether filters or freshness removed it.
- Evaluate generation: Given the captured context, check answer correctness, faithfulness, completeness, and appropriate abstention.
- Fix the first broken stage: Change indexing, ranking, context assembly, or generation independently and rerun both layers.
Frequently asked questions
What should I do first?
Mark the document or passage needed to answer each test question correctly
How should the result be measured?
Cases where required evidence appears within the usable context window. Fix retrieval when this misses regardless of answer quality.
When is the change ready to ship?
Ship only after the protected cases pass, the primary metric clears its agreed boundary, and the team reviews the remaining failure modes instead of relying on one aggregate score.