How to Measure RAG Answer Faithfulness
Check whether each material claim is supported by retrieved evidence without confusing relevance with grounding.
Short answer
Measure faithfulness by decomposing the answer into material claims and checking whether each claim is entailed by the context actually retrieved for that run. Score unsupported, contradicted, and unverifiable claims separately; a relevant answer can still be unfaithful.
Key takeaways
- Evaluate against retrieved context, not the whole knowledge base.
- Separate unsupported claims from incorrect retrieval.
- Keep citations attached to the exact passages used as evidence.
Signs this is the problem
Start by confirming the symptom before changing prompts, models, or infrastructure.
- Answers sound correct but include details absent from retrieved chunks.
- A single relevance score is used for retrieval and generation.
- Evaluators can see the answer but not the source context from that run.
Step-by-step approach
- 1
Freeze the evidence packet
Store the query, retrieved passages, document versions, answer, citations, and retrieval settings.
- 2
Extract material claims
Split facts, dates, quantities, entities, and consequential recommendations into checkable units.
- 3
Judge support
Label each claim supported, contradicted, or not established using only the retrieved evidence.
- 4
Aggregate carefully
Report claim-level failures and protected factual slices before any answer-level summary.
What to measure
| Metric | What it measures | How to use it |
|---|---|---|
| Supported-claim rate | Share of material claims directly supported by retrieved context. | Review unsupported high-impact claims even when the answer score passes. |
| Contradiction rate | Claims that conflict with the retrieved evidence. | Treat contradictions more severely than cautious omissions. |
| Evidence coverage | Important answer claims with an attributable supporting passage. | Use gaps to improve answer scope or retrieval. |
Common mistakes
- Checking against facts the model may know but did not retrieve.
- Rewarding a citation merely because one is present.
- Hiding one dangerous unsupported claim inside a long answer.
Practical checklist
- Freeze the evidence packet: Store the query, retrieved passages, document versions, answer, citations, and retrieval settings.
- Extract material claims: Split facts, dates, quantities, entities, and consequential recommendations into checkable units.
- Judge support: Label each claim supported, contradicted, or not established using only the retrieved evidence.
- Aggregate carefully: Report claim-level failures and protected factual slices before any answer-level summary.
Frequently asked questions
What should I do first?
Store the query, retrieved passages, document versions, answer, citations, and retrieval settings
How should the result be measured?
Share of material claims directly supported by retrieved context. Review unsupported high-impact claims even when the answer score passes.
When is the change ready to ship?
Ship only after the protected cases pass, the primary metric clears its agreed boundary, and the team reviews the remaining failure modes instead of relying on one aggregate score.