How to Test AI Citations and Source Attribution
Verify that cited sources exist, support the nearby claim, match the retrieved version, and remain accessible.
Short answer
Validate citations at four levels: the reference resolves, it identifies the source actually retrieved, the cited passage supports the associated claim, and the presentation does not overstate the evidence. A real link is not automatically a correct citation.
Key takeaways
- Separate link validity from claim support.
- Bind citations to source versions and passages captured during the run.
- Require abstention or caveats when evidence is incomplete.
Signs this is the problem
Start by confirming the symptom before changing prompts, models, or infrastructure.
- Citations resolve but lead to unrelated pages.
- Several claims point to one generic source without passage evidence.
- The answer cites a newer document than the one retrieval actually used.
Step-by-step approach
- 1
Parse claims and references
Associate each material claim with its local citation instead of scoring the bibliography as a whole.
- 2
Validate identity
Confirm the URL or document ID resolves to the captured source and version.
- 3
Check entailment
Verify the cited passage directly supports the claim's scope, entities, dates, and certainty.
- 4
Test failure behavior
Use missing, conflicting, stale, and partial sources to confirm the system qualifies or refuses the answer.
What to measure
| Metric | What it measures | How to use it |
|---|---|---|
| Citation validity | References that resolve to the intended source identity. | Treat redirects and mutable pages according to your evidence policy. |
| Citation correctness | Cited passages that support their associated claims. | Review high-impact unsupported claims individually. |
| Attribution coverage | Material claims with sufficient local evidence. | Do not reward citation volume without coverage. |
Common mistakes
- Counting links instead of checking support.
- Evaluating against a live page that changed after the run.
- Attaching one citation to a paragraph with several distinct claims.
Practical checklist
- Parse claims and references: Associate each material claim with its local citation instead of scoring the bibliography as a whole.
- Validate identity: Confirm the URL or document ID resolves to the captured source and version.
- Check entailment: Verify the cited passage directly supports the claim's scope, entities, dates, and certainty.
- Test failure behavior: Use missing, conflicting, stale, and partial sources to confirm the system qualifies or refuses the answer.
Frequently asked questions
What should I do first?
Associate each material claim with its local citation instead of scoring the bibliography as a whole
How should the result be measured?
References that resolve to the intended source identity. Treat redirects and mutable pages according to your evidence policy.
When is the change ready to ship?
Ship only after the protected cases pass, the primary metric clears its agreed boundary, and the team reviews the remaining failure modes instead of relying on one aggregate score.