How to Compare Two AI Agent Trajectories
Find the first meaningful divergence in plans, tools, evidence, retries, costs, and outcomes.
Short answer
Normalize both trajectories into ordered events, align them by task milestones rather than raw token position, and find the first meaningful divergence. Compare tool choices, arguments, evidence, errors, recovery, cost, and final state; allow multiple valid paths when the contract permits them.
Key takeaways
- Align on semantic milestones, not identical hidden reasoning.
- The first divergence is more actionable than the final difference.
- Compare only runs with compatible task and environment evidence.
Signs this is the problem
Start by confirming the symptom before changing prompts, models, or infrastructure.
- Reviewers place two raw transcripts side by side manually.
- Different but valid tool order is marked as failure.
- A candidate looks cheaper because it stopped before completing the task.
Step-by-step approach
- 1
Check comparability
Confirm task, inputs, environment, permissions, source versions, and outcome contracts match.
- 2
Normalize events
Represent models, tools, handoffs, approvals, state changes, errors, and evidence with stable types and IDs.
- 3
Align milestones
Match equivalent goals and state transitions while preserving extra, missing, or reordered steps.
- 4
Explain the decision
Connect divergences to quality, safety, latency, cost, and verified final outcome.
What to measure
| Metric | What it measures | How to use it |
|---|---|---|
| Milestone match | Required states reached by both runs, including order constraints where meaningful. | Allow flexible paths when the product contract allows them. |
| First-divergence location | Earliest event where the candidate departs from expected evidence or action. | Use it to select the owning layer for a fix. |
| Outcome-adjusted efficiency | Resource use among trajectories that satisfy the same outcome. | Do not reward incomplete candidates. |
Common mistakes
- Comparing private chain-of-thought text.
- Assuming identical event counts mean equivalent behavior.
- Diffing runs from different data or permission states.
Practical checklist
- Check comparability: Confirm task, inputs, environment, permissions, source versions, and outcome contracts match.
- Normalize events: Represent models, tools, handoffs, approvals, state changes, errors, and evidence with stable types and IDs.
- Align milestones: Match equivalent goals and state transitions while preserving extra, missing, or reordered steps.
- Explain the decision: Connect divergences to quality, safety, latency, cost, and verified final outcome.
Frequently asked questions
What should I do first?
Confirm task, inputs, environment, permissions, source versions, and outcome contracts match
How should the result be measured?
Required states reached by both runs, including order constraints where meaningful. Allow flexible paths when the product contract allows them.
When is the change ready to ship?
Ship only after the protected cases pass, the primary metric clears its agreed boundary, and the team reviews the remaining failure modes instead of relying on one aggregate score.