How to Evaluate a Multi-Step AI Agent
Test planning, tools, recovery, state, and completion across the full task instead of grading only the final answer.
Short answer
Evaluate the whole trajectory and the final state. Test whether the agent understood the goal, chose permitted tools, used correct arguments, incorporated results, recovered safely, stayed within budgets, and satisfied independently verified postconditions.
Key takeaways
- A good final answer can hide a wasteful or unsafe path.
- Score milestone and final-state requirements separately.
- Repeat long tasks to expose compounding nondeterminism.
Signs this is the problem
Start by confirming the symptom before changing prompts, models, or infrastructure.
- Evaluation reads only the final response.
- Tool errors and retries are absent from the score.
- The agent completes easy paths but degrades with task length.
Step-by-step approach
- 1
Define task contracts
Specify goals, milestones, allowed actions, budgets, approvals, and observable final state.
- 2
Capture the trajectory
Record ordered model, tool, handoff, state, error, retry, and approval events with parent links.
- 3
Score layers separately
Evaluate plan relevance, tool behavior, recovery, policy, response, and verified outcome without one opaque grade.
- 4
Test variation
Repeat representative tasks and inject partial failures to measure stable success rather than one lucky path.
What to measure
| Metric | What it measures | How to use it |
|---|---|---|
| Verified task success | Runs that satisfy all required postconditions. | Use as the primary outcome while preserving partial results. |
| Milestone completion | Required intermediate states reached in the correct order. | Use gaps to locate the first divergence. |
| Trajectory efficiency | Steps, retries, latency, and cost among successful runs. | Optimize only after correctness and safety qualify. |
Common mistakes
- Rewarding fluent explanations of work that was not completed.
- Requiring one exact path when several safe paths are valid.
- Ignoring failures that recover only by duplicating side effects.
Practical checklist
- Define task contracts: Specify goals, milestones, allowed actions, budgets, approvals, and observable final state.
- Capture the trajectory: Record ordered model, tool, handoff, state, error, retry, and approval events with parent links.
- Score layers separately: Evaluate plan relevance, tool behavior, recovery, policy, response, and verified outcome without one opaque grade.
- Test variation: Repeat representative tasks and inject partial failures to measure stable success rather than one lucky path.
Frequently asked questions
What should I do first?
Specify goals, milestones, allowed actions, budgets, approvals, and observable final state
How should the result be measured?
Runs that satisfy all required postconditions. Use as the primary outcome while preserving partial results.
When is the change ready to ship?
Ship only after the protected cases pass, the primary metric clears its agreed boundary, and the team reviews the remaining failure modes instead of relying on one aggregate score.