---
title: "How to Evaluate a Multi-Step AI Agent"
description: "Test planning, tools, recovery, state, and completion across the full task instead of grading only the final answer."
canonical: "https://www.evalgate.com/guides/evaluate-multi-step-ai-agents"
date-published: "2026-09-02"
last-updated: "2026-09-02"
category: "Tools and trajectories"
keywords: "evaluate multi-step AI agent, agent trajectory evaluation, AI agent testing"
---

# How to Evaluate a Multi-Step AI Agent

Test planning, tools, recovery, state, and completion across the full task instead of grading only the final answer.

**Question:** How do I evaluate a multi-step AI agent?

## Short answer

Evaluate the whole trajectory and the final state. Test whether the agent understood the goal, chose permitted tools, used correct arguments, incorporated results, recovered safely, stayed within budgets, and satisfied independently verified postconditions.

## Key takeaways

- A good final answer can hide a wasteful or unsafe path.
- Score milestone and final-state requirements separately.
- Repeat long tasks to expose compounding nondeterminism.

## Signs this is the problem

- Evaluation reads only the final response.
- Tool errors and retries are absent from the score.
- The agent completes easy paths but degrades with task length.

## Step-by-step approach

### 1. Define task contracts

Specify goals, milestones, allowed actions, budgets, approvals, and observable final state.

### 2. Capture the trajectory

Record ordered model, tool, handoff, state, error, retry, and approval events with parent links.

### 3. Score layers separately

Evaluate plan relevance, tool behavior, recovery, policy, response, and verified outcome without one opaque grade.

### 4. Test variation

Repeat representative tasks and inject partial failures to measure stable success rather than one lucky path.

## What to measure

| Metric | What it measures | How to use it |
| --- | --- | --- |
| Verified task success | Runs that satisfy all required postconditions. | Use as the primary outcome while preserving partial results. |
| Milestone completion | Required intermediate states reached in the correct order. | Use gaps to locate the first divergence. |
| Trajectory efficiency | Steps, retries, latency, and cost among successful runs. | Optimize only after correctness and safety qualify. |

## Common mistakes

- Rewarding fluent explanations of work that was not completed.
- Requiring one exact path when several safe paths are valid.
- Ignoring failures that recover only by duplicating side effects.

## Practical checklist

- [ ] Define task contracts: Specify goals, milestones, allowed actions, budgets, approvals, and observable final state.
- [ ] Capture the trajectory: Record ordered model, tool, handoff, state, error, retry, and approval events with parent links.
- [ ] Score layers separately: Evaluate plan relevance, tool behavior, recovery, policy, response, and verified outcome without one opaque grade.
- [ ] Test variation: Repeat representative tasks and inject partial failures to measure stable success rather than one lucky path.

## Where EvalGate fits

EvalGate can reconstruct ordered trajectories, compare baseline and candidate runs, and combine path evidence with final-state checks.

[Analyze agent trajectories](https://www.evalgate.com/docs/platform/trajectory-analysis) or [start with the EvalGate quickstart](https://www.evalgate.com/docs/quickstart).

## Frequently asked questions

### What should I do first?

Specify goals, milestones, allowed actions, budgets, approvals, and observable final state

### How should the result be measured?

Runs that satisfy all required postconditions. Use as the primary outcome while preserving partial results.

### When is the change ready to ship?

Ship only after the protected cases pass, the primary metric clears its agreed boundary, and the team reviews the remaining failure modes instead of relying on one aggregate score.

## Related guides

- [https://www.evalgate.com/guides/know-if-ai-agent-finished-task](https://www.evalgate.com/guides/know-if-ai-agent-finished-task)
- [https://www.evalgate.com/guides/compare-ai-agent-trajectories](https://www.evalgate.com/guides/compare-ai-agent-trajectories)
- [https://www.evalgate.com/guides/test-ai-agent-tool-calls](https://www.evalgate.com/guides/test-ai-agent-tool-calls)