How to Measure AI Agent Task Success Rate
Define a denominator, verify outcomes, preserve partial completion, and connect quality to business results.
Short answer
Count a task as successful only when its observable postconditions are satisfied. Define eligible attempts, separate user cancellation and infrastructure failure, preserve partial and ambiguous outcomes, and report success by intent, difficulty, and risk—not just one blended percentage.
Key takeaways
- The denominator must include every eligible attempt, not only completed conversations.
- Use independent final-state checks whenever possible.
- Pair success rate with severity and resource use.
Signs this is the problem
Start by confirming the symptom before changing prompts, models, or infrastructure.
- Success is inferred from the agent saying it finished.
- Abandoned and timed-out tasks disappear from reports.
- Easy high-volume tasks dominate the aggregate metric.
Step-by-step approach
- 1
Define eligible attempts
Specify when a task starts, which cancellations are excluded, and how retries or resumed sessions are counted.
- 2
Write postconditions
Map each intent to final-state, response, policy, and user-confirmation evidence.
- 3
Classify outcomes
Use success, partial, failed, ambiguous, user-cancelled, and infrastructure-failed states with precedence rules.
- 4
Report useful slices
Break results down by intent, difficulty, tool path, environment, cost, latency, and failure topic.
What to measure
| Metric | What it measures | How to use it |
|---|---|---|
| Verified task success rate | Successful eligible tasks divided by all eligible attempts. | Use an independent verifier rather than self-report. |
| Partial completion rate | Attempts that satisfy some required postconditions. | Use missing milestones to prioritize fixes. |
| Cost per verified success | Total run cost divided by verified successful outcomes. | Compare efficiency only among quality-qualified paths. |
Common mistakes
- Changing the denominator between releases.
- Counting HTTP or tool success as task success.
- Treating ambiguous missing evidence as a pass.
Practical checklist
- Define eligible attempts: Specify when a task starts, which cancellations are excluded, and how retries or resumed sessions are counted.
- Write postconditions: Map each intent to final-state, response, policy, and user-confirmation evidence.
- Classify outcomes: Use success, partial, failed, ambiguous, user-cancelled, and infrastructure-failed states with precedence rules.
- Report useful slices: Break results down by intent, difficulty, tool path, environment, cost, latency, and failure topic.
Frequently asked questions
What should I do first?
Specify when a task starts, which cancellations are excluded, and how retries or resumed sessions are counted
How should the result be measured?
Successful eligible tasks divided by all eligible attempts. Use an independent verifier rather than self-report.
When is the change ready to ship?
Ship only after the protected cases pass, the primary metric clears its agreed boundary, and the team reviews the remaining failure modes instead of relying on one aggregate score.