How to Debug an AI Agent That Calls the Wrong Tool
Find whether wrong-tool failures come from tool descriptions, routing, context, arguments, permissions, or recovery logic.
Short answer
Debug wrong-tool calls by replaying the decision with the exact tool inventory and context the agent saw. Then separate selection errors from argument errors, permission failures, bad tool results, and orchestration mistakes; each requires a different fix.
Key takeaways
- Preserve the available tool definitions and context for every failing run.
- Test selection, arguments, execution, and result use as separate stages.
- A model change is rarely the first fix when tools overlap or return vague errors.
Signs this is the problem
Start by confirming the symptom before changing prompts, models, or infrastructure.
- Two tools have similar names or descriptions but different side effects.
- The selected tool is reasonable, yet its arguments omit a required scope or identifier.
- The tool succeeds, but the agent ignores the result and continues from stale context.
Step-by-step approach
- 1
Freeze the failing input
Save the user request, system instructions, tool definitions, prior messages, model settings, and runtime state. Reproducing only the final prompt usually removes the cause.
- 2
Label the failure stage
Decide whether the defect is no call, wrong selection, invalid arguments, execution failure, bad result interpretation, or unsafe retry. Do not use one 'tool failed' label for all six.
- 3
Reduce ambiguity
Make tool names, descriptions, schemas, and error contracts distinct. Remove irrelevant tools for the task and add explicit preconditions where two operations can be confused.
- 4
Replay protected cases
Run correct-choice, tempting-wrong-choice, malformed-result, timeout, and permission-denied cases. Require both the right action and safe handling when no action is allowed.
What to measure
| Metric | What it measures | How to use it |
|---|---|---|
| Tool selection accuracy | Correct tool or correct abstention for each labeled scenario. | Slice by intent because a good average can hide one dangerous route. |
| Argument validity | Calls that satisfy schema and scenario-specific constraints. | Track separately from selection to avoid fixing the wrong layer. |
| Recovery success | Runs that safely recover from an expected tool error without duplicate side effects. | Require this before enabling automatic retries on mutations. |
Common mistakes
- Adding more prompt instructions while leaving overlapping tools unchanged.
- Testing only happy-path tool responses.
- Rewarding any tool call when the correct behavior was to ask, deny, or abstain.
Practical checklist
- Capture the exact tool inventory visible to the agent.
- Give every tool a distinct purpose and return contract.
- Label selection, arguments, execution, and recovery separately.
- Include tempting but incorrect tools in regression cases.
- Verify side effects after retries or timeouts.
Frequently asked questions
Do longer tool descriptions improve selection?
Not automatically. Descriptions should be precise about purpose, preconditions, and exclusions. Repeated or overlapping prose can make routing less clear.
Should I expose every tool on every turn?
Usually no. A smaller relevant inventory reduces ambiguity and permission risk, provided routing does not hide a tool the task genuinely needs.
How do I test abstention?
Create cases where required information or authority is absent and assert that the agent asks for it or stops without invoking a side effect.