Testing agents with simulated tool failures

Test failures that occur after partial success

Build tests around realistic tool behavior: timeouts, malformed responses, permission errors, rate limits and successful writes with lost acknowledgments. Include failures between steps in a multi-system workflow. Use a suitable test environment and synthetic records so the evaluation does not create real-world consequences.

Worked example

Illustrative scenario; not a claim of measured client results.

For an agent that creates a CRM lead and schedules a follow-up, simulate the lead being created while scheduling fails. The correct response should preserve the lead identifier and report the remaining step. Repeating the entire workflow may create another lead. Test whether the agent distinguishes a retryable service error from a permission boundary it must not bypass.

Checks, limitations and next decisions

Evaluate the resulting state, not only the wording of the agent's explanation. Check for duplicates, unintended edits and misleading success claims. Add discovered failures to a regression set. The purpose is to learn how the complete workflow behaves under stress, including whether an operator can understand and recover the partial result.

A practical starting point

Work through these checks in order. Use the results to decide what needs to change in your own workflow.

1. Create controlled failure cases for each important tool.
2. Verify that the agent distinguishes retryable from permanent errors.
3. Check that user-facing reports match what actually completed.

A mistake to avoid

Do not test destructive failures against live customer systems.

How to check progress

Measure safe recovery and truthful completion reporting by failure type.