Evaluating an agent on complete tasks

Judge the final state and unintended effects

Define the complete task outcome, including constraints and prohibited side effects. Evaluate whether the final system state matches that outcome, not only whether individual tool calls were valid. An agent can perform several technically successful actions and still update the wrong record or leave a workflow incomplete. Include evidence requirements in the evaluation.

Worked example

Illustrative scenario; not a claim of measured client results.

For a website task limited to one domain, success means the intended pages are published and verified while unrelated sites remain unchanged. A sequence of successful uploads is insufficient if the archive included another project's files. Test scope preservation, recovery and the accuracy of the completion report alongside content correctness.

Checks, limitations and next decisions

Use representative tasks with ambiguity, interruptions and tool failures. Inspect unexpected writes and unresolved work after each run. Separate task success, safety boundary adherence and user effort so one favorable score cannot hide another failure. The evaluation should explain whether the agent can reliably finish the real job under the conditions in which it will operate.

A practical starting point

Work through these checks in order. Use the results to decide what needs to change in your own workflow.

1. Define the expected external result for each test.
2. Check permissions, data integrity and completion reporting.
3. Include tasks requiring clarification or refusal of an unauthorized step.

A mistake to avoid

Do not count a plausible final message as proof of successful execution.

How to check progress

Measure verified task completion and unintended changes together.