How to Test AI Agents Before They Go Live
To test an AI agent, run it repeatedly against a set of real cases, score what it decided and did as well as what it said, and then attack it on purpose. Ordinary software tests aren't enough, because the same input can produce a different path on each run.
This doesn't make agents untestable. It means you measure pass rates over many runs, not a single pass or fail.
Step 1: Build a test set from real work
Collect 50 to 100 real cases to begin with. Include:
- Typical cases that make up most of the work
- Hard cases that confuse new staff
- Cases it should refuse or escalate
- Incomplete or contradictory requests
For each, write down what a correct outcome looks like. Ask the people who do this job today, because they know where the difficult cases are.
Whenever the agent fails in production, add that case to the set. Over time, this becomes one of your most valuable assets.
Step 2: Score more than the final answer
An agent can reach the right answer in a dangerous way. Check four layers:
| Layer | Question |
|---|---|
| Outcome | Was the task completed correctly? |
| Path | Did it use the right tools with the right inputs? |
| Limits | Did it stay within its permissions and rules? |
| Cost | How many steps, how long and how much money did it take? |
Use code to check whatever you can: the correct record changed, the refund stayed under the limit, the right tool was called. Use a person or a second model to judge things like tone. Spot-check that second model as well, since it can be wrong too.
Step 3: Run each case more than once
Run each case several times. An agent that passes 7 of 10 runs is not the same as one that passes all 10, even though a single test could make them look alike. Look closely at the variation. A case that sometimes fails often points to an unclear instruction.
Step 4: Test failure conditions
Make things go wrong on purpose:
- A tool times out or returns an error
- A search returns nothing
- The data conflicts
- The user changes their mind partway through
- The task can't be completed at all
A good agent stops, retries sensibly or escalates. A bad one invents a result and reports success.
Step 5: Attack it
Hide instructions in the emails and documents it reads, and try to get it to reveal data or exceed its limits. Check that it stops at its step and spending caps. Run these attacks in a test environment with fake data, never with live customer records. Our AI agent security checklist covers what else to review.
Step 6: Set a pass mark and shadow it
Decide what score is good enough before you see the results. It should be based on what a mistake would cost and how well people do this job today. Some checks should have zero tolerance. An agent that exceeds a payment limit should fail outright, whatever its average score.
Then run it in shadow mode, where it makes decisions on real work but people still do the job. Compare the results before you allow it to act.
After launch
Testing doesn't end at launch. Run the full set whenever you change the model, prompts or tools, since any of these can change behavior. Sample real conversations regularly, and watch task success, escalation rate and cost per task for drift.
Talk to us
Have an agent that works in demos but makes you nervous about production? DPR Software Labs can help you build the tests that show whether it's ready.