Imagine I tell you that in our latest round of AI agent evaluations, our newest agent passed 95% of them. Sounds pretty good, right?
Now, imagine I tell you something else: All of the tests involved complete information.
Every external tool worked.
Every document was current.
Every customer had a unique identity.
No source contradicted another source.
No task required more than one consequential step.
And none of the tests involved situations where the correct answer was to do nothing.
Still impressed by the 95%?
Probably less so.
This gets at what I think is becoming one of the biggest problems in AI agent evaluation: We are getting very good at measuring pass rates before we have gotten very good at measuring what the tests actually cover.
A company may have 500 evals, or even 5,000. It may have a beautiful dashboard with green bars everywhere.
- Pass rate: 95%.
- Hallucination rate: 1.8%.
- Tool-call accuracy: 97%.
And yet, the company may still have surprisingly little evidence that the agent will behave safely and reliably in production, because the important question is not merely, How many evals passed?
The important question is, What parts of the agent’s failure surface did those evals actually test?
That’s a coverage problem because coverage (not raw test count) is rapidly becoming one of the most important concepts in agentic assurance.
95% accurate at what, exactly?
We have become accustomed to evaluating AI systems with aggregate numbers:
- Accuracy
- Precision
- Recall
- Pass rate
- Task completion
These are useful metrics, but every percentage has an implicit denominator.
If an agent is 95% accurate, we should immediately ask: Across which kinds of cases?
An agent might be excellent when all of the necessary information is present and terrible when one critical field is missing.
It might perform perfectly when its search tool returns a result and hallucinate an answer when the tool times out.
It might correctly identify relevant information but fail to notice that the information is three years out of date.
It might handle every individual step correctly but make the wrong decision when those steps are combined.
It might perform well when the appropriate action is obvious but badly when the correct action is to abstain, to suppress an alert, or to escalate to a human.
Average performance can conceal all of this, making a 95% score only as meaningful as the territory underneath it.
Think like a mapmaker
Suppose someone handed you a map of the United States containing exquisite detail.
Every street in Manhattan is labeled, along with every subway entrance, every restaurant, every alley. We’re talking thousands and thousands of data points.
There is just one problem.
California, Texas, Florida and the other 46 states are blank.
Would you call that a detailed map of America?
Of course not.
You would say it has extraordinary depth in one tiny part of the country and terrible coverage of the rest.
Agent evaluation can suffer from exactly the same problem.
A team may have hundreds of extremely detailed tests concentrated around the agent’s happy path.
That creates the appearance of rigor.
But adding the 501st variation of a behavior you already understand may tell you much less than adding the first test of an entirely different failure mode.
This distinction between test volume and test coverage matters enormously.
One thousand cases testing the same underlying behavior are not necessarily more informative than one hundred cases spanning ten different failure surfaces.
Agents have a failure surface
Traditional software testing has long wrestled with the concept of coverage.
Did we execute this line of code? This branch? This function? This path through the system?
AI agents make the problem considerably harder. Their important “branches” are often not explicitly written into code. They emerge from the combination of:
- Prompts
- Models
- Tools
- Retrieved information
- Memory
- Policy rules
- External systems
- Intermediate reasoning
- Prior actions
That means the relevant unit of coverage is not simply whether a particular code path executed. It’s becoming increasingly important to ask whether we exercised a meaningful behavioral condition.
Consider an agent investigating a customer.
- Did we test what happens when two people have the same name?
- Did we test the same name and same city?
- Did we test an alias?
- Did we test an alias supported by one weak source?
- Did we test contradictory dates of birth?
- Did we test a missing date of birth?
- Did we test stale information?
- Did we test what happens when the search tool fails?
- Did we test what happens when the search tool returns incomplete information?
- Did we test what happens when the evidence is relevant but insufficient to justify action?
Those are all different pieces of the system’s failure surface, and an agent can perform beautifully on one while failing spectacularly on another.
Missing information is not an edge case
One of the easiest ways to make an agent look good is to give it complete information.
Production rarely returns the favor.
Documents are missing.
Fields are blank.
API responses fail.
Databases disagree.
Customers omit information.
Search results are partial.
Pages disappear.
OCR mangles a number.
The question for an agent is therefore not just: Can you reason correctly from good information?
It is also: Can you recognize when the information needed to reach a conclusion is not actually available?
These are very different capabilities.
A poorly designed agent may treat missing evidence as negative evidence.
It may fill gaps with inference. It may silently make assumptions. It may confidently complete a task that should have been escalated. Or, it may simply fabricate the missing bridge between two facts.
Any serious evaluation program should therefore deliberately vary information completeness.
Take the same scenario, and run it with all necessary evidence. Then, remove one important field.
Then remove two.
Then introduce a contradiction.
Then make one source inaccessible.
Then make the evidence ambiguous.
What does the agent do?
A production-ready system should not merely know things. It should behave appropriately when it cannot know things.
Tools create another failure surface
Agents do more than generate text. They search databases, call APIs, read documents, send emails, update records, query internal systems, calculate values, and initiate workflows.
That makes tool behavior part of the agent. Unfortunately, tools fail.
They time out.
They return null.
They return malformed data.
They return stale data.
Permissions change.
Schemas change.
Queries retrieve the wrong record.
A tool may even return a plausible answer that happens to be incorrect. Yet, many agent eval suites implicitly assume that every tool behaves exactly as intended. That is a little like testing an autonomous car only on roads where every traffic light works.
A meaningful agent evaluation should ask the following:
- What happens when the tool returns nothing?
- What happens when it returns too much?
- What happens when two tools disagree?
- What happens when the first tool fails but the second succeeds?
- Does the agent retry?
- Does it switch tools?
- Does it tell the user what happened?
- Does it proceed anyway?
- Does it invent the missing result?
And critically: Does the agent understand the difference between “the tool found no evidence” and “the tool failed to search for evidence”?
Those two states can look deceptively similar, but they are not remotely the same.
Multi-step agents create combinatorial risk
Things become even more interesting when agents perform workflows rather than isolated tasks. Imagine a five-step process.
- Identify the customer.
- Retrieve relevant documents.
- Interpret those documents.
- Apply a policy.
- Take an action.
Suppose each step works correctly 98% of the time. That sounds excellent, but even under an extremely simplified assumption of independent errors, the probability that all five steps execute correctly is only about 90%.
And real agent errors are not independent.
One mistake contaminates the next step.
The wrong identity retrieves the wrong document.
The wrong document produces the wrong interpretation.
The wrong interpretation triggers the wrong policy.
The wrong policy leads to the wrong action.
The error compounds.
This means evaluating each component independently is necessary but insufficient.
You also have to test cross-step interactions.
Can the agent recover when an earlier step produced a questionable result?
Does it verify critical facts before a consequential action?
Does confidence from one stage improperly carry into the next?
Can a small initial error cascade into a large downstream failure?
The system may pass every unit test and still fail the workflow.
One of the hardest things to test is “do nothing”
There is another category of coverage that I think is badly neglected: suppression cases. These are situations where the agent encounters something interesting but should not take action.
A potentially relevant article, but it’s about another person.
A possible discrepancy that turns out to be immaterial.
A suspicious transaction with an innocent explanation.
A document containing a keyword that superficially triggers a rule but does not satisfy the actual policy.
A customer record that looks incomplete but contains enough information elsewhere to resolve the issue.
Many benchmarks overweight positive action.
Find the problem.
Flag the record.
Take the next step.
But safe systems must also learn restraint.
Sometimes, the correct action is:
- Do not flag.
- Do not escalate.
- Do not send.
- Do not change the record.
- Do not infer.
- Do not proceed.
In regulated environments, false positives can themselves create serious harm. A system that catches 100% of suspicious cases by flagging absolutely everyone is not a particularly good or useful system.
This is one reason I think suppression behavior deserves to be an explicit coverage category.
Can the agent recognize when not to act?
Build a coverage matrix, and not just a case library
One practical implication follows from all of this: Teams should stop thinking about eval suites only as collections of examples. Instead, they should also think of them as coverage matrices.
Imagine one dimension describing the underlying task:
- Identity resolution
- Document interpretation
- Policy application
- Calculation
- Tool use
- Communication
- Action execution
Now, add another describing operating conditions:
- Complete information
- Missing information
- Conflicting information
- Stale information
- Ambiguous information
- Tool failure
- Unexpected format
Then, add consequence:
- Correct action
- Incorrect action
- Abstention
- Escalation
- Suppression
Then, perhaps another dimension:
- Single-step
- Multi-step
- Cross-too,
- Cross-source
Suddenly, the relevant question changes.
Instead of, “How many evals do we have?” You can ask, “Which cells in the matrix have we never tested?”
That is a much more interesting conversation.
You may discover that you have 400 examples of successful document extraction and zero examples of an agent confronting two contradictory documents.
Or 200 sanctions-screening cases and none in which a plausible identity match should be rejected.
Or hundreds of tool calls without a single test of a timeout.
The empty cells are often more informative than the full ones.
Coverage is not the same thing as testing everything
Of course, there is a problem. The possible state space for an AI agent is effectively infinite.
We cannot test every prompt, document, sequence, tool response, user behavior, or combination. Therefore, coverage cannot mean exhaustive testing.
The objective is to build a structured picture of material failure modes:
- What can go wrong?
- Which failures matter most?
- Which operating conditions make those failures more likely?
- Which controls are supposed to catch them?
- And what evidence do we have that those controls actually work?
This begins to look much less like benchmarking and much more like traditional risk management, which is exactly what needs to happen.
Risk-weight your coverage
Not every blank cell matters equally. Suppose an internal research agent occasionally formats a citation incorrectly. That may be annoying. Now, suppose an autonomous servicing agent occasionally sends money to the wrong bank account.
That is different. Potentially catastrophic.
Evaluation coverage should therefore be risk-weighted.
For each behavior or failure mode, consider some combination of the following:
- Likelihood. How often could this condition arise?
- Severity. What happens if the agent gets it wrong?
- Detectability. Will somebody notice the failure?
- Recoverability. Can it be fixed after the fact?
- Autonomy. Can the agent act without human review?
The highest-risk scenarios deserve the deepest testing.
This also explains why a single “agent score” can be misleading. An agent that fails 10% of low-consequence formatting tasks and never fails a money-movement control might be safer than an agent with 99% aggregate performance that occasionally sends a payment incorrectly.
The average obscures consequence.
Agentic assurance has to restore it.
Green dashboards can create false confidence
None of this means dashboards are bad. Quite the opposite, in fact. Organizations need dashboards, metrics, trend lines, and pass rates. But the dashboard should answer two separate questions:
- How well did the agent perform on the tests we ran?
- How much of the relevant risk surface did those tests cover?
Those are not the same question. A green dashboard without coverage information is a little like being told, “Every bridge we inspected is structurally sound.”
Wonderful, but it’s lacking context.
How many bridges did we inspect?
Which ones?
Did we inspect the oldest ones?
The busiest ones?
The ones near fault lines?
The ones that use an unusual construction method?
The inspection result matters, and so does the sampling frame.
The confidence unit should be the failure mode, not the test case
This leads to a useful shift in how we talk about agent evaluations. The goal is not to accumulate the largest possible number of tests. Rather, the goal is to accumulate evidence about the agent’s behavior across the failure modes that matter.
That makes the basic unit of confidence something closer to: “We understand how this system behaves under this condition.”
We have tested for the following:
- Identity ambiguity
- Missing evidence
- Contradictory sources
- Stale information
- Tool outages
- Cross-step error propagation
- Abstention
- Suppression
- Escalation
And, just as importantly, we know what we haven’t tested.
That last part is important. A good assurance program should not merely tell you what you know. It should make your unknowns visible.
Agentic assurance is ultimately about knowing what you have not tested
There is a temptation in AI to reduce confidence to a number, but complex systems do not become trustworthy merely because we can summarize them neatly.
The better question is, What would have to happen for this agent to fail in production (and have we actually tested that)?
If your agent works perfectly when the information is complete, test it when information is missing.
If it works when every tool succeeds, break a tool.
If it handles obvious identities, give it lookalikes.
If it performs individual steps correctly, test the full chain.
If it knows when to take action, test whether it knows when not to.
The objective is not to produce a dashboard that looks safe. It’s to discover where the system is not safe yet, because the most dangerous part of an eval suite may not be the tests your agent failed.
It may be the tests you never thought to run.
So yes, you have evals.
But before you take comfort in that 95% pass rate, ask one more question: Do you have coverage?
