The Hardest Agentic AI Evaluation Cases Are the Ones That Look Almost Right

There is an uncomfortable truth about agentic AI evaluation: The easiest way to make an AI agent look good is to give it easy questions.

Give an agent an obvious fraudster and ask whether the transaction deserves scrutiny.

Give it a customer named John Smith and a sanctions record for Vladimir Putin and ask whether they are the same person.

Give it a document that plainly says, “DO NOT APPROVE,” and ask what action should be taken.

Run enough tests like these, and pretty soon you have a dashboard full of green checkmarks.

• Accuracy: 98%.
• Precision: 97%.
• Pass rate: 99%.

Everyone feels great, but there’s a problem:

You may have learned almost nothing about whether the agent is safe enough to deploy.

The hardest and most revealing evaluation cases are rarely the obviously right or obviously wrong ones. They’re the ones that look almost right.

A name that nearly matches.

A document that appears relevant actually belongs to somebody else.

An alias that is plausible but insufficiently supported.

A customer record contains several matching attributes alongside one critical contradiction.

A policy provision sounds applicable until you read the exception three paragraphs later.

An apparently persuasive piece of evidence should inform a decision but should not, by itself, trigger an action.

These are the cases where AI systems get into trouble, and increasingly they’re becoming the cases that should sit at the center of serious agentic assurance.

Easy negatives in agentic AI evaluations are comforting. Hard negatives are informative.

Machine-learning experts have named this problem: hard negatives.

Imagine you are training a system to distinguish pictures of dogs from pictures of airplanes.

A Boeing 747? Easy negative.

A wolf? Hard negative.

Both are technically “not a dog,” but only one tells you whether the model actually understands what makes a dog a dog. The same principle applies to AI agents.

Suppose an agent is responsible for researching customers against adverse-media sources.

An article about an entirely different person with a completely different name and location is an easy negative.But consider this:

David Rodriguez, age 47, lives in Miami and owns Rodriguez Imports LLC.

The agent finds an article about:

David A. Rodriguez, age 48, from Miami, investigated in connection with an import-export fraud scheme.

Now we have an evaluation case. Same name, similar age, same city, similar business … this appears to be highly relevant-looking evidence. But perhaps the date of birth is different. Or the business registration identifies a different person. Or one David Rodriguez has lived in Florida for 20 years while the other has never lived there.

The question is no longer simply, Can the agent find relevant information?

The question becomes: Can the agent resist acting on information that looks relevant but is not sufficiently supported?

That is a much harder problem, and it’s significantly closer to the problem you actually have in production.

Production is full of lookalikes

Real-world decision environments are messy. Here are just a few examples that can throw off accuracy:

• Names repeat
• Addresses change
• Records conflict
• Documents contain stale information
• Entities use aliases
• Sources disagree
• Policies contain exceptions
• Databases contain duplicates
• Context shifts the meaning of otherwise identical facts

Because generative AI is extraordinarily good at pattern recognition, it can sometimes create an unusual failure mode: It can become very persuasive precisely when it is almost right.

That’s different from the classic software failure.

Traditional software frequently fails loudly: Something crashes. A field is null. An API returns an error.

AI agents can fail elegantly, producing coherent explanations, citing seemingly relevant sources, connecting facts, and recommending action. The whole thing can look entirely reasonable except for one potentially devastating inconvenient detail: The conclusion is wrong.

This is why evaluating agents cannot simply be an exercise in counting how often they reach an acceptable answer. Users must understand where the decision boundary is by asking these questions at the very least:

• What evidence changes the answer?
• What evidence should not?
• What contradictions does the agent notice?
• How much ambiguity will it tolerate before acting?
• When should it abstain?
• When should it escalate?

And perhaps the most critical question: Can it distinguish “probably related” from “sufficiently established to take action”?

That distinction matters enormously in financial services. Failure here can be catastrophic and lead to nightmarish customer failures and the ensuing negative attention.

A good agent must know when evidence is insufficient

Many agentic workflows are fundamentally evidence-aggregation systems. The agent gathers information, weighs evidence, applies rules or policies, reaches a conclusion, and sometimes takes an action.

That last step changes the stakes. Consider an agent performing any of the following functions:

• fraud investigation;
• sanctions or watchlist screening;
• customer due diligence;
• complaint investigation;
• underwriting support;
• quality assurance;
• transaction monitoring;
• document verification;
• servicing;
• collections;
• insurance claims review;
• regulatory compliance monitoring.

In each of these contexts, finding evidence is only half the problem. The other half is deciding whether the evidence is good enough.

An adverse-media agent that misses an obviously relevant article has a problem, but an adverse-media agent that confidently attributes somebody else’s criminal history to your customer may have an even bigger one.

An underwriting agent that overlooks an obvious document may fail, but an underwriting agent that seizes on a superficially relevant number while ignoring the document’s context may fail much more subtly.

An agent that refuses everything uncertain will not be useful.

An agent that confidently resolves every ambiguity is unsafe.

Good agent design lives somewhere in between, and that means good evaluations have to live there, too.

The benchmark should attack the decision boundary

This is where many AI evaluation programs go wrong.

They assemble a representative sample of ordinary cases, run the agent, calculate aggregate performance metrics and declare victory.

Representative samples are important, but they’re insufficient.

If 90% of production cases are straightforward, then a purely representative benchmark may consist mostly of straightforward cases. Your agent can perform extremely well overall while systematically failing on the exact 5% of cases where judgment matters most.

The average hides the edge.

This is why sophisticated evaluation should deliberately concentrate some testing around the agent’s decision boundary. Here’s an example:

• Take a true-positive case and perturb it.
• Change one digit of the date of birth.
• Move the address next door.
• Replace a verified alias with an unverified one.
• Make the company names nearly identical.
• Introduce conflicting evidence from a second source.
• Remove one critical corroborating fact.
• Add an irrelevant but emotionally salient piece of evidence.
• Replace an authoritative source with a secondary source quoting it incorrectly.
• Keep everything else the same.

Then ask: Does the agent’s behavior change when it should?

That tells you far more than another hundred obvious cases.

Near-misses test reasoning, not recognition

This distinction matters because many benchmarks inadvertently reward recognition. The model sees familiar words, detects a pattern, and produces the expected classification.

But production systems often require something more demanding: discrimination among highly similar alternatives.

Not discrimination in the legal sense, but rather discrimination in the statistical sense: the ability to distinguish one thing from another. For instance:

• Same person versus different person.
• Relevant evidence versus dispositive evidence.
• Correlation versus causation.
• Policy rule versus policy exception.
• Strong signal versus weak signal.
• Action versus escalation.
• Confidence versus overconfidence.

A system that performs beautifully when the categories are far apart may perform badly when the categories begin overlapping. And unfortunately, production is mostly overlap.

The best negative examples should make you uncomfortable

There is a simple test I increasingly like for benchmark design: Could a smart human plausibly get this wrong at first glance?

If the answer is no, the example may not be doing much work. The best negative examples should create a moment of hesitation. You should look at the record and think, Hmm. That actually might be the same person.

Then you inspect the underlying evidence more carefully, and you realize it’s not.

That moment of uncertainty is valuable because it identifies exactly where the agent has to reason rather than merely recognize. Of course, hard negatives cannot simply be adversarial puzzles invented to trick the model. That creates another bad benchmark.

The examples should remain:

Realistic. They should resemble situations the agent can encounter in production.

Source-backed. The ground (or constructed) truth should be defensible.

Fair. The example should contain enough information for a reasonable system to reach the correct conclusion.

Representative of actual ambiguity. Difficulty for difficulty’s sake is not the objective.

The goal is not to make the agent fail.

The goal is to understand how it fails before a customer, employee, regulator or examiner discovers the answer for you.

Accuracy is not enough

This also points to a broader problem with agent evaluation.

A single aggregate accuracy number is usually an impoverished description of agent performance.

Imagine two agents:

• Agent A gets 96% of cases right but makes most of its mistakes on ambiguous identity matches, confidently taking action against the wrong person.
• Agent B gets 94% right but identifies ambiguous cases and reliably escalates them for human review.

The preferred agent depends on the workflow, but in many high-stakes environments, the second system may be considerably safer.

This is why agentic assurance should evaluate more than whether the final answer matched a benchmark. We should also be asking questions like:

• Did the agent rely on the right evidence?
• Did it ignore irrelevant information?
• Did it properly resolve contradictory sources?
• Did it distinguish facts from inference?
• Was its confidence calibrated?
• Did it follow the required policy?
• Did it know when to abstain?
• Did it escalate when uncertainty exceeded an acceptable threshold?
• Would small changes in irrelevant facts cause the answer to change?
• Would meaningful changes in critical facts cause the answer to change?

Those questions begin to tell us whether the agent is actually reasoning robustly.

Think less like a demo. More like a crash test.

There is a useful analogy here to automobile safety. Nobody would evaluate a new car by driving it around an empty parking lot for six hours and announcing, “Great news! Nothing went wrong!”

We deliberately crash cars.

We hit them from different angles.

We vary the speed.

We test different occupant sizes.

We simulate unusual conditions.

We examine how failures propagate.

Why?

Because the objective of a safety evaluation is not to demonstrate that the product works under ideal circumstances. It is to discover how the product behaves when circumstances stop being ideal.

AI agents deserve the same philosophy. A benchmark should not merely certify competence, it should expose fragility.

The harder the agent is to distinguish from a capable human during normal operation, the more important these tests become because sophisticated systems tend to produce sophisticated failures.

Edge cases are not outside the product

Teams sometimes describe difficult examples as “edge cases.” I dislike the phrase more each time I hear it.

It subtly implies that these cases sit somewhere outside the real product when they explicitly do not. If your agent operates long enough, edge cases become production cases.

A one-in-a-thousand error sounds rare, but given one million decisions, it happens a thousand times. This failure rate may be unacceptable, especially for those most affected by it.

And if the failure disproportionately occurs in one particular class of ambiguous cases, an aggregate metric may conceal an important systematic weakness.

So the goal should not be to eliminate every conceivable failure. That’s unrealistic.

The goal is to understand the system’s failure topology:

• Where does it break?
• Under what conditions?
• How consequential are those failures?
• Are they detectable?
• Are they recoverable?
• Do safeguards catch them?
• And does the system know when it does not know?

This work is the core of robust validation.

Agentic assurance should make systems harder to fool

The AI industry is understandably obsessed with making agents more capable. We strive to build better models, add more tools, create longer context windows, improve reasoning, introduce more autonomy. This list goes on, but capability and reliability are different properties.

As agents become capable of taking consequential actions inside banks, insurers and other regulated enterprises, we will need a corresponding discipline focused on answering a different question: How do we know this thing works well enough to trust?

You cannot answer that question with a demo.

You cannot answer it with twenty golden-path test cases.

And you certainly cannot answer it with a benchmark where every wrong answer looks obviously wrong.

You have to test the places where truth and error sit uncomfortably close together:

• The duplicate name.
• The misleading document.
• The plausible alias.
• The contradictory source.
• The technically relevant but insufficient evidence.
• The case that looks right until you inspect it one level deeper.

Unfortunately, these are not annoying exceptions to the evaluation.

They are the evaluation.

Because the real test of an intelligent agent is not whether it knows the difference between right and obviously wrong.

It is whether it knows the difference between right and almost right.

And that difference is where a very large portion of AI risk lives.