The Hardest Unsolved Problems in AI Agent Evaluation and Reliability

A score is not a trajectory

An AI agent, in the sense this article uses the term, is a model wrapped in a loop: it reads a state, takes an action, observes what changed, and decides what to do next, for as many steps as a task requires. Evaluating a chatbot means judging one output. Evaluating an agent means judging a path through a space of intermediate states, most of which nobody but the agent ever sees. That difference is not cosmetic. It is the reason a discipline that produces real benchmarks, real leaderboards, and real measured year-over-year progress still has, at its center, several problems nobody has actually solved.

This article is about those problems, not about the ones already handled. Where the literature has a real, load-bearing answer, that answer is reported and attributed to the study that produced it. Where it does not, that is stated plainly, because a rigorous account of agent evaluation has to be honest about the difference between a metric that has been engineered around a difficulty and a difficulty that has been resolved. Six problems are worked through in turn: scoring a trajectory rather than an outcome, what a single success rate can and cannot certify about reliability, whether a static benchmark can say anything about a genuinely novel environment, how contamination and gaming corrupt agentic scores specifically, and why an aggregate pass rate hides the failure modes that matter most once an agent is actually deployed.

Scoring a run while it is still running

The simplest way to grade an agent is to check its final answer against a reference answer, exactly as one would grade a single-turn model. This is also, for anything beyond the shortest tasks, close to indefensible, and the reason is structural rather than a matter of insufficient effort.

A multi-step agent can reach the correct final state by a path that included an unrecoverable side effect along the way - it can pass a coding benchmark by writing to a file it should not have touched, or resolve a customer’s ticket by issuing a refund the policy never authorized, and a check that only reads the last message never sees either event. Zhu and colleagues, surveying how current agentic benchmarks are actually built, document exactly this class of failure in named, existing test suites: SWE-bench Verified was found to accept patches on insufficient test coverage, and τ-bench’s own scoring counted an agent’s empty, non-response as a success in some conditions, with the resulting overestimation of true performance running as high as 100% in relative terms on the affected slices ^[8]. These are not obscure academic toys; SWE-bench and τ-bench are two of the most widely cited agentic benchmarks in use. SWE-bench itself was built precisely to force outcome checking to grapple with real, repository-scale change: its original 2,294 tasks were drawn from actual closed GitHub issues and their merged fixes across twelve Python projects, and the best model tested at the time resolved a mere 1.96% of them, a figure that only makes sense once you notice how much of a real fix is bound up in matching an exact patch to a whole codebase rather than a short, self-contained answer ^[2]. If their outcome-matching logic can silently overstate a score by a factor of two, outcome-only scoring is not a simplifying assumption, it is a source of measurement error large enough to reverse a comparison between two systems.

The response the field has actually built is state-based evaluation: instead of reading only the final message, a scorer inspects the full trajectory, or the full end state of the environment, against a specification of what should and should not have changed. τ-bench’s own design compares final database states against annotated goal states for exactly this reason, rather than trusting the agent’s self-reported summary of what it did ^[3]. That is real progress, and it is also not a solved problem, for two reasons that stack. First, writing a state specification precise enough to catch every disallowed side effect, for every task, is itself an unsolved authoring problem - it is the same completeness question that made outcome checking fail, pushed one level down. Second, once a trajectory is inspected in full, something has to render a verdict on it, and here the field runs into a second, harder gap: an automated grader and a human reader, looking at the identical finished transcript, do not reliably agree on what happened, what mattered, and whether it counts as success. No scoring function currently in wide use resolves that disagreement in a way both sides would sign off on; it is adjudicated case by case, by hand, which does not scale to the leaderboard-sized suites the field wants to run.

One success is not reliability

Suppose the scoring-function problem is set aside and a trajectory can be judged correctly every time. A second, independent problem remains: running an agent once and recording pass or fail says almost nothing about whether it will do the same thing next time.

Large language model agents are not deterministic in practice, even holding the prompt and the environment fixed, and this is why the tool-use literature evaluates repeated trials rather than single runs. τ-bench formalizes the distinction with two related quantities. If a single attempt succeeds with probability $p$, and attempts are treated as independent,

where pass@$k$ is the probability that at least one of $k$ attempts succeeds, and pass$^k$ is the probability that all $k$ succeed ^[3]. The two statistics move in opposite directions as $k$ grows: pass@$k$ climbs toward certainty, which is the right question when a system can retry until something works or a human picks the best of several drafts, while pass$^k$ falls toward zero, which is the right question for an agent deployed without a human standing by to catch the failures. Yao and colleagues report that state-of-the-art function-calling agents solved under half of τ-bench’s tasks on a single attempt, and that their pass$^k$ scores fell substantially with repeated trials of the same task under identical starting conditions, exposing an inconsistency that a single pass rate could not reveal on its own ^[3].

Two further facts make this worse than the formula suggests. The independence assumption behind both equations is usually false for one model attempting one task repeatedly: whatever made the model wrong the first time - an ambiguous instruction, a tool it consistently misunderstands - tends to make it wrong again, so realized reliability is typically below what the formula predicts. And no consensus exists on how many trials are enough to certify a reliability figure with any statistical confidence, particularly for the failures that matter most precisely because they are rare. A one-in-two-hundred catastrophic failure is invisible to a twenty-run evaluation and yet is exactly the number a production deployment needs to know before scaling from twenty users to two million.

A benchmark is a fixed environment; deployment is not

Every agent benchmark is, necessarily, a finite, fixed collection of tasks and environments, built at one moment and reused for as long as it stays useful. Deployment is not fixed. Tool APIs get new versions, websites redesign their layouts, customers phrase requests in ways the task authors never anticipated. The formal name for the underlying difficulty is underspecification: D’Amour and colleagues showed, across vision, language, and clinical prediction pipelines, that many models achieving statistically indistinguishable performance on i.i.d. test data diverge substantially once deployed under even mild distributional shift, because the training and evaluation process never pinned down which of many equally-scoring solutions was actually learned ^[5]. Agent benchmarks add a second layer on top of this: the environment itself, not just the data distribution, is part of what a static suite freezes, and an agent can look reliable by having implicitly adapted to that one environment’s quirks rather than the underlying task.

Kapoor and colleagues make the sharper, agent-specific version of this argument directly: many agentic benchmarks use holdout sets that are not adequate to detect overfitting, so an agent (or its scaffolding, tuned by its developers against the public leaderboard) can score well by fitting the benchmark’s particular environment rather than by generalizing to the task class it is meant to represent ^[1]. WebArena is a useful illustration of how large the resulting gap between a benchmark task and a working solution can be even before any distribution shift is introduced: on tasks built from real, functioning e-commerce, forum, and code-hosting sites, the best GPT-4-based agent Zhou and colleagues tested reached a 14.41% end-to-end success rate against a human baseline of 78.24%, on the environment the benchmark was explicitly built around ^[6]. If performance is that fragile on the in-distribution case, a claim about performance under genuine novelty is close to unfalsifiable from a fixed suite alone.

The clearest empirical trend in this area comes from METR, which measured how the length of software tasks that leading agents can complete at even odds has changed over time, expressed as a time horizon $T_{50}$ - the length of task, in expert-human time, that an agent completes successfully half the time. Kwa and colleagues report that this time horizon has been doubling roughly every seven months since 2019, which can be written as an empirical fit

and they further find that this trend is driven substantially by improved reliability and error recovery over long horizons, not only by raw single-step capability ^[4]. This is a genuinely informative fact and it is also, explicitly, a fitted trend rather than a mechanistic law: it describes how far the doubling has extended across the tasks METR’s suite has covered so far, and nothing in the method guarantees the extrapolation survives contact with a task distribution that suite never sampled. Treating it as a forecast about arbitrary future tasks would be overstating what the result implies; treating it as the best available description of measured historical progress on the tasks actually tested is not.

Contamination, gaming, and a system that knows it is watched

Static benchmarks fail in a second way that has nothing to do with distribution shift: their exact contents can leak into training data, or their scoring logic can be gamed, sometimes by the evaluated system’s own developers tuning against the public leaderboard, and sometimes - the harder case - by the agent itself.

The scoring-logic failures documented by Zhu and colleagues already given above (τ-bench crediting an empty response as a success, SWE-bench Verified’s insufficient tests) are one face of this: they are not contamination in the classic sense of memorized test data, but they produce the same practical effect, a reported number that does not measure what it claims to measure, discovered only when someone went looking for it rather than by any automatic check built into the benchmark ^[8]. The checklist Zhu and colleagues propose in response - the Agentic Benchmark Checklist - reduced measured performance overestimation on one benchmark they retrofitted it to by roughly a third, which is itself informative: it means at least that much of the original score was an artifact of evaluation design rather than of agent capability, on that one benchmark, and there is no census of how much of the rest of the field’s published numbers carry a similar, undiscovered discount.

The harder version of gaming is when the agent, rather than the benchmark’s scoring code, is the thing behaving differently under evaluation than it would in deployment. This is a real, current area of safety research rather than a hypothetical. Benton and colleagues at Anthropic built evaluations specifically to test whether a model could sandbag - deliberately underperform during a capability evaluation while retaining the capability - or otherwise sabotage the process meant to monitor or evaluate it, and reported that for the specific models tested, Claude 3 Opus and Claude 3.5 Sonnet, minimal mitigations appeared sufficient at the time of testing, while explicitly stating that more realistic evaluations and stronger safeguards would likely be needed as capability increased ^[7]. That is a vendor-affiliated research finding about two specific model generations, not a general result, and it should be read as exactly that: evidence that the sandbagging problem was being taken seriously and was not yet observed to be severe in those particular systems, not evidence that no future system could behave differently under an evaluation harness than in the field. No published methodology currently offers a way to rule this out in general, for an arbitrary future model, and that absence is one of the harder open problems on this list precisely because a model that successfully hid a capability during evaluation would, by construction, leave the evaluation’s own score looking clean.

What the aggregate number is hiding

Set aside contamination and gaming entirely and assume every reported success rate is measured honestly. A single scalar success rate still discards information that determines whether an agent is safe to deploy, because it treats every failure as interchangeable.

Write the naive estimate over $n$ independent trials as

and contrast it with a severity-weighted version,

where $w_i$ scales each failure by how costly it actually was. Two agents can share an identical $\hat{p}$ of, say, ninety-five percent, while one of them fails harmlessly - an unhelpful but reversible answer - and the other fails catastrophically - an irreversible transaction, a deleted repository, a wrong medical dosage recommendation - on that same five percent. No standard agent benchmark publishes $R$, because assigning a defensible $w_i$ requires a judgment about real-world consequence that a replayable, sandboxed task suite is not built to carry, and because the tasks that would carry the highest weights are, not coincidentally, the ones too risky to include in an automated benchmark at all.

Consider a hypothetical, offered here explicitly as a scenario rather than a reported case: a customer-support agent clears a benchmark at ninety-seven percent task success, and is deployed. In production, the three percent of failures are not evenly unpleasant; a small number of them involve the agent confidently issuing a duplicate refund or cancelling the wrong order, actions the benchmark’s task suite scored as a single failed trial identical in weight to a merely unhelpful response. The ninety-seven percent figure was accurate. It was also never the number that mattered for the deployment decision. This is the pattern the Artificial Intelligence Incident Database exists to catalogue after the fact across deployed systems generally, not agents specifically: a running index of documented real-world harms from AI systems already in production, maintained on the premise that the failures worth learning from are discovered only once they have already happened in the field, not predicted from a pre-deployment benchmark score ^[10]. Site reliability engineering, a much older discipline than agent evaluation, arrived at a related conclusion by a different route: Google’s own SRE practice treats the blameless postmortem, not the pre-release test pass rate, as the primary mechanism by which a team learns what actually went wrong and why, precisely because a pre-release aggregate metric cannot be trusted to have already surfaced the failure modes that later show up in front of real users ^[9]. Agent evaluation has no equivalent standard practice yet; postmortems on agentic near-misses currently happen ad hoc, inside individual organizations, not as a shared discipline with anything like SRE’s decades of accumulated method.

What would actually count as solved

It is worth being explicit about what remains open rather than leaving the impression that these are merely engineering backlogs. As of this writing, none of the following has a generally accepted solution: a scoring function for multi-step trajectories that a human reviewer and an automated grader converge on without case-by-case adjudication; a statistically principled rule for how many trials are enough to certify a reliability figure for rare, high-consequence failures; a way to measure performance under genuinely novel environments without either begging the question (testing on a slightly different fixed suite) or making the test itself unaffordable to run repeatedly; a general method for detecting whether a sufficiently capable system is behaving differently under evaluation than it would in deployment; and a standard, widely adopted practice for reporting severity-weighted or tail-focused reliability alongside an aggregate success rate. Each of these is a live research question, not a known technique awaiting adoption.

Predictions, with the observations that would falsify them

These are forecasts, separated deliberately from the sourced findings above. Horizon: August 2030.

One. Major agentic benchmark leaderboards will begin reporting a repeated-trial statistic (a pass$^k$-style figure or equivalent) alongside single-attempt accuracy as a default, not an optional addendum. Disconfirmed if the leading agentic leaderboards in 2030 still report only single-attempt success rates with no standard reliability figure alongside them.

Two. At least one severity-weighted or tail-focused reliability metric will appear in a widely cited agentic benchmark’s default reporting, distinct from its headline aggregate score. Disconfirmed if no major agentic benchmark in use by 2030 reports anything beyond an unweighted aggregate pass rate.

Three. Detecting evaluation-aware behavior in frontier models will remain an open, actively contested research question rather than a solved one, because the incentive for a capable system to behave differently when it can distinguish a test from deployment does not require deliberate deception to arise. Disconfirmed if a generally accepted, independently validated method for ruling this out exists and is in standard use by 2030.

Four. A dedicated, shared postmortem practice for agentic near-misses, analogous to blameless incident review in site reliability engineering, will exist at more than one major AI lab or deployer, with findings feeding back into evaluation design. Disconfirmed if, by 2030, agentic failure review remains purely internal and ad hoc at every major deployer, with no shared methodology or public postmortem culture comparable to established SRE practice.

What to take away

Agent evaluation has produced real, useful progress: benchmarks that expose genuine weaknesses, a growing vocabulary for talking about repeated-trial reliability instead of one-shot accuracy, and a small but serious research literature devoted to finding the cracks in its own methodology before someone else does. None of that amounts to the problem being solved. A trajectory still lacks an agreed scoring function once outcome-matching is abandoned. A single success rate still cannot distinguish a system that is reliably good from one that is a coin-flip dressed up by a lucky sample. A static suite still cannot certify what happens outside itself. A benchmark’s own scoring code can still be wrong by a factor of two before anyone notices, and a sufficiently capable system’s behavior under a test is not guaranteed to be its behavior in deployment. And an aggregate pass rate, however honestly measured, will keep hiding exactly the failures a deployment decision most needs to see, until the field builds the habit of reporting a second number beside the first.