Measuring AI Agent Reliability: What the Evidence Actually Supports

A green checkmark is not a measurement

A demo succeeds. The transcript is clean, the tool calls resolve, the final answer matches the target, and someone writes down that the agent “solved” the task. That sentence is doing more work than it can support. One successful run establishes that success is possible under the conditions of that run. It does not establish that success is likely under those conditions, still less that it is likely under the conditions a production system will actually face. Possibility and reliability are different claims, they require different evidence, and the gap between them is exactly where most public claims about agent capability quietly slide from one to the other.

The confusion is understandable, because a single-trial outcome is not obviously a bad estimator. If an agent really does succeed with high probability, most single trials will show success, and a skeptic demanding repeated trials can look like they are moving the goalposts. The trouble is that a single binary outcome cannot distinguish a system that succeeds nineteen times out of twenty from one that succeeds barely more than half the time — both will very plausibly hand you a pass on a given run, and the run alone carries no information about which one you were looking at. Turning “it passed” into “it is reliable” requires a sample, not an anecdote, and it requires the right statistic computed on that sample. This article works through what that statistic actually is, what the published research says it looks like once anyone bothers to measure it, why the answer gets worse as tasks get longer, and why a number obtained by careful, well-resourced elicitation still tends to diverge from what a deployed system delivers.

None of what follows is exotic. It borrows the combinatorics behind two specific metrics that different research groups introduced for different reasons, a statistical framework for treating evaluation scores as estimates rather than facts, and the documented experience of people who build and evaluate agent harnesses for a living. The point of assembling it in one place is that these pieces are usually cited separately, when the honest description of “is this agent reliable” requires all of them at once.

Two different questions wearing one number

Start with the metric that made repeated sampling a standard practice. When OpenAI’s Codex team evaluated a code-generating model against the HumanEval benchmark, they needed a way to score the strategy of drawing several candidate solutions from the model and keeping the best one. The obvious approach — generate exactly $k$ samples per problem and check whether any of them pass — has an undesirable property: it is a valid estimate but a high-variance one, since it throws away information every time you happen to generate more or fewer than $k$ samples. Their fix was to over-sample: draw $n$ total samples per task, observe how many of them, $c$, actually pass, and then compute the exact probability that a random draw of $k$ items from those $n$ would contain at least one success ^[2].

That combinatorial form is the unbiased estimator: it uses every one of the $n$ observed outcomes rather than discarding some of them, and it has materially lower variance than the intuitive shortcut of raising a single measured success rate to a power. The paper is explicit that the naive alternative is biased, not merely noisier, so the difference is not a technicality — using the wrong formula changes which system looks better ^[2]. With $n=100$ samples per problem, their Codex model solved 70.2% of HumanEval problems by this best-of-many-samples criterion, against 28.8% at pass@1 ^[2]. That gap is not a measurement error. It is the honest size of the difference between “can this model ever produce a correct solution” and “does its single best guess happen to be correct.”

Pass@k answers a specific question: given a budget of $k$ independent attempts, what is the chance that at least one succeeds? That is the right question for a search-and-verify workflow, where a cheap checker can identify the one attempt that worked among several candidates. It is the wrong question for almost everything else an agent does, because most agentic tasks do not offer a free, cheap oracle that can pick the winning attempt out of a pile of candidates after the fact — the “attempt” is the deployment. For that setting, Yao and colleagues, building the tau-bench benchmark for tool-using agents interacting with simulated customers under domain policies, proposed the complementary statistic: the probability that every one of $k$ independent trials on the same task succeeds, rather than at least one of them ^[3].

The two formulas share the same $n$, $c$, and $k$ and the same combinatorial machinery, and that similarity is precisely the point: they are estimating two different probabilities from the same data, one about the best of $k$ attempts and one about the consistency of all $k$ of them. A system can score well on the first and badly on the second, and a report that only states pass@k while an agent is actually being deployed to run the same class of task repeatedly, unsupervised, with no way to pick the lucky run, is answering a question nobody was asking.

What happens when someone actually checks

The tau-bench numbers make the distinction concrete rather than abstract. Testing function-calling agents built on frontier models against realistic retail and airline customer-service scenarios — each requiring the agent to follow domain policy, call the right APIs, and leave the backend database in a state matching an annotated goal — the authors found that even strong agents succeeded on well under half of tasks at pass@1, and that reliability across repeats fell sharply as $k$ increased: an agent’s pass$^k$ score in the retail domain dropped below 25% by $k=8$, and airline-domain performance, already lower at $k=1$, degraded further from there ^[3]. Put in words rather than symbols: an agent that a casual observer would call “usually right” on any given try is, on the paper’s own numbers, more likely than not to slip up at least once if you ask it to repeat the same class of task eight times. The authors’ own conclusion is not that the agents were unusually bad — several were the strongest publicly available systems at the time — but that consistency, not raw single-shot competence, was the thing missing ^[3].

That finding is not isolated to conversational, tool-using agents narrowly. AgentBench, an earlier and broader benchmark spanning eight environments — including operating-system control, database interaction, web shopping, and web browsing — tested twenty-five language models as autonomous agents across many rounds of interleaved reasoning and action per task. Its authors identified “poor long-term reasoning, decision-making, and instruction following” as the primary obstacle separating usable agents from unusable ones, a description that lines up with the tau-bench data: the weak point was less a single wrong fact than an accumulating drift across a multi-step interaction ^[10]. Between the two benchmarks, the pattern that emerges is that agent failure is disproportionately a consistency problem rather than a ceiling problem — most agents can, at least once, execute a given class of task correctly. Whether they can be trusted to do it the same way every time is the separate and harder question, and it is the one production deployment actually asks.

The mean conceals more than it reveals

Even when a team does report a rate rather than a single anecdote, a rate on its own understates how uncertain that number is. Evan Miller’s statistical treatment of language-model evaluation makes an argument borrowed from experimental science generally: an evaluation score is an estimate drawn from a finite, noisy sample, not a fact about the system, and it should be reported the way any other science reports a measurement — with an uncertainty attached ^[4]. His worked illustration is a useful gut check on how much data “enough” actually requires. To reliably detect an absolute difference of three percentage points between two systems, with conventional statistical standards for false positives and false negatives, the calculation runs as follows.

Plugging in a 5% false-positive rate, 80% power, and a three-point difference gives roughly 969 independent questions — not thirty, not one hundred, close to a thousand ^[4]. Most published agent evaluations, run against benchmarks with a few dozen to a few hundred tasks because each task requires an expensive, stateful, sometimes hours-long rollout, fall well short of that bar by construction, which means many of the differences reported between agent versions or between competing systems are, on this analysis, indistinguishable from noise even when they are reported as a clean percentage-point win. Miller’s own demonstration on real evaluation data found gaps of 3.1 and 2.7 percentage points on two separate benchmarks that did not clear statistical significance once treated properly, and he further showed that once repeated comparisons on the same fixed question set are accounted for with clustered standard errors, the true uncertainty can run more than three times larger than the naive calculation suggests ^[4]. His practical recommendation is unglamorous: report the standard error next to the mean, the way other sciences do, rather than a bare percentage that invites over-reading.

A second, independent line of evidence shows the same instability from a different angle. Hochlehnert and colleagues re-examined a wave of reported reinforcement-learning gains on mathematical reasoning benchmarks and found the results were acutely sensitive to implementation details that are rarely reported in full: decoding parameters, prompt formatting, random seeds, and even hardware and software configuration all moved scores by amounts comparable to, or larger than, the improvements being claimed ^[5]. Once they reassessed the same methods under standardized evaluation practices, many of the reinforcement-learning gains shrank to modest amounts, well below the original claims, with a tendency to overfit small benchmarks rather than generalize ^[5]. Their subject was reasoning benchmarks rather than agent trajectories specifically, so the finding should not be read as a direct claim about agent evaluation — but the underlying statistical failure mode is the same one Miller’s framework predicts, and it generalizes to any evaluation regime built on a small number of trials with unreported seeds: a single seed, prompt template, or decoding setting is one draw from a distribution of plausible results, and reporting it alone as “the” score hides exactly how much that number would have moved under a different, equally defensible draw.

Length is not a neutral axis

Everything so far treats a task as a single opaque trial. Real agent work is not that; it is a sequence of dependent steps, and the sequence’s length changes the measured reliability even if nothing about the underlying per-step competence changes. The clearest empirical treatment of this comes from METR, which defined a “50%-task-completion time horizon” — the length of task, measured in the time a skilled human would need, at which a given AI system’s success probability crosses 50% ^[1]. Timing domain experts against a mix of software and machine-learning tasks assembled from existing benchmarks plus newly built shorter ones, they found that a frontier model of the March 2025 era, Claude 3.7 Sonnet, had a 50% time horizon of roughly fifty minutes, and that this horizon for the strongest available model has been doubling approximately every seven months since 2019, with some evidence the trend accelerated after 2024 ^[1].

Two things about that result matter more than the specific horizon length, which will already be dated by the time this is read. First, the metric is explicitly a reliability metric, not a raw-capability one: the authors attribute the growth in time horizon primarily to “greater reliability and ability to adapt to mistakes,” alongside logical reasoning and tool use, rather than to any single new trick ^[1]. That framing matches the picture built up in the previous sections — the binding constraint on longer tasks is not whether the model ever understands what to do, but whether it keeps not-failing for long enough to finish. Second, because the horizon is defined at the 50% success threshold, a task twice as long is not twice as hard in an additive sense; it is a task where the same system’s success probability has already fallen to a coin flip, and pushing the task longer still pushes that probability down further, exactly the pattern predicted by treating each additional step as another opportunity for something to go wrong that the model must first detect and then recover from. A headline capability score measured on short benchmark tasks and reported without reference to task length is, on this evidence, silently reporting a number from the flat part of a curve that gets much steeper once the task in front of a real user is longer than a benchmark item.

The gap between an elicited number and a deployed one

Every number discussed so far assumes the evaluator got the most out of the system being tested — that the prompting, the tool access, and the scaffolding were good enough that a low score reflects the system’s actual ceiling rather than a badly configured harness. That assumption is not automatically true, and the organizations that build evaluation suites for a living say so about their own work. METR’s own update on its general capability evaluation suite is candid that its elicitation was “only very basic,” that many of its human comparison baselines rest on just one or two runs, and that measured agent performance appeared to plateau well before its token budget was exhausted — a pattern consistent with the ceiling being set by the scaffolding rather than by the underlying model ^[9]. The organization frames this explicitly as a lower bound subject to revision as elicitation techniques improve, not as a settled fact about what a given model can do ^[9]. A number obtained under weak elicitation understates a system’s true capability in one direction; a number obtained on a friendly, curated demo can overstate its reliability in the other. Both distortions are real, they run opposite ways, and a single reported figure cannot disclose which one it suffers from.

Anthropic’s own account of building a production multi-agent research system describes the second distortion directly, from the inside of a live deployment rather than a benchmark. Their engineers report that “agents make dynamic decisions and are non-deterministic between runs, even with identical prompts,” and that “minor changes cascade into large behavioral changes,” a property they say makes writing conventional tests for a long-running agentic process remarkably difficult, because a single early misstep can send a run down an entirely different, unpredictable trajectory rather than simply producing a slightly worse version of the same output ^[6]. They also note that traditional pass/fail testing does not transfer cleanly to open-ended agent tasks, since “even with identical starting points, agents might take completely different valid paths to reach their goal,” which forces evaluation toward rubric-based grading of final outcomes and heavy reliance on human review to catch failure modes automated grading misses ^[6]. This is a materially different failure surface than the clean, resettable, single-question trials that pass@k and pass^k were built to summarize, and it is the reason both metrics should be read as a floor on the measurement problem rather than a complete solution to it: they discipline what can be counted, but a production agent’s stateful, path-dependent behavior generates variance that a fixed-question benchmark, however large, does not fully capture.

What the official guidance actually recommends

Faced with this, the platforms that ship agent harnesses have converged on broadly similar practical advice, and it is worth reading their documentation as a response to the measurement problem above rather than as generic engineering hygiene. OpenAI’s guidance for evaluating agent workflows recommends starting from traces — the complete record of an individual run’s model calls, tool calls, guardrail checks, and handoffs — to debug specific failures, then explicitly moving beyond individual traces to “repeatable datasets and eval runs” once a team understands what a successful run looks like, so that later changes can be benchmarked against a consistent, repeated sample rather than judged from a handful of anecdotal transcripts ^[7]. The documentation is also explicit that grading should evaluate the trajectory — tool selection, routing, and policy compliance along the way — rather than only the final answer, which is the same trajectory-versus-outcome distinction that separates a task that happened to finish correctly from one that finished correctly because each intermediate decision was sound ^[7].

Anthropic’s guidance for building agents with the Claude Agent SDK makes a closely related point about where verification has to live inside the agent loop itself, arguing that “agents that can check and improve their own output are fundamentally more reliable” because they catch mistakes before those mistakes compound across further steps, and recommending that teams “build a representative test set for programmatic evaluations… based on customer usage” once they observe that performance varies as new features are added ^[8]. Both pieces of guidance converge on the same structural fix: replace a one-off demo, or even a one-off benchmark run, with a standing, repeated evaluation harness that a team re-runs on a schedule, against a fixed and preferably large sample of representative tasks, so that pass@k, pass^k, and their confidence intervals become numbers a team actually has on hand rather than a statistical ideal nobody’s evaluation budget supports.

A short protocol for reporting agent reliability

Collecting the previous sections into practice yields a short list of rules, each directly traceable to a specific piece of evidence above rather than to general caution.

State which question is being answered. A best-of-$k$ number (pass@k) and an every-time number (pass$^k$) are both legitimate, and they can diverge sharply on the same system and the same task set ^[3]. Reporting one while a reader assumes the other is a common, avoidable source of overconfidence.

Report an interval, not a point. A single percentage without its standard error invites exactly the kind of overreading Miller’s analysis quantifies, and differences smaller than a few percentage points are frequently statistical noise on the sample sizes agent evaluations typically use ^[4].

Disclose $n$, seeds, and decoding settings. Reproducibility failures traced to unreported seeds, prompts, and decoding parameters are not a hypothetical risk; they were the dominant source of disagreement Hochlehnert and colleagues found when re-checking published reasoning-benchmark results ^[5].

Report reliability as a function of task length, not a single number. METR’s time-horizon framing shows that the same system can be highly reliable on short tasks and near chance on long ones, and a score without a stated task length collapses that entire curve into one misleading point ^[1].

Separate elicitation effort from deployment condition. A number obtained with expert scaffolding is a ceiling estimate; a number obtained in production, under Anthropic’s own description of non-deterministic, path-dependent agent behavior, is a different and generally lower quantity, and conflating the two misrepresents both ^{[9, 6]}.

Grade the trajectory, then the outcome. Both major agent SDK vendors document this as the practical fix for the previous point: fixed, repeated, trajectory-aware evaluation, run on a schedule, is what actually distinguishes engineering progress from a lucky sample ^{[7, 8]}.

Predictions, with the observations that would falsify them

These are forecasts, kept separate from the sourced findings above. Horizon: August 2029.

One. Vendor system cards and agent-benchmark leaderboards will increasingly report an explicit reliability statistic — a pass$^k$-style figure or a confidence interval — alongside a headline pass@1 or pass@k number, because the gap between the two is now well documented rather than speculative. Disconfirmed if major leaderboards in 2029 still report a single unqualified success rate with no repeated-trial or uncertainty disclosure.

Two. Published task-length-versus-reliability curves, in the style METR introduced, will become a standard part of how frontier agent systems are compared, replacing single fixed-length benchmark scores as the primary comparison tool for long-horizon capability claims. Disconfirmed if comparisons in 2029 still rest primarily on fixed-length benchmark suites with no reported length dependence.

Three. The measured gap between best-elicited benchmark performance and typical production performance for the same agent will narrow but not close, because the sources of production variance Anthropic describes — non-determinism and path-dependent drift — are architectural rather than incidental. Disconfirmed if independent audits find production and benchmark reliability converging to within a small margin for a widely deployed agent system.

What to take away

A single successful run is evidence that a task is solvable, not evidence that a system solves it reliably, and the statistics that separate the two claims already exist: pass@k for whether at least one of several attempts succeeds, pass$^k$ for the much stricter question of whether every one of them does, confidence intervals for whether an observed difference is more than noise, time-horizon curves for how reliability changes as a task grows, and a clear accounting of how a number was elicited before it is compared against how a system actually behaves once deployed. None of this is a call for more skepticism in the abstract. It is a call to ask, of any reported agent capability, which of these specific measurements was actually taken — and to treat a claim that skips all of them as an anecdote wearing a percentage sign.