A pass rate is not a capability
An AI agent is a language model wired into a loop: it observes some state, proposes an action, receives the result of that action, and decides what to do next, usually with some form of memory carried across the steps and sometimes with other agents in the loop alongside it. The architecture that made this loop practical at scale interleaves reasoning and acting explicitly, having the model narrate a short plan, take a tool action, read back the result, and revise — a pattern formalised under the name ReAct and shown to reduce hallucination on knowledge tasks while outperforming action-only baselines on interactive decision-making benchmarks [8]. Almost everything built since sits on some variant of that loop: a planner that decomposes a goal, a memory store that persists state across steps, a tool-use layer that executes and reads back, and, increasingly, a coordination layer that splits work across more than one agent.
The question this article works through is not whether such systems work, in some general sense, but what has actually been measured about how well they work, under what conditions, and how much of that measurement can be trusted at face value. A single benchmark score compresses at least four separate choices into one number: which task instances were sampled, which scaffold and tool access the agent was given, how many trials were run and how they were aggregated, and how strictly the grading logic checked the result. Two labs reporting “82%” on the same benchmark name are not necessarily reporting the same thing. That compression is the subject here, not a claim that agents cannot do useful work — plainly, on the evidence below, some of them can, part of the time, on specific classes of task.
What the benchmarks actually test, and what they don’t
Agent benchmarks split fairly cleanly along the architectural components they exercise, and treating them as interchangeable “AI agent” scores is the first and most common error in reporting on this field.
SWE-bench evaluates tool use against real software-engineering work: an agent is given a genuine GitHub issue and the corresponding repository, and is graded on whether its patch makes the associated hidden tests pass, drawn from 2,294 instances across twelve popular Python repositories [2]. GAIA evaluates a different slice — general-assistant tasks that require web browsing, multi-modal handling, and tool composition on questions designed to be conceptually simple for a human but to require several correct steps chained together without error [4]. WebArena measures long-horizon interactive web navigation across fully functional e-commerce, forum, software-development, and content-management sites, scoring whether the end state left behind matches the task’s intent [5]. OSWorld extends the same idea to an entire desktop: agents operate real Ubuntu, Windows, and macOS environments across genuine applications and file systems, rather than a sandboxed subset of one [10]. tau-bench measures something narrower and arguably closer to deployed practice: multi-turn conversations between a simulated user and a tool-calling agent that must also follow a written domain policy, scored by comparing the resulting database state to an annotated goal state [6]. AgentBench takes the broadest sweep, testing reasoning and decision-making across eight distinct environments including operating systems, databases, and games, specifically to compare commercial and open-source models as agents rather than as raw text generators [7].
None of these measures “agentic capability” as a single latent quantity. Each is a specific claim about a specific slice: correctly patching real code, correctly answering a browsing-and-tool-use question, correctly finishing a web task, correctly finishing a desktop task, correctly following a policy across a multi-turn dialogue, or correctly acting across a battery of unrelated environments. A model that scores well on one says comparatively little, on its own, about the others.
The headline numbers, and what produced them
Read in publication order, the reported numbers describe a field moving from barely functional to strikingly capable on paper, and the movement is real but its size is easy to overstate by comparing across incompatible conditions.
At its original publication, the best-performing model on SWE-bench — Claude 2 — resolved just 1.96% of the 2,294 instances [2]. GAIA reported a wide human/machine gap at its own publication: human annotators reached 92% accuracy on its question set, while GPT-4 augmented with plugins reached only 15% [4]. WebArena reported that its best GPT-4-based agent achieved a 14.41% end-to-end task success rate, against a human success rate of 78.24% on the same tasks [5]. OSWorld reported an even starker split for real computer-use tasks: humans completed more than 72.36% of tasks successfully, against 12.24% for the best model evaluated [10]. tau-bench, evaluating tool-and-policy following rather than raw task completion, reported that even a strong function-calling agent built on GPT-4o succeeded on fewer than half its tasks [6].
Two years and several model generations later, a human-filtered 500-instance subset of SWE-bench called SWE-bench Verified was constructed specifically to fix under-specified problem statements and overly strict test patches present in the original set [3]. Against that harder-to-game version of the benchmark, Anthropic’s own announcement for Claude Sonnet 4.5 states a score of 77.2%, averaged over ten trials, using no test-time compute and a 200,000-token thinking budget on the full 500-problem set; the same announcement states that with parallel test-time compute the score rises to 82.0% [12].
That is a vendor’s own reporting of its own model, not an independently reproduced figure, and it should be read as exactly that — a claim, made by the party with the largest interest in a favourable number, about a specific configuration of a specific model. It is also not a clean two-year progress line from “1.96%” to “77–82%”: the benchmark changed from the original SWE-bench to the harder-audited Verified subset, the trial protocol changed from a single run to ten trials averaged, and the reporting party changed from an independent academic paper to a company’s product announcement. Stacking these four numbers on one implied curve of “agent progress” would misrepresent every one of them.
Reliability is a separate axis from capability
A pass rate answers whether a system can do a task at all, under the conditions of one attempt. It does not answer whether the system does the same task the same way twice, which is a materially different question for anyone deciding whether to rely on it. tau-bench is one of the few widely cited agent benchmarks built explicitly to separate the two, using a metric its authors call pass^k: the fraction of tasks on which all k independent trials succeed, rather than the more familiar pass@k, which asks whether at least one of k trials succeeds [6].
The distinction matters because the two metrics behave very differently even under the most favourable possible assumption — that trials are statistically independent. Write
The first quantity rises quickly toward one as
tau-bench’s reported results are worse than even that pessimistic independence assumption predicts: the authors report pass^8 below 25% in the retail domain, a figure that is lower than
Measuring duration instead of a pass rate
A different research program has measured agents along an entirely different axis: not whether a fixed set of tasks is passed, but how long a task — measured by the time a skilled human needs to complete it — a model can be trusted to carry to a correct conclusion. METR defines a model’s 50%-task-completion time horizon as the human-baseline task duration at which the model succeeds on first attempt half the time, and reports that current frontier models, including Claude 3.7 Sonnet at the time of the study, had a 50% time horizon of around 50 minutes [1].
The paper’s second finding is the one usually quoted, and it is worth stating precisely: measured across model generations back to 2019, this time horizon has been doubling approximately every seven months [1]. That is a documented empirical regularity fitted to a specific history of models on a specific suite of software and research tasks with human-timed baselines — a fact about a measured trend, not a law of nature and not a guarantee about any other class of task. The paper’s authors go on to note that, if the trend continues, it would imply many current month-long professional software tasks becoming automatable within roughly five years. That extrapolation is a forecast built on an assumption that a seven-month doubling rate persists outside the regime in which it was measured, and it should be labelled as such rather than folded into the same sentence as the measured 50-minute figure as if both carried equal evidential weight. A trend line fitted to six years of data is a real finding; whether it continues at the same rate for another five is a separate claim resting on the continuation of the same rate of algorithmic and infrastructure progress that produced the historical trend, and on the future task suite resembling the one already measured.
Where the measurements themselves have been shown to be wrong
The benchmarks above are not neutral rulers whose only flaw is being narrow. A direct audit of several widely used agentic benchmarks found that flaws in task setup and reward design produce measurable distortion in the reported scores themselves, independent of how capable the underlying model actually is [11]. The authors report that agents can be scored as passing SWE-bench without having generated a correct patch at all in 7.7% of SWE-bench-Lite tasks and 5.2% of SWE-bench-Verified tasks, because the associated test cases are insufficiently specific to the intended fix; that tau-bench’s grading logic counts an empty agent response as a success in some conditions; and that in OSWorld’s Chrome-browser task section, 13 of 46 problems had become broken since the benchmark’s construction because the underlying websites had changed layout, URLs, or functionality [11]. Synthesising across the benchmarks they examined, the authors state that such issues can produce under- or over-estimation of an agent’s true performance by up to 100% in relative terms — meaning a reported score can be double, or half, what a more carefully audited version of the same benchmark would report for the identical model.
The paper’s proposed remedy, a synthesised Agentic Benchmark Checklist, is offered as a partial fix rather than a claim of having solved the problem: applied to CVE-Bench, a benchmark the authors describe as having a particularly complex evaluation design, the checklist reduced measured performance overestimation by 33% [11]. A one-third reduction from one audit is meaningful evidence that grading-logic errors were present and substantial; it is not evidence that the remaining two-thirds of whatever gap existed has been eliminated, on that benchmark or on any other one the checklist has not yet been applied to. The finding that matters most for reading any single benchmark score is structural: the instrument that produces the number has, in specific documented cases, itself required correction before the number could be trusted, and most public leaderboard entries do not report whether that correction work has been done for the score being displayed.
What is not yet measured at all
Multi-agent coordination — several agents dividing labour, handing off partial work, or negotiating over a shared resource — is the architectural component named in this article’s framing that has the thinnest standardised quantitative measurement of any discussed here. The foundational systems paper in this space, Generative Agents, gave a group of 25 language-model-driven agents a persistent memory architecture built from three parts: a natural-language experience stream recording what each agent observed and did, a reflection process that periodically synthesises those raw experiences into higher-level insights, and a retrieval mechanism that surfaces relevant memories to plan future behaviour [9]. Populated into a sandbox town, the agents produced coherent emergent social behaviour without being scripted to — coordinating to organise and attend a Valentine’s Day party is the paper’s own illustrative example [9].
That is a genuine and influential architectural result: it demonstrates that a specific memory design causes coordinated multi-agent behaviour to emerge, rather than requiring it to be hand-authored. It is not, however, a benchmark in the sense that SWE-bench or GAIA are benchmarks. The paper’s central evaluation is a human-judged assessment of how believable the agents’ behaviour appeared, conducted in one sandbox environment, rather than a standardised pass rate on a fixed, reusable task suite that other multi-agent systems can be run against for direct comparison. No benchmark discussed elsewhere in this article — SWE-bench, GAIA, WebArena, OSWorld, tau-bench, or AgentBench — targets multi-agent coordination specifically as its measured quantity; each evaluates a single agent’s tool use, planning, or policy-following. This is a genuine evidence gap rather than a claim that coordination cannot be measured in principle: it means that claims about how well multi-agent systems coordinate currently rest on architectural demonstrations and qualitative evaluation far more than on the kind of repeatable, audited pass-rate evidence available for single-agent tool use.
Reading a benchmark claim
Five practices follow directly from the evidence above, and they apply whether the number in front of you comes from an academic paper or a vendor announcement.
Ask which harness produced the number, not only which model. The same underlying weights can report meaningfully different scores depending on tool access, thinking budget, and whether test-time compute is used in parallel — Anthropic’s own 77.2% and 82.0% figures for one model on one benchmark are a single, self-reported illustration of exactly this swing [12].
Ask how many trials were run and how variance was reported. A single-trial pass rate cannot distinguish a reliable capability from a lucky sample, particularly once correlated failure across repeated trials has been documented directly, as tau-bench’s pass^k results show [6].
Ask whether the grading logic has been audited. Documented cases of scores passing without correct output, and of broken task environments going undetected, show that a benchmark’s own instrument can be miscalibrated in ways that shift the headline number substantially [11].
Never build a cross-vendor ranking from mismatched benchmarks, scaffolds, or dates. A SWE-bench Verified score obtained with one lab’s custom scaffold on one date is not comparable, without controlling for all three, to a different lab’s score on a different benchmark variant at a different date.
Separate a documented trend from a forecast built on it. METR’s measured seven-month doubling in task time horizon is an empirical finding about the past; the claim that it implies month-long task automation within five years is a conditional projection, and the two should never be quoted as though they carry the same evidential status [1].
Predictions, with the observations that would falsify them
These are forecasts, clearly separated from the sourced analysis above. Horizon: 22 August 2028.
One. Vendor system cards will increasingly disclose trial count and scaffold configuration alongside headline agent-benchmark scores, because a bare percentage is already indefensible once a single model’s own reported score can move by five points depending on compute configuration alone. Disconfirmed if major 2028 system cards still report single agent-benchmark figures with no disclosure of trial count or scaffold.
Two. Repeat-trial reliability metrics in the style of tau-bench’s pass^k will spread from tool-and-policy benchmarks into coding and computer-use benchmarks, because a single-trial pass rate already understates the practical gap between occasional and dependable success. Disconfirmed if the leading coding and computer-use leaderboards in 2028 still report only single-trial pass rates with no repeated-trial reliability figure published alongside them.
Three. At least one major benchmark maintainer will require an audit against a published checklist, in the style of the Agentic Benchmark Checklist, before accepting new leaderboard submissions, following documented cases of grading-logic errors producing up to 100% relative overestimation. Disconfirmed if no major agent-benchmark leaderboard has adopted or required any such audit by the horizon date.
Four. The measured doubling time for software-task time horizons will lengthen measurably beyond seven months before it shortens further, because the current measurement suite is drawn from a narrow class of well-specified software tasks that labs can specifically target. Disconfirmed if a comparable measurement on the same or an extended task suite still shows a doubling time at or below seven months through the horizon date.
None of these predictions requires a capability breakthrough. They follow from a pattern already visible in the evidence above: every layer of the measurement stack — harness, trial count, grading logic, and the gap between a fitted trend and its extrapolation — has already produced a documented distortion in at least one widely cited agent benchmark.
What to take away
A published agent-benchmark score compresses five separate choices into one number: which instances were sampled, which architectural component is actually being exercised, what harness and compute budget the agent ran under, how many trials were averaged and how, and whether the grading logic itself has been checked against a known-correct standard. The evidence surveyed here shows measurable distortion at every one of those layers in at least one major, widely cited benchmark — not as a reason to distrust every number that follows, but as the specific list of questions to answer before treating any one of them as a settled fact about a model rather than a reading taken under stated, checkable conditions.
