A Practitioner's Map of Agent Evaluation Frameworks

Five rooms, not one leaderboard

Ask which agent benchmark is “the best one” and the question already contains the mistake. SWE-bench hands an agent a broken software repository and checks whether the tests pass afterward. tau-bench hands it a simulated, opinionated customer and checks whether the account ends in the right state, across repeated tries. GAIA hands it an everyday question that takes a competent adult a few minutes and checks one exact answer. WebArena hands it a browser pointed at a frozen, self-hosted website and checks the world state at the end. AgentBench hands it eight unrelated toy worlds and checks a little of everything. None of these is a harder or easier version of another. They are different instruments built by different people to answer different questions, and a single agent can rank first on one and near the bottom on another without anything about the agent having changed.

That would be a minor pedantic point if the five frameworks were not so routinely cited in the same sentence, on the same slide, sometimes on the same axis of the same chart, as though “68% on SWE-bench” and “34% on tau2-bench” were two readings of one underlying quantity called agent capability. They are not readings of one quantity. This article treats each framework on its own terms — what environment it runs in, what its score actually certifies, what its own authors or independent auditors have found wrong with it, and what has grown up to supersede or supplement it — and ends with a way to choose among them that does not require pretending they are commensurable. The goal is a map, not a ranking.

SWE-bench: real GitHub issues, and what “verified” fixed

SWE-bench’s environment is the most literal of the five: a real, cloned software repository, checked out to the exact commit before a real GitHub issue was filed, handed to the agent inside a container with a shell. The agent edits files and submits a patch. Scoring is execution-based rather than textual — the patch is applied and the repository’s own test suite is run, and success requires a defined set of previously failing tests to now pass while a defined set of previously passing tests continues to pass. The original release drew 2,294 such instances from twelve popular Python repositories, and the difficulty at launch was severe: the best model tested, Claude 2, resolved barely two percent of them ^[1].

That severity did not survive contact with scrutiny. An independent audit that manually reviewed successful patches from a SWE-agent-plus-GPT-4 run found that just under a third of the successful patches had their solution effectively supplied inside the issue report or its own comment thread, and a further roughly third passed only because the associated tests were too weak to actually verify correctness — accepting patches that did not really fix the bug. Filtering out both categories of contaminated instance dropped the measured resolution rate from 12.47% to 3.97% for that same agent-model pairing, a collapse to under a third of the original figure ^[2]. The same audit noted that well over ninety percent of the underlying issues predate common LLM training cutoffs, which raises a separate and harder-to-rule-out risk: models may have simply seen the fix during pretraining on public GitHub data, rather than derived it.

The benchmark’s own originators responded to exactly this class of problem. Working with OpenAI, they produced SWE-bench Verified: five hundred instances screened by professional software developers for well-scoped, unambiguous issues and appropriately targeted tests, explicitly intended to supersede both the original set and the smaller SWE-bench Lite ^[3]. Screening for well-posed tasks is not the same as screening for contamination-free ones, and curation has its own cost: an independent evaluator running the same set reports excluding sixteen instances that do not reliably run in its infrastructure, and notes that labs routinely exclude different subsets when they report a number — one frontier model’s headline score came from 477 tasks, an earlier model’s from 489, later ones from the full 500 — so two published “Verified” percentages are not automatically the same measurement. That evaluator also recorded a major harness upgrade in February 2026 substantial enough that it now separates results by harness version rather than plotting one continuous line ^[5].

By early 2026, the benchmark’s co-sponsor delivered a more decisive verdict. OpenAI audited the quarter of the dataset models most often failed and found a majority of those instances had flawed tests — some rejecting genuinely correct submissions, others demanding unrequested extra behaviour. Progress had slowed to a crawl at the top of the range, and the stated concern was that remaining gains increasingly reflected training-time exposure to the benchmark’s public source repositories rather than fresh problem-solving. OpenAI announced it would no longer report SWE-bench Verified as a frontier-coding metric and pointed to a successor instead ^[4]. That successor, SWE-bench Pro, sources much of its 1,865 instances from copyleft and private commercial codebases a frontier model is unlikely to have trained on, and demands larger changes — over a hundred lines across four files on average. The gap this opened is stark: models that clear seventy-plus percent on Verified score in the low twenties on Pro’s public split, and lower still on its held-out and commercial splits ^[6].

The practical lesson is not that SWE-bench “doesn’t work.” It is that a SWE-bench-family number is only informative alongside its variant name, its harness version, and its task subset — and that for any comparison spanning more than a few months, the variant itself may need to change under you.

tau-bench and tau2-bench: reliability, not just success

tau-bench puts an agent on the other side of a different kind of interaction: not a static codebase but a live, simulated conversation with a difficult customer. A separate language model plays the user, holding a private goal the agent does not see, while the agent has access to domain-specific tools and a written policy document it is expected to follow — in a retail domain and an airline domain at launch. Crucially, the benchmark does not grade the conversation’s text. It grades the final state of a backing database against an annotated goal state, so an agent that talks convincingly but leaves an order, refund, or seat unchanged does not pass ^[7].

tau-bench’s more influential contribution may be a metric rather than an environment. Instead of running a task once, it runs each task instance several independent times against the same policy and reports pass^k: the fraction of tasks that succeed on every single one of k attempts, as distinct from the far more forgiving “at least one of k succeeded.” The two numbers move in opposite directions as k grows, and the gap between them turned out to be large — strong models cleared under half of retail tasks on a typical single run, and their all-eight-trials pass rate fell under twenty-five percent in that domain, meaning the same model that looked reasonably capable on a single pass looked unreliable the moment consistency was actually demanded ^[7].

The framework’s authors identified their own environment’s biggest structural gap and then built its successor. Original tau-bench is single-control: only the agent can act on the shared world, while the simulated user can merely describe what it wants. Real support interactions are frequently not like that — a customer troubleshooting a router is also pressing buttons on it. tau2-bench reframes the interaction as a dual-control environment, formalised as a partially observable process in which both the agent and the user simulator can invoke tools against a shared state, and adds a telecom domain built around exactly this kind of joint troubleshooting. The cost of adding that realism was immediate: identical models whose pass@1 ran in the fifty-six to seventy-four percent range on tau-bench’s original single-control domains fell to roughly thirty-four percent once the customer could also act, isolating coordination — not tool use in isolation — as the harder remaining problem ^[8].

For anyone evaluating a customer-facing or technical-support agent, the operative choice is not “tau-bench or something else” but which repeat count k matches the deployment’s actual tolerance for occasional failure, and whether the user in the target deployment is genuinely passive or, like a telecom customer resetting a router, an active second hand on the same controls.

GAIA: a general-assistant test that stayed deliberately small

GAIA asks a different kind of question again: not “can you fix this code” or “can you resolve this ticket” but “can you answer this, the way a competent, resourceful adult would.” Its questions are everyday in register — they require reasoning, some degree of multimodal handling, web browsing, and general tool use — and each has one unambiguous, checkable answer rather than an open-ended deliverable. The explicit philosophy behind that design choice is worth stating plainly: the authors argue that progress toward general assistants should be measured by robustness on ordinary questions an average human handles without difficulty, not by matching narrow specialists on professional exams. At launch, human annotators reached 92% while GPT-4 augmented with plugins reached 15% — a wide gap in the direction that made the benchmark useful ^[9].

GAIA is stratified into three tiers by required effort rather than by subject: Level 1 questions need fewer than five steps and at most one tool, and a human typically answers in about a minute; Level 2 needs roughly five to ten steps across multiple tools and takes a human close to five minutes; Level 3 involves long, open-ended chains of reasoning and tool use that can occupy a human for half an hour or more ^[10]. That stratification matters more than the blended overall score, because a single averaged number can rise sharply just from clearing the easy tier while the genuinely hard tier barely moves — and independent commentary on the benchmark’s current state notes exactly this dynamic, alongside a set of limitations that have nothing to do with model capability: the total item count is modest enough that noise within any one category is real, the questions are English-only, exact-match scoring is unforgiving of borderline-correct phrasing, and because the benchmark is static rather than periodically refreshed, indirect contamination through old web discussion of its specific questions cannot be fully excluded. Even the human baseline is imperfect, since some Level 3 chains are long enough that capable human annotators disagree about them ^[10].

There is also a measurement problem specific to how GAIA is now used. Because the benchmark rewards whatever scaffolding of search, code execution, and multi-step planning an evaluator wraps around a base model, two vendors reporting a GAIA number for what is nominally “the same” underlying model can be reporting the performance of substantially different systems — the harness around the model, not just the model, decides much of the outcome. That is a specific case of a general problem this article returns to below ^[17]. The practical implication for a reader is to treat GAIA as a probe of a whole assistant stack, tool orchestration and all, never as an isolated reading on a base model, and to ask for the Level 3 subset score specifically rather than accepting a blended figure.

WebArena and VisualWebArena: a browser that does not forget

WebArena’s environment is a set of fully functional websites — e-commerce, a Reddit-style forum, a GitLab-style code-hosting site, content management, plus general utility sites — self-hosted so that the benchmark controls and can reset the ground truth rather than depending on the live, ever-changing internet. An agent drives a real browser through the site, and success is judged against the functional outcome recorded in that environment’s final state — did the right item end up in the cart, did the right issue get filed — rather than the text of whatever the agent claims to have done. The gap at launch was enormous: the strongest GPT-4-based agent tested completed only 14.41% of tasks end to end, against a human baseline of 78.24% on the same set ^[12].

The original environment observes pages chiefly as structured text — an accessibility tree and DOM content — which leaves an entire class of failure invisible to it: an agent that cannot actually interpret an image, an icon, or the visual layout of a page can still pass a WebArena task built around text alone. VisualWebArena was built specifically to close that gap. It adds three new environments — a classifieds site, a shopping site, and a Reddit-style forum — comprising 910 tasks constructed so that the correct action genuinely depends on visual content: comparing product photos, reading information conveyed only through layout or image, not just prose. Even capable multimodal agents managed only 16.4% success at launch, and the authors report specific weaknesses in OCR, spatial reasoning, and contextual grounding that the text-only original could never have surfaced, because it never asked a question that required them ^[13].

Both environments share the same limitation by construction: they are frozen, self-hosted snapshots, optimised for reproducibility rather than for the churn of a real, live website that changes underneath an agent day to day. Independent commentary on this style of benchmark makes the resulting caution explicit — a browsing agent that scores strongly against a static sandbox does not necessarily convert at anything close to that rate against a live production site, because both benchmarks are, by design, happy-path completion tests against a world that holds still ^[11]. For a reader evaluating a browsing agent, WebArena is the right instrument when the task is fundamentally textual and DOM-legible; VisualWebArena is the right one when the agent must genuinely see the page; and neither number should be read as a forecast of conversion on a live, unfrozen website.

AgentBench: breadth over depth, and what grew past it

AgentBench takes the opposite design bet from the other four: rather than one deep environment, it packages eight unrelated interactive environments into a single suite — an operating-system shell the agent must drive with real bash commands, a live multi-table SQL database, a partially observable knowledge graph, a turn-based digital card game, a set of lateral-thinking puzzles, a household-task environment adapted from prior simulator work, a web-shopping environment, and a web-browsing task adapted from prior benchmark work — explicitly built as the first attempt to score LLM-as-agent behaviour across a genuinely diverse spread of interaction styles at once, rather than one style at a time. Its headline finding was as much about the field as about any one model: top commercial models showed real, if uneven, aptitude for acting as agents across this spread, open-source models of the time lagged well behind, and the leading failure modes were long-horizon reasoning breakdowns and simple instruction-following slips rather than any single missing skill ^[14].

Breadth was purchased with depth, and that trade is the benchmark’s most consistent documented weakness. Packing eight environments into one suite necessarily means far fewer tasks and shallower interaction chains within any single one of them than a benchmark built around just that domain would offer — SWE-bench’s 2,294 issues or WebArena’s full browser sessions are both far deeper single-domain probes than any one of AgentBench’s eight slices. A recent synthesis pooling findings across nineteen distinct agent benchmarks flags exactly this style of wide, shallow, multi-environment suite as prone to fragmented, hard-to-compare metrics and unusually high reproduction cost, and separately warns that failures compound nonlinearly with task length in ways a short per-environment task count is poorly positioned to reveal ^[18]. The same general critique that inadequate holdout sets let agents take shortcuts and overfit benchmarks applies here with particular force, because a wide suite composed of small slices offers more individually memorisable surface than one deep, harder-to-game domain ^[16].

Where AgentBench’s breadth-first approach has been most clearly succeeded is by benchmarks that chose the opposite trade deliberately. OSWorld was built explicitly to go deeper where prior interactive-agent benchmarks — AgentBench among them — stayed shallow or relied on non-interactive, pre-recorded demonstrations rather than a live, executable environment: 369 real tasks running inside actual Ubuntu, Windows, and macOS virtual machines, each checked by a bespoke execution-based validation script rather than a shared shallow harness spread across many small toy worlds. The gap it exposed was, again, severe at launch — a 72.36% human baseline against a 12.24% best-model result — and squarely aimed at the GUI-grounding and real-desktop operational knowledge that a suite of eight small, largely text-driven slices was never built to probe ^[15]. The sensible reading is not that AgentBench was wrong, but that it is a first-pass breadth screen: useful for noticing which interaction style an agent is weakest at before committing the much larger effort a deep, single-domain benchmark demands.

Why a cross-framework ranking is not a coherent thing to build

Two separate arguments, from two different directions, both land on the same conclusion: a single ranking spanning SWE-bench, tau-bench, GAIA, WebArena, and AgentBench is not merely unhelpful, it is not a well-formed comparison to begin with.

The first argument is about what actually produces a reported score. Recent work formalises a claim that practitioners have suspected for a while and terms it the binding-constraint thesis: for long-horizon agent tasks, the execution harness — how context is constructed, how tool calls are issued and retried, how outputs are verified — frequently matters more to the final number than which underlying model sits inside it. The paper reports harness-induced variance exceeding model-induced variance in its tests, and, more sharply, cases of outright model-ranking reversal: which of two models looks better can flip depending purely on the scaffolding wrapped around them, with nothing about either model having changed ^[17]. If two published numbers on the very same benchmark, with the very same models, are not safely comparable without knowing the harness behind each one, then two numbers from two different benchmarks — with different environments, different scoring rules, and different harnesses by construction — are comparable in name only.

The second argument is about how much any one number can actually tell you, independent of harness effects, purely from how many tasks produced it. Treat a benchmark’s headline pass rate as an estimate $\hat p$ of a task-set success probability, drawn from $n$ roughly independent trials. The familiar normal approximation to a binomial confidence interval,

gives a rough sense of how much noise sits under a given $n$, even granting the generous and almost certainly false assumption that every task in the set is an equally difficult coin flip. At the widest plausible spread, $\hat p = 0.5$, a 95% interval ($z_{\alpha/2} \approx 1.96$) works out to roughly $\pm 13.9$ points on a fifty-task set — close to the size of tau-bench’s original airline domain or GAIA’s seventy-five-question Level 3 tier — versus roughly $\pm 4.4$ points on SWE-bench Verified’s five hundred tasks, or $\pm 5.1$ points on OSWorld’s 369. A three- or four-point difference between two systems on a fifty- or seventy-five-task slice is, on this generous accounting, close to noise; the same gap on a five-hundred-task set is not. Real task-to-task difficulty variation only widens these intervals further, so treat the numbers above as a floor on the uncertainty, not the whole of it. The lesson is not to distrust small benchmarks — tau-bench’s airline domain and GAIA’s Level 3 tier are deliberately small because hard, well-verified tasks in those domains are expensive to build — but to stop reading a two-point movement on either of them as a finding.

Put the two arguments together and the shape of the problem is clear: even before asking whether SWE-bench and GAIA measure the same thing, which they plainly do not, the numbers from a single one of these frameworks depend on which harness produced them and how many tasks they were averaged over. A ranking built by stacking headline percentages from five incommensurable environments inherits every one of these problems at once, which is why this article does not attempt to build one, and treats any that does with active suspicion.

A compass, not a leaderboard

The more useful question is not “which framework wins” but “which framework is built to test the thing I am about to ship.” If an agent edits real code inside a real repository, SWE-bench Verified is defensible only as a comparability anchor against older published numbers, and even then only alongside its harness version and excluded-instance count; SWE-bench Pro’s held-out and commercial splits are the better source of live signal precisely because a frontier model has not already seen them. If an agent negotiates with a customer who can also act — resetting a router, changing their own order — tau2-bench’s dual-control telecom domain is closer to the deployment than tau-bench’s original single-control retail and airline domains, and the number that matters is pass^k at whatever repeat count the deployment actually demands, not a single pass@1. If an agent is meant to be a general-purpose assistant answering everyday questions with tools and browsing, GAIA is the right shape of test, but the Level 3 subset carries the information a blended score hides, and a vendor-reported number is only interpretable with its harness disclosed. If an agent drives a browser, WebArena is sufficient when the task is legible from page text and structure alone; VisualWebArena is necessary when it is not; and both numbers describe a frozen sandbox, not a live site that keeps changing under the agent’s feet. And if the job is a first breadth pass across many unfamiliar interaction styles before deciding where to invest in a deeper benchmark, AgentBench’s eight small worlds are a reasonable place to start — provided the next step is graduating to a deep, single-domain benchmark built for whichever environment turned out to matter, in the way OSWorld already did for desktop computer use.

None of this produces a single number to put on a slide. That is the point: the frameworks were not built to produce one, and forcing them to is where the misrepresentation this article is arguing against actually happens.

Predictions, and what would falsify them

These are forecasts, kept separate from the sourced material above. Horizon: 12 August 2028.

One. More agent benchmarks will follow SWE-bench Verified’s path into SWE-bench Pro — rotating in held-out or private task splits specifically to resist training-data contamination — rather than relying indefinitely on a single static public set. Disconfirmed if the major coding and agent benchmarks still cited in 2028 remain fully public and static, with no held-out or periodically refreshed component.

Two. Harness disclosure — the scaffolding, retry policy, and context-management choices wrapped around a model — will become a standard, structured field on agent leaderboards rather than an optional footnote. Disconfirmed if 2028’s most-cited agent leaderboards still publish headline scores with no comparable, standardised harness disclosure attached.

Three. Reliability-at-repeat-count metrics in the style of tau-bench’s pass^k will spread into coding and browsing benchmarks, not stay confined to conversational ones, as agents move from single-shot demonstrations toward always-on deployment. Disconfirmed if 2028’s leading coding and browsing leaderboards still report only single-attempt success rates with no repeat-trial reliability figure alongside them.

Four. No single benchmark will consolidate coding, browsing, conversational, and generalist-assistant evaluation into one dominant “agent capability” instrument; the field keeps adding narrow, environment-matched benchmarks rather than converging on a general one. Disconfirmed if by 2028 one benchmark suite is the dominant citation across all four of those evaluation needs simultaneously.

None of these predictions requires a capability breakthrough. They follow from what is already visible in the last two years of this landscape: contamination forcing benchmark churn, harness effects large enough to reverse rankings, and reliability metrics proving their worth in the one domain that adopted them first.

What to take away

SWE-bench tests whether an agent can fix a real, broken codebase and have the tests agree. tau-bench and tau2-bench test whether an agent can hold a policy-constrained conversation with an opinionated, sometimes actively participating customer and leave the account in the right state, across repeated tries. GAIA tests whether an agent is a robust everyday generalist, not a narrow specialist. WebArena and VisualWebArena test whether an agent can actually operate a website — textually, and then visually — without being taken at its word. AgentBench tests a wide, shallow spread of interaction styles as a first screen before a deeper benchmark is warranted. Each of these is a defensible, well-documented instrument for the question it was built to answer, and each has a documented limitation — contamination, single-control blindness, static-corpus staleness, frozen-sandbox unreality, shallow per-environment depth — that its own authors or independent auditors have already put in writing.

What none of them is, is a unit of the same thing as the other four. The discipline this survey argues for is not more benchmarks or bigger ones. It is naming the evaluation question before reaching for an instrument, reading a benchmark’s own documented limitations before trusting its number, and refusing the leaderboard that quietly assumes five different rooms were always measuring the same walk.