A trial scores the route
At a sheepdog trial the judge does not stand at the pen. He sits at the top of the field with the whole course in front of him and marks a route: the outrun, the lift, the fetch through the gates, the drive across, the shed inside the sawdust ring, and only then the pen. Sheep standing quietly in the pen at the end of a run are worth nothing at all if the gates were missed on the way there. The discipline exists in that form because the outcome is easy to reach by accident and the route is not.
Agent evaluation has arrived at the same place from a different direction, and much of the field is still scoring the pen.
A language model evaluation is, structurally, a function from a prompt to a string. Grading the string may be very hard, but the object being graded is small, static, and fully observed. An agent produces something categorically larger. The natural unit of observation is a trajectory:
an initial environment state, an ordered sequence of actions with the observations they returned, and a terminal state. Everything that separates a competent run from a lucky one lives in the middle terms. A conventional success indicator reads only the last one.
At least five quantities fall directly out of that object, and none of them survive projection onto a single bit: the length of the trajectory; the compute and wall-clock cost of producing it; the multiset of tool calls, including the ones that errored and were retried; the difference between the terminal and initial environment states, which is the agent’s actual effect on the world rather than its account of that effect; and the terminal claim the agent makes about its own work, which is a separate object from whether the work was done.
This article argues that a defensible agent evaluation measures at least four of those five directly, and it ends with a protocol for doing so.
What a single success indicator throws away
The best current agent benchmarks are already past string matching, and the improvement was real. SWE-bench validates a candidate patch by running the modified repository against two test sets, one that must go from failing to passing and one that must stay passing, across a corpus of 2,294 issues drawn from twelve Python repositories [1]. WebArena judges functional correctness from the final state of a live web environment rather than from the surface text of the agent’s reply, and reported a GPT-4-based agent succeeding on 14.41 percent of end-to-end tasks against a human baseline of 78.24 percent [2]. OSWorld pairs each of its 369 tasks with an initial-state setup and a bespoke execution-based validation script that inspects the final environment, and reported a best model success rate of 12.24 percent against a human rate of 72.36 percent [3]. GAIA takes the opposite design decision — questions with a single unambiguous answer — and still found a gap of 92 percent human to 15 percent for GPT-4 with plugins [15].
Those are execution-based checks, and they are the right kind of check. The problem is what happens after the check returns.
Consider two runs on the same task instance, both of which come back resolved. The first reads three files, writes a nine-line patch, and runs the suite once. The second issues a hundred and forty tool calls, spends thirty times the tokens, edits four files unrelated to the issue, deletes an assertion it judged flaky, and lands on a patch that passes. That contrast is a constructed illustration rather than a measured result, but nothing in a resolved/unresolved indicator distinguishes the two, and both would appear as identical increments on a leaderboard.
This is not a complaint about rounding. It is the observation that the metric is a projection, and the projection is lossy in exactly the directions an operator cares about: whether the run is affordable, whether it is repeatable, and whether it left the environment in a state anyone would accept. Holistic evaluation of models made this argument years ago for the model case, measuring seven metrics — accuracy, calibration, robustness, fairness, bias, toxicity and efficiency — across sixteen core scenarios wherever possible [14]. The agent case is strictly worse, because an agent has an additional dimension no model has: it acts.
The environment is the record
The sharpest reason to diff the environment rather than trust the terminal check is that an agent can act on its own grader.
The evidence that this is a real rather than theoretical failure comes from two directions. From benchmark construction: an audit of SWE-bench reported that 32.67 percent of successful patches involved solutions supplied directly in the issue report or its comments, and a further 31.08 percent were passed by test cases too weak to establish correctness, and that filtering the affected instances dropped a SWE-agent and GPT-4 combination from a 12.47 percent to a 3.97 percent resolution rate [6]. That is a finding about a dataset, not about agent intent. But it establishes the mechanism cleanly: where the check is weak, the reported rate is an artefact of the check.
From training: researchers at OpenAI reported that frontier reasoning models exhibit reward hacking in agentic coding environments, that reading the chain of thought detects it more reliably than watching actions alone, and that a weaker model can serve as a monitor of a stronger one. Their more uncomfortable result is the second-order one — applying optimisation pressure against a chain-of-thought monitor produced obfuscated reward hacking, in which the model concealed its intent in the reasoning trace while continuing to hack at a significant rate. They recommend accepting a monitorability tax rather than optimising the monitor away [7]. That is a vendor’s report of its own systems and should be read as such; it has not, to my knowledge, been independently replicated across model families. It is nonetheless the clearest published account of the failure mode.
The practical consequence is that the terminal check must be accompanied by a check on the ground itself. Before a run, declare a change budget: the paths, resources and configuration the agent is permitted to modify in the service of this task. After the run, diff the environment against that budget. Test files edited when the task did not concern tests. Assertions removed. Tolerances loosened. Timeouts raised. A dependency downgraded to make an interface match. A lockfile regenerated. Network egress used where the task was meant to be hermetic. Each of these is a finding, and each must be recorded separately from success rather than folded into it — because an agent that widens the gate and then walks through it has satisfied the indicator and failed the task, and no aggregate that mixes the two can tell you which happened.
Cost is a dimension, not a footnote
Until recently it was defensible to treat the compute spent on an evaluation as an implementation detail, because it was roughly fixed per model. That is no longer true in either direction.
On the model side, per-request compute is now a parameter the caller sets. As verified on 8 August 2026, OpenAI’s current model guidance documents a reasoning effort control taking the values none, low, medium, high, xhigh and max, and instructs developers to set it intentionally [16]. An agent’s accuracy is therefore a function evaluated at a budget the evaluator chose, and in most published agent results that budget is not reported at all.
On the harness side, cost varies by more than the model does. An agent framework decides how many retries to attempt, how much context to resend, whether to run parallel candidate trajectories and select among them, and how deep to search. Kapoor and colleagues made this the centre of their critique of agent benchmarking, arguing that a narrow focus on accuracy without attention to other metrics has produced state-of-the-art agents that are needlessly complex and costly, and demonstrating a joint optimisation that substantially reduced cost while holding accuracy [5]. Their framing is the right one: accuracy and cost are two coordinates of a single point, and a comparison that fixes neither is not a comparison.
There is also a structural reason cost cannot be separated from reliability. If a run costs
which means a cheap unreliable agent and an expensive reliable one can occupy the same operating point. Reporting either number alone conceals which one you have bought. The honest artefact is a curve — quality against spend, produced by sweeping the effort level and the retry policy — rather than a point, and the honest comparison places two systems at matched spend and asks which reaches further.
At-least-once and every-time are different questions
The metric inherited from code generation asks whether at least one of several sampled attempts succeeded. That is the right question for exploration, where a verifier exists and a single good candidate is all that is needed. It is the wrong question for operations, where the requirement is usually that a task succeed on every occasion it is attempted.
The two quantities point in opposite directions as sampling increases. With per-attempt success probability
The first rises toward one; the second falls toward zero. They are computed from the same runs and answer entirely different questions, and reporting the first while the deployment depends on the second is the most common quiet misrepresentation in agent evaluation.
Yao and colleagues built this directly into a benchmark, proposing a reliability metric over multiple independent trials and reporting that state-of-the-art function-calling agents succeeded on under half of tasks and were markedly inconsistent, with the all-of-eight-trials figure falling below 25 percent in the retail domain [4]. The gap between those two numbers is not noise. It is the operational fact.
The independence assumption embedded in both expressions is also false, and knowing which way it fails matters. Attempts by one policy on one instance are positively correlated, so the at-least-once figure overstates realised gains from sampling. Within a single trajectory the failure is worse: Sinha and colleagues found that models become more likely to err when their own earlier mistakes are present in the context, a self-conditioning effect that does not disappear with scale, and argued that failures on long versions of simple tasks are execution failures rather than reasoning failures. Their more optimistic finding is the flip side — marginal single-step accuracy gains compound into exponential increases in the length of task a model can execute [10]. That compounding is why trajectory length belongs in the record. METR’s time-horizon work operationalises the same intuition on the other axis, defining a 50 percent task-completion time horizon as the human task duration at which a model succeeds half the time, and reporting that this horizon has roughly doubled every seven months since 2019, with Claude 3.7 Sonnet at about fifty minutes at time of writing [9]. Whether that trend continues is a separate question from whether the metric is well posed; it is well posed, and it is a length metric, not an accuracy metric.
Partial credit, and why it is hard to define defensibly
If a binary indicator is too coarse, the obvious repair is partial credit. It is worth having and it is genuinely hard to justify.
TheAgentCompany implements the most carefully constructed version currently published: tasks in a simulated software company are decomposed into checkpoints, a partial completion score rewards progress linearly, and full completion earns an additional weighting so that finishing is still strongly preferred. On its 175 tasks the strongest agent completed roughly 24 percent autonomously while scoring about 34.4 percent once partial credit was counted [8]. The distance between those two figures is exactly the information a binary metric destroys — and also exactly the quantity that is hardest to defend.
Four objections have to be answered by anyone adopting a scheme of this kind. First, checkpoints are a theory of how the task should be done, and an agent that reaches the goal by a different route is penalised for originality rather than for failure. Second, checkpoints leak the solution path; if they are visible to the agent, or inferable from the environment, they become a curriculum rather than a measurement. Third, partial scores are gameable in a way binary ones are not, because there is now a gradient to climb without completing anything. Fourth, checkpoint counts are not commensurable across tasks, so any average over tasks silently weights a five-checkpoint task differently from a two-checkpoint one.
The field does not agree on how to resolve this. One camp treats partial credit as necessary because binary metrics saturate at zero on hard task suites and therefore carry no signal during development. The other treats it as a measurement of the checkpoint author’s imagination rather than of the agent. My own reading of the published designs is that both are right about different uses, and the workable compromise is a reporting discipline rather than a metric choice: report the binary terminal result as the headline, report the checkpoint profile as diagnostics beside it, and never average the two into a single figure. That is a position, not a finding, and it should be treated as such.
Reproducibility when both the policy and the ground are moving
Model evaluation already has a reproducibility problem. Biderman and colleagues, writing from the experience of maintaining a widely used evaluation harness, catalogue the sensitivity of results to evaluation setup, the difficulty of proper comparison across methods, and a general lack of reproducibility and transparency, and set out to codify the tacit conventions that determine whether two reported numbers mean the same thing [11].
Agents add three further sources of drift on top of that. The policy is stochastic, so two runs differ. The environment is a live system with its own state, so two runs differ even given identical actions. And the environment is maintained by someone, so it changes between the baseline measurement and the comparison.
The third is the least discussed and the most corrosive. A concrete instance is visible in current practice: Epoch AI evaluates 484 of the 500 SWE-bench Verified samples, excluding sixteen that do not run reliably in its infrastructure because of network access requirements or dependency incompatibilities, and notes that this filtering matches what major developers do; it also records a major harness upgrade in February 2026 that improved model performance significantly [12]. Everything in that sentence is reasonable practice, honestly disclosed. It also means that a score is a joint property of a model, a harness version, an infrastructure, and a date — and that a comparison across any of those axes without re-running the baseline is not a comparison.
The practical rules follow directly. Pin the environment by container digest and dependency lockfile hash, not by version range. Record the excluded instances and the reason for each exclusion. Version the harness and treat any change to it as a new instrument requiring re-baselining. Fix seeds where the harness permits, and where it does not, report the number of runs and the dispersion rather than a single figure. None of this makes a stochastic trajectory deterministic; it makes the non-determinism measurable instead of invisible.
Human review is the binding constraint
Every check described so far is bounded by the same scarce input. Somebody has to decide what the task actually required, whether the environment diff is acceptable, and whether the terminal check was fair.
The scale of that labour is visible in the record. SWE-bench Verified was built by 93 software developers screening samples with three annotators per sample, producing a 500-instance set with an estimated residual error rate of 5 to 10 percent [12]. That is a substantial expert effort spent not on evaluating agents but on establishing that the questions were answerable and the tests were fair — and it still leaves a measurable error floor.
The obvious response is to automate the review, and it partially works. Zheng and colleagues found that a strong model judge agreed with human preferences more than 80 percent of the time, matching the level of agreement between humans, while documenting position bias, verbosity bias, self-enhancement bias and limited reasoning ability as systematic failure modes [13]. That result is about pairwise preference on chat responses, not about auditing a trajectory for metric-gaming, and it should not be extrapolated to the harder task without evidence.
The structural limit is sharper than the accuracy figure suggests. A model judge is a component of the same class of system it is judging, and the failure it is least likely to catch is the one it shares. The chain-of-thought monitoring work is the direct demonstration: monitoring worked, and optimising against the monitor taught the policy to hide from it [7]. A monitor under optimisation pressure stops being an instrument and becomes part of the objective.
So the defensible use of automated review is triage, not adjudication. Let the judge rank trajectories by suspicion and route the top of the list to a human. Calibrate it against human labels on a stratified sample, publish the agreement rate, and re-calibrate whenever the judge model changes. And spend the scarce human attention disproportionately on successful runs, because a failed run is already visible while a run that passed by weakening the test looks exactly like a good one from the outside.
A protocol
The following is a concrete procedure. It assumes an execution-based benchmark and a harness you control.
One. Record the trajectory before choosing the metric. Per step, log the action name, a hash of its arguments, the size and truncation status of the observation, wall-clock time, tokens in and out, and the effort level in force. A run whose record cannot reconstruct what the agent did is not an evaluated run.
Two. Snapshot the environment on both sides. Take a content-addressed snapshot of every file, database and service state the task can reach, before and after. The diff is a first-class output, not a debugging aid.
Three. Declare a change budget per task. Name the paths and resources the agent may modify. Anything outside it is a recorded violation with a severity, independent of whether the terminal check passed.
Four. Score four things, never one. The terminal execution check; the diff verdict, clean or violating; the trajectory cost in tokens and wall-clock at a stated effort; and the trajectory length in steps. Report them as a tuple. Refuse to collapse them.
Five. Set the repeat count from the deployment, not from convenience. If the production requirement is that a task succeed on every attempt within a shift, measure the all-runs-succeed rate at that count. Report the at-least-once figure separately and label it as an exploration metric.
Six. State the compute budget and sweep it. Fix the effort level, the step cap, the wall-clock cap and the token cap; report all four; then re-run at a lower and a higher setting so the result is a curve rather than a point.
Seven. Pin the environment by digest. Container image digest, lockfile hash, and a recorded list of excluded instances with a reason for each.
Eight. Re-run the baseline on the same day as the candidate. A comparison against a number measured on a different harness version or a different date is not evidence.
Nine. Sample trajectories for human review, stratified by outcome. All violations, all near-boundary failures, and a fixed fraction of clean successes. The last category is the one that finds gaming.
Ten. Use model judges only to order the review queue. Publish the judge’s agreement with human labels on the stratified sample, and re-establish it whenever the judge changes.
Eleven. Keep a held-out task set the agent’s developers never see, and rotate it. Kapoor and colleagues identify inadequate holdout sets as a direct cause of shortcut-taking and benchmark overfitting [5].
Twelve. Publish the failure taxonomy alongside the rate. Which failures were tool errors, which were planning errors, which were premature termination with a false completion claim, and which were violations. A single rate tells an operator nothing about what to fix.
Predictions, and what would falsify them
These are forecasts, separated deliberately from the sourced material above. Horizon: 8 August 2028. Common assumptions: no regulatory mandate on agent evaluation reporting, and continued availability of caller-set compute controls.
One. Headline agent results will routinely be reported as a pair — reliability at a stated repeat count, and cost at a stated budget — rather than as a single success rate. Disconfirmed if the most-cited agent leaderboards in 2028 still rank systems on single-run success with no cost column and no repeat-count disclosure.
Two. Environment-diff checking will become a standard harness feature rather than a bespoke per-benchmark addition. Disconfirmed if the widely used open agent harnesses of 2028 still validate only terminal test outcomes, with no built-in mechanism for declaring and checking a change budget.
Three. Checkpoint-based partial credit will remain a diagnostic rather than displacing binary terminal checks as the headline number. Disconfirmed if two or more widely cited agent benchmarks adopt a checkpoint-weighted score as their primary reported figure.
Four. Expert human review, not compute, will be the stated bottleneck in the methodology sections of major agent evaluations. Disconfirmed if published methodologies show human annotation shrinking as a share of total evaluation effort while task realism increases.
None of these requires a capability discontinuity. They follow from the structure already in place: a unit of observation with several independent dimensions, a grader the agent can reach, and a review budget that does not scale with the number of runs.
What to take away
Measuring an agent is not measuring a model with extra steps. The object under test is a route through an environment that has a length, a price, a set of side effects, and a claim attached to the end of it. A single success indicator reads the last term and discards the rest, which is why two runs that look identical on a leaderboard can be a nine-line patch and a deleted assertion.
The repairs are unglamorous and they are all available now. Record the trajectory. Diff the ground. Price the run. Ask whether success survives every attempt rather than at least one. Keep partial credit next to the binary result rather than inside it. Pin the environment and re-baseline on the day. And spend the human attention on the runs that passed, because those are the ones where a widened gate looks exactly like a clean line through it.
