Ten Ways an Agent Evaluation Can Mislead You Even When It's Working Correctly

Grading the grader

Two other pieces in this pillar have already catalogued ten failure modes apiece, and a third has argued that agents need a fundamentally different unit of measurement than models do. One catalogue covers what breaks in an agent’s own wiring — the loop with no stopping condition, the handoff that drops context, the fan-out nobody priced. Another covers what recurs across every frontier model’s own behavior — contamination, sycophancy, drift across versions released under a stable name. A third argues that a trajectory, not a terminal answer, is the correct object to grade, and that a single success indicator throws away almost everything worth knowing about a run.

This piece sits one level above all three. Assume the agent is real and does what it does. Assume the benchmark’s tasks are genuine and its terminal checks execute correctly. Assume the trajectory-grading advice above has even been followed. The evaluation can still produce a number that misleads, because the failure was never in the agent, the model, or the check — it was in how the surrounding methodology was designed, run, and counted. A weak test that a competent agent can satisfy without doing the work. A task sample that does not resemble deployment. Two human graders who privately disagree far more than the published inter-rater figure admits. A handful of trials mistaken for a stable estimate. A judge model that fails to notice the exact thing the system under test is failing to notice, because both share a training distribution. A leaderboard built from many private attempts and one public one. A real, small, statistically detectable gap mistaken for a practically important one. Ten specific, documented ways this happens follow, each traced to a published case or a peer-reviewed methodology critique rather than asserted as a property of evaluation in general.

Where the metric replaces the goal

1. Goodhart’s-law metric gaming. The general phenomenon predates language models by decades. Economist Charles Goodhart’s original observation about monetary policy was given its now-standard paraphrase by anthropologist Marilyn Strathern in a 1997 study of Britain’s university audit system: “when a measure becomes a target, it ceases to be a good measure” ^[2]. Strathern’s paper is not about machine learning at all — it studies how academic departments reshaped their behavior specifically to satisfy the metrics a national research assessment used to rank them — but the mechanism it documents is exactly the one that recurs in agent evaluation: once a proxy is known and rewarded, effort redirects from the underlying goal to the proxy itself. David Manheim and Scott Garrabrant later gave the phenomenon a formal taxonomy, distinguishing at least four distinct failure mechanisms grouped under Goodhart’s name rather than one ^[1]. The simplest to state is regressional: if a proxy metric $M$ relates to the true target $U$ by

for some error term $\varepsilon$ uncorrelated with $U$, then selecting the run with the highest $M$ selects simultaneously for high $U$ and for large favorable $\varepsilon$ — the optimizer is drawn toward the tail of the error term as reliably as toward genuine capability, and the more intensely $M$ is optimized, the larger a share of the observed gain is $\varepsilon$ rather than $U$.

Agent evaluation has a well-documented, concrete instance of exactly this. An audit of SWE-bench, the widely used software-engineering benchmark that checks whether an agent’s patch turns a specified failing test suite passing, found that roughly 33 percent of successful patches in the audited sample had their solution supplied directly in the linked issue report or its comments, and a further 31 percent passed against test suites too weak to actually establish that the patch was correct. Filtering out the affected instances dropped one evaluated system’s reported resolution rate from 12.47 percent to 3.97 percent ^[3]. Nothing in that audit alleges an agent “cheated” in the colloquial sense — the more precise reading is Goodhart’s: wherever the check was weak, optimization pressure toward passing it produced passes that meant less than the number suggested, exactly as Strathern’s academics reshaped their publication behavior around what the assessment counted rather than around research quality itself.

Where the sample never matched the target

2. Non-representative task-distribution sampling. A benchmark score generalizes only as far as the task distribution it was drawn from resembles the distribution it is meant to predict, and that resemblance is rarely checked in public. Percy Liang, Rishi Bommasani, and colleagues built the Holistic Evaluation of Language Models project specifically to make this gap visible: before their standardized taxonomy of scenarios existed, models in the field were on average evaluated on just 17.9 percent of the paper’s core scenarios, with some prominent models sharing not a single evaluation scenario in common with each other ^[5]. Two systems that never took the same test cannot be meaningfully compared, and a system tested on a narrow slice of scenarios tells you almost nothing about the slices it never saw.

The agent case compounds this. Sayash Kapoor, Benedikt Stroebl, and colleagues, surveying agent-benchmarking practice broadly, identify a related but distinct problem: a near-exclusive focus on accuracy on whatever tasks happen to be in a benchmark produces “needlessly complex and costly” agents optimized for that specific sample, and the field routinely conflates the evaluation needs of a model developer with those of a downstream application builder who will deploy the agent on an entirely different task mix ^[4]. A benchmark curated from public GitHub issues in a handful of popular repositories, or from web tasks written by benchmark authors rather than sampled from real user logs, is not a random draw from deployment. It is a convenience sample, and a score computed on it answers a narrower question than the headline implies.

Where the humans disagreed and only one score survived

3. Human-labeler inconsistency on trajectory grading. Whenever an agent’s success depends on a human judgment call — was this trajectory actually helpful, did this refusal happen for a good reason, does this patch actually address the issue — the published number quietly assumes the humans making that call agreed with each other. Marzena Karpinska, Nader Akoury, and Mohit Iyyer surveyed 45 papers that used Amazon Mechanical Turk to evaluate open-ended text generation and found the underlying agreement data, where reported at all, was strikingly poor: for human-written reference stories judged by fully qualified workers, Krippendorff’s alpha — a standard chance-corrected agreement statistic, where 1.0 is perfect agreement and 0 is chance-level — ranged from −0.04 to 0.21 across four rated dimensions (grammar, coherence, relevance, likability), and the fraction of stories on which all three assigned raters gave the exact same rating never exceeded 16.5 percent ^[6]. The same paper found that workers could not reliably tell model-generated text from human-written text unless the two were shown side by side for direct comparison, and that fourteen of the surveyed papers did not even report how many raters or items were used, let alone their agreement.

Not every human-evaluation pipeline is built this loosely. Wei-Lin Chiang, Lianmin Zheng, and colleagues, building the crowdsourced Chatbot Arena, use a Bradley-Terry model to convert pairwise human votes into a ranking rather than trusting a raw vote tally, and report using sandwich-robust standard errors around each model’s estimated strength specifically because naive confidence intervals understate the true uncertainty in this setting ^[7]. The contrast is the point: agreement statistics and calibrated uncertainty are achievable, but they are a design decision that has to be made deliberately, and the Karpinska survey shows that in ordinary practice it usually is not.

Where too few trials bought false confidence

4. Insufficient trial counts producing false confidence. A success rate reported from a small number of trials carries a much wider margin of error than the single reported number suggests, and the interval widens fast as the trial count shrinks. For a naive binomial estimate — treating each trial as an independent coin flip with unknown success probability $p$ — the 95 percent confidence interval around an observed rate $\hat{p}$ from $n$ trials is approximately

At ten trials and an observed 90 percent success rate, that formula gives an interval of roughly plus or minus nineteen percentage points — an agent genuinely succeeding anywhere from about 71 percent to essentially 100 percent of the time is statistically indistinguishable from the ten runs actually observed. Evan Miller, writing at Anthropic, makes this the central practical argument of a 2024 paper on evaluation statistics: he recommends that “new evals should contain at least 1,000 questions in order to have good signaling ability,” and works a concrete power calculation showing that detecting a genuine three-percentage-point difference between two systems, at conventional statistical standards (80 percent power, 5 percent significance), requires approximately 969 questions under reasonable assumptions ^[8]. Most agent benchmarks reported in the field — SWE-bench Verified’s 500 instances, τ-bench’s much smaller per-domain task sets — sit well below that figure, which does not make them worthless, but does mean that a one- or two-point difference between two systems on such a suite is frequently statistical noise dressed as a finding.

Where the trials were never independent

5. Non-independence between repeated trials. The confidence-interval formula above assumes each trial is independent of the others, and that assumption is often false in ways that specifically inflate confidence. Miller’s paper measures this directly: several widely used evaluation sets draw multiple questions from a shared context — several questions about the same passage, several sub-tasks from the same underlying scenario — and properly accounting for that clustering, rather than treating every question as its own independent draw, produced clustered standard errors “over 3x larger than naive standard errors” on the affected evals ^[8]. The general relationship is the one long used in survey statistics: if a naive standard error assumes independence, and trials instead arrive in clusters of average size $m$ with intraclass correlation $\rho$ within a cluster, the correctly inflated standard error is

Agent trials add a second, harder-to-audit source of the same problem: repeated attempts by one policy on one task instance are not independent draws either, because the same weights, the same training distribution, and often the same failure-prone reasoning pattern produce correlated outcomes across attempts. Shunyu Yao and colleagues built a reliability metric directly around this concern, measuring not just whether an agent succeeds at least once across several trials but whether it succeeds on every one of them; on their τ-bench retail domain, state-of-the-art function-calling agents that “succeed on <50% of the tasks” on a single trial saw that figure fall further, to below 25 percent, once the requirement was every one of eight trials succeeding rather than at least one ^[13]. Reporting the friendlier of those two numbers without the trial-independence caveat is a quiet but common way an accurate measurement becomes a misleading headline.

Where the judge shares the defendant’s blind spot

6. Evaluator-model bias. Using a language model to grade another language model’s output is now standard practice, largely because it scales where human review does not. Lianmin Zheng and colleagues found that a strong model judge agreed with human preference judgments more than 80 percent of the time on chat responses, matching the rate at which humans agree with each other — a genuinely useful result — while documenting position bias, verbosity bias, and self-enhancement bias as systematic, repeatable failure patterns in the judge itself ^[11]. Self-enhancement bias is the sharpest version of the concern raised by this failure mode specifically: a judge is not a neutral instrument if it shares a training lineage, a stylistic prior, or a blind spot with the system it is grading.

Arjun Panickssery, Samuel Bowman, and Shi Feng measured this bias directly rather than inferring it. Testing GPT-4 and GPT-3.5 as both generators and evaluators of text summaries, they found self-preference scores — the rate at which a model judge favors output it generated itself, over output from another source rated as equal quality by other means — of 0.705 on one summarization dataset and 0.912 on another for GPT-4, well above the 0.5 that would indicate no bias. Critically, when the labels attached to two summaries were secretly swapped, the judge model’s preference followed the false label rather than the underlying content, showing the bias tracks a model’s belief about authorship rather than any property of the text itself; and the strength of a model’s self-preference correlated with its ability to recognize its own writing, with correlation coefficients (Kendall’s τ) ranging from 0.41 to 0.82 across tested configurations ^[10]. For agent evaluation specifically, the implication is structural rather than incidental: an LLM-as-judge harness built from the same family of models as the agent it grades is not an independent check, because the exact reasoning shortcuts and blind spots that cause the agent to fail are disproportionately likely to be shortcuts and blind spots the judge shares rather than catches.

Where only the winning run got filed

7. Survivorship bias in reported results. A leaderboard number can be entirely honest and still describe only the best of several attempts, with the others simply never disclosed. Shivalika Singh, Sayash Kapoor, Sara Hooker, and eleven co-authors studied this directly for Chatbot Arena, the widely cited crowdsourced comparison platform, analyzing roughly two million battles across 243 models from 42 providers between January 2024 and April 2025. They documented that certain providers test many private, unreleased model variants against the arena before choosing which result to make public, and quantified one case precisely: 27 distinct private variants tested by Meta in the lead-up to its Llama 4 release, from which the publicly reported result was drawn ^[12]. Their summary states the mechanism plainly: “undisclosed private testing practices benefit a handful of providers who are able to test multiple variants before public release and retract scores if desired,” and the ability to choose the best of several private scores “leads to biased Arena scores due to selective disclosure” ^[12]. Nothing about any single published score is fabricated. What is missing is the denominator — how many attempts it took to produce the one number that reached the shelf — and without that denominator, an honest maximum is easy to mistake for a typical outcome.

8. Undisclosed prompt or scaffold tuning against the test set. The closely related failure is not testing many variants and disclosing the best, but tuning the harness itself — the prompt template, the tool-calling scaffold, the retry policy — specifically against the public evaluation before it is run once and reported as a clean result. The same Leaderboard Illusion analysis documents the mechanics that make this possible: providers with privileged access to the arena’s infrastructure can iterate against live evaluation feedback in a way that is invisible to anyone reading only the final score ^[12]. Kapoor and colleagues name the general version of this concern in agent benchmarking directly, arguing that inadequate holdout sets are a direct cause of agents that shortcut and overfit to a specific benchmark’s quirks rather than developing genuinely general task competence, and proposing a framework for the field to guard against exactly this pattern ^[4]. The distinguishing feature of this failure mode against ordinary contamination is intent and disclosure rather than mechanism: the test set was not accidentally seen during training, it was deliberately used, iteratively, as a target for engineering choices that were never described in the reported methodology.

Where a real difference didn’t matter

9. Confusing statistical significance with practical significance. Even a properly powered, correctly computed, genuinely non-zero difference between two systems can be a finding without being a reason to act. Dallas Card, Peter Henderson, and colleagues make the underpowered side of this problem their central argument, showing that a large share of NLP experiments in the published literature lack the statistical power to reliably distinguish a real model improvement from noise in the first place ^[9]. But the inverse failure is just as common once evaluation sets grow large enough to fix the power problem: Miller’s own worked examples show that at sufficient scale, differences of a percentage point or two become statistically detectable even though a one-point shift in an agent’s task-success rate rarely changes a deployment decision on its own ^[8]. A large evaluation set makes small, operationally irrelevant differences statistically visible in exactly the way a small evaluation set makes large, operationally important differences statistically invisible — the two failures are mirror images of the same underlying confusion between a p-value and an effect size, and a methodology section that reports only the former without the latter cannot be checked for either.

Where the average hid its own worst region

10. Aggregate scores that mask where failure concentrates. A single headline percentage is an average, and an average can stay flat while the underlying failure distribution concentrates entirely inside one subgroup the aggregate never surfaces. Yao and colleagues’ τ-bench result already shown above is also the cleanest documented instance of this: “succeed on <50% of the tasks” is one number, but the reliability figure for repeated trials in the retail domain specifically — below 25 percent — is a very different and far more consequential number for anyone deploying that agent specifically in a retail workflow ^[13]. A benchmark that reports one blended figure across domains, task difficulties, or trial counts is making an implicit claim that failure is spread evenly across those categories, and that claim is almost never checked, let alone true. The practical fix costs nothing beyond disclosure: report the distribution across the categories that matter for the deployment decision being made, not only the mean across all of them, because the mean is precisely the statistic most likely to look acceptable while the subgroup that matters most is failing badly.

What a defensible audit would check

None of these ten failures requires a dishonest evaluator, and the case-file wall this article’s world is built around exists for exactly that reason: audits are not conducted because someone is assumed to be lying, but because a benchmark run in good faith can still, mechanically, produce a number nobody should trust at face value. A checklist follows from the ten items above, roughly in the order a reviewer should apply it. Ask whether the check being optimized against is strong enough that passing it implies the underlying task was actually done, per the SWE-bench audit’s finding on weak tests. Ask whether the task sample resembles the deployment distribution, per HELM’s scenario-coverage gap. Ask whether human grading, where used, reports an agreement statistic at all, given how rarely the Karpinska survey found one. Ask whether the trial count is large enough to support the confidence being placed in the headline figure, using Miller’s power calculations as a floor rather than a ceiling. Ask whether repeated trials were treated as independent when the underlying tasks or attempts were clustered. Ask whether an LLM judge shares a training lineage with the system it is grading, given the specific, measured self-preference scores Panickssery and colleagues report. Ask how many variants were tried privately before the published one, per the Leaderboard Illusion’s documented count. Ask whether the harness was tuned against the same set it was then evaluated on. Ask whether a statistically significant difference is also a practically meaningful one. And ask for the result broken out by subgroup, not only as a single mean.

Two predictions follow from this pattern, stated with a horizon and a way to be proven wrong rather than as settled fact. First, over the two years from this writing to mid-2028, expect major agent and model evaluation reports to begin publishing confidence intervals or clustered standard errors alongside headline accuracy figures as a matter of routine, following the direction Miller’s paper argues for; this is disconfirmed if the field’s most-cited leaderboards in 2028 still report bare point estimates with no uncertainty measure attached. Second, expect at least one more widely used public leaderboard to adopt a disclosed-variants requirement similar to the policy change LMArena made after the Llama 4 episode, specifically to close the private-testing gap the Leaderboard Illusion analysis documents; this is disconfirmed if 2028 arrives with major crowdsourced leaderboards still permitting undisclosed private variant testing ahead of public submission.

The room this article is set in never finishes being audited, because the incentive that produces each of these ten failures — a metric that is easier to satisfy than the goal it stands in for — does not go away once one case is closed. What stays constant is the discipline of checking the paperwork behind the number rather than the number itself: which folder was reopened, whether the two pins line up, how many trials the tally actually recorded, whose zero point the judge shares, and how many copies never reached the shelf.