A percentage that is not a description
When a model launch announcement leads with a line like “SWE-bench Verified: 74%,” it is easy to read that number as a report card: this model does a bit better than three-quarters of some general software-engineering job. That reading is almost always wrong, not because the number is false, but because it answers a narrower and more specific question than the one a reader supplies for it. A benchmark score is the output of a pipeline: a fixed task set, a scoring rule, an agent scaffold wired around the model, a compute budget, and a date, and the published percentage compresses all five into one figure. This briefing works through what one well-documented benchmark actually measures, why a model’s raw capability and a harness’s ability to draw that capability out are different things, and why two credible organizations can report two different scores for what is nominally the same model.
What the benchmark actually asks
SWE-bench draws its tasks from real, merged pull requests on twelve popular open-source Python repositories: a model is given an issue description and the codebase at the commit before the fix, and has to produce a patch. Success is not a human judgment; it is a unit-test outcome, whether the patch makes the tests that verify the fix pass while leaving the rest of the test suite passing [1]. That is a real, demanding, and useful task. It is also a much narrower one than “software engineering”: twelve repositories, one language, and issues that were, in every case, already resolved and merged by a human maintainer in the historical record, which is a different population than an open backlog or a greenfield design problem.
A further narrowing sits inside the number many launches actually cite. A meaningful share of the original 2,294 problems turned out to be ungradable in practice: issue descriptions that did not specify enough to fix the bug, or test patches so strict that a genuinely correct fix could still fail them. SWE-bench Verified is a 500-instance subset built by paying professional developers to check each problem for a clear issue description, a correct test patch, and solvability given only the stated information [2]. That screening step makes Verified a better instrument, but it is worth stating plainly: “Verified” describes the instances, not the model. A “74% on SWE-bench Verified” is a claim about 500 specific, human-checked repository problems, not a claim about code in general.
The harness is doing part of the work
A raw model does not resolve a GitHub issue by itself. Something wraps it: a scaffold that decides how many attempts it gets, which tools it can call, how it reads the repository, when it gives up and tries something else, and how much budget it is allowed to spend per problem. That wrapper is usually called a harness, and recent independent work argues it deserves to be named every time a score is quoted, because harness-induced variance can be larger than the variance produced by swapping the underlying model, and can be large enough to change which of two models looks stronger [3]. A model is a fixed set of weights; a score is what a specific harness, at a specific configuration, manages to extract from those weights on a specific day. Those are related quantities, not the same quantity, and the gap between them is where most of the disagreement between confident, contradictory-sounding claims actually lives.
That gap also creates an incentive. Because most agent benchmarks reward the final accuracy number and say little about the cost or complexity spent getting there, teams have a straightforward reason to build elaborate, benchmark-specific scaffolding, extra retries, task-specific heuristics, larger tool budgets, that raises the reported score without necessarily producing a system that is more useful outside that benchmark’s instances. A widely cited critique of agent evaluation practice argues that this pattern is compounded by weak holdout discipline: without a genuinely unseen test set, a scaffold can be tuned, deliberately or not, against the very instances it will later be scored on, and the resulting number describes the scaffold-and-benchmark pair more than it describes general capability [4].
Why two labs can print two different numbers
Even holding the model and the harness fixed, the reported score can still move, because “the same benchmark” is not always the same 500 problems run the same way. One organization that independently reproduces frontier-model scores on SWE-bench Verified evaluates only 484 of the 500 published instances, because the remaining sixteen do not reliably run in its own infrastructure, and it notes that this kind of exclusion is standard practice elsewhere too, citing evaluations of 477 instances for one model and 489 for another [5]. A reader comparing two headline percentages may be comparing runs over different problem sets before any question of model quality or harness design even arises. The same source estimates a five-to-ten percent instance-level error rate even within the human-screened Verified set, meaning some fraction of any reported pass or fail reflects ambiguity in the grading rather than the model’s output [5].
Add to that the choices a vendor makes and does not always disclose in a launch post, how much reasoning effort or token budget the model was allowed, whether the number comes from the vendor’s own tuned harness or a shared reference implementation, whether it is a single run or an average over several, and a single published percentage stops looking like a measurement of “the model” and starts looking like one point sampled from a family of numbers the same weights could plausibly have produced. None of this makes the number fabricated. A vendor’s benchmark claim is a real claim, worth taking seriously as a claim; it is just not the unconditional fact it is usually presented as, and treating a vendor’s self-report and an independently reproduced figure as interchangeable is its own separate error.
What to ask before a score changes your mind
Four questions do most of the work of turning a headline percentage back into something interpretable. Which instances were actually run, and out of how many were published: a subset quietly narrowed for infrastructure reasons is a different test than the full set. Whose harness produced the completions, and is it the vendor’s own tuned scaffold or a shared reference agent that other models were also run through. Is the number a vendor’s self-report or an independent reproduction, since those two carry different evidentiary weight. And measured when, since a model name can point to a system whose serving stack, and sometimes whose weights, change without the name changing.
None of this argues against benchmarks. SWE-bench and its Verified subset are genuinely useful, carefully documented instruments, and the screening and reproduction work described above exists precisely because researchers keep pressure on the number rather than accept it at face value. The right response to a headline like “SWE-bench 74%” is not suspicion. It is the habit of asking what set of problems, whose harness, whose measurement, and when, and treating the percentage as the compressed answer to those four questions rather than as a self-contained fact about a model’s mind.
