How to Actually Compare Frontier AI Models Without Building a Misleading Leaderboard

A score is a coordinate, not a verdict

Every few months a chart circulates with model names down the side, a single percentage beside each, and a bar highlighted in the sponsor’s colour. It reads like a finish line. It is closer to a coordinate with three of its five values missing.

A reported evaluation score is produced by a specific configuration: a model snapshot, an evaluation harness, a sampling setup, a prompt template, and a date. Change any one of those five while holding the model fixed and the number moves — sometimes by more than the gap the chart is trying to illustrate. This is not a hypothetical risk. It happened, in public, to OpenAI’s own flagship release. The GPT-5 system card reports its highest SWE-bench Verified figure — 74% pass@1 on the 477-task subset OpenAI evaluates internally — and then adds a footnote that the number in the GPT-5 launch blog post was 74.9%, on the same benchmark, because the launch post used the API’s default “medium” verbosity setting while the system card’s preparedness evaluations ran at the model’s maximum trained-in verbosity. OpenAI states plainly that “changes in verbosity can lead to variation in eval performance” ^[4]. Two numbers, one model, one benchmark name, published by the same company within the same document — because a decoding setting, not the weights, was different.

This article is the opener for a series on frontier-model comparisons. Its job is not to rank OpenAI against Claude against Gemini — that would violate the discipline the whole series exists to enforce. Its job is to work out what a defensible comparison requires, so that when the siblings in this series do run head-to-head evaluations, the reader can tell the difference between a real gap and an artifact of how the test was set up.

The stakes are not academic. A team choosing which frontier model to build a product on, or a procurement group comparing vendor quotes, routinely makes that decision by reading exactly the kind of chart described above — a handful of percentages, no configuration attached, treated as though the number describes the model the way a clock speed describes a processor. It does not. Two systems can be placed side by side honestly only once someone has pinned down which snapshot, which harness, which sampling setup, and which date each number came from; skip that step and the decision is being made on noise dressed up as a finding.

Where a number comes apart, first: the harness

“SWE-bench Verified” is not one measurement. It is a benchmark specification — 500 real GitHub issues with held-out tests — wrapped in whatever scaffold the evaluator built to let a model read the repository, edit files, run commands, and decide when to stop. The scaffold is software, and different scaffolds produce different resolve rates from the identical underlying model, sometimes by tens of points.

Anthropic’s own model card addendum for the upgraded Claude 3.5 Sonnet makes this concrete without any outside party needing to demonstrate it. The document reports the model’s “state-of-the-art pass@1 performance on SWE-bench Verified of 49.0%,” immediately followed by: “We believe dedicated scaffolding and prompting can further improve the results.” A footnote then compares this number to “the current state-of-the-art score of 45.2% as reported on SWE-bench’s leaderboard as of October 22nd, 2024” ^[3] — a different number, on the same nominal benchmark, at the same point in time, because the public leaderboard entry used a different scaffold than Anthropic’s internal one.

Even the denominator moves. Anthropic’s own addendum evaluates on 489 of the benchmark’s 500 tasks for one model generation; Epoch AI’s independent implementation excludes 16 tasks it judges unreliable, evaluating on 484 ^[10]; OpenAI’s GPT-5 system card evaluates on 477 ^[4]. Google DeepMind’s methodology note for Gemini 3.1 Pro goes further and documents the mechanism directly: its team found three SWE-bench Verified tasks with broken reference tests in the public harness — one with a test referencing a function that does not exist in the target repository, two more where a required flag silently reverts when run against the current official Docker images — and after patching those three locally, adjusted its own model’s reported score up by 0.6 percentage points ^[12]. None of these organisations is being careless. The benchmark’s name is stable; the software running it is not, and every serious evaluator ends up patching, subsetting, or rescaffolding it slightly differently.

Where it comes apart again: sampling and decoding

A second axis moves scores independently of the harness: how the model is sampled. The clearest illustration sits inside a single table in Anthropic’s original Claude 3.5 Sonnet addendum. On GPQA Diamond — a set of graduate-level physics, chemistry, and biology questions built so that non-experts with unrestricted web access for over half an hour still cannot reliably answer them, hence “Google-proof” ^[6] — the model scores 59.4% under zero-shot chain-of-thought, single sample, and 67.2% under “Maj@32 5-shot CoT” — thirty-two independent samples per question, five worked examples in the prompt, and the majority answer taken as the model’s response ^[2]. That is an eight-point swing for the same weights, produced entirely by how many times the model was asked and how the answer was aggregated. A single-sample estimate and a 32-sample majority vote are not two measurements of the same quantity at different precision; they are estimates of two different quantities, and only one of them resembles what a single user turn actually gets.

Temperature compounds this. Artificial Analysis, an independent benchmarking group, discloses using a temperature of 0 for standard models and 0.6 for reasoning models “as adjusted per model lab recommendations,” alongside pass@1 scoring aggregated over three to five repeats per task ^[11]. Google’s own evaluation methodology note states plainly that its results for non-Gemini models are “sourced from providers’ self-reported numbers unless mentioned otherwise,” and that for several competing models the company defaults “to reporting maximum thinking/reasoning settings available,” falling back to “best available reasoning results” when the maximum setting was not published ^[12]. That is an honest disclosure of a real problem: a single comparison table can quietly mix a competitor’s number run at low effort with your own model’s number run at maximum effort, because the competitor never published the alternative.

Formally, an observed score $s$ is not a property of a model $\theta$ alone. It is closer to

a function of the weights $\theta$, the harness $\mathcal{H}$, the sampling temperature $T$, the number of samples $k$ and how they are aggregated, the prompt template $\mathcal{P}$, the date $t$ a snapshot was queried, plus sampling noise $\varepsilon$. A citation that fixes $\theta$ and leaves the other five free has not specified a comparable quantity. OpenAI’s own simple-evals table makes the same point from a different angle: rows the lab ran itself under a disclosed zero-shot chain-of-thought protocol sit beside rows for competitors’ models labelled “unknown” prompt style, sourced from those vendors’ own announcements rather than reproduced on OpenAI’s harness ^[5]. Even holding the harness fixed, dated snapshots of the same public model name diverge: that table lists three different GPT-4o release dates scoring 46.0%, 49.9%, and 53.1% on GPQA under the identical simple-evals protocol — a seven-point range attributable to nothing but which week the model was queried ^[5].

Contamination: when the test has already been seen

A third failure mode does not depend on how the model is sampled at all — it depends on whether the model has already memorised the questions. Zhang and colleagues tested this directly by constructing GSM1k, a fresh benchmark built to mirror GSM8K’s style, difficulty, and format exactly, using problems guaranteed to be absent from any pretraining corpus. Accuracy fell by as much as eight percentage points when models were moved from the familiar GSM8K to the fresh GSM1k, several model families showed systematic overfitting across nearly every size tested, and the researchers found a measurable correlation between how readily a model could regenerate a GSM8K example verbatim and how large its performance gap was between the two benchmarks — evidence consistent with partial memorisation rather than pure reasoning. The frontier models tested showed comparatively little of this effect ^[8]. That last finding matters as much as the headline one: contamination is a real, measurable risk, but it is not evenly distributed, and asserting it about a specific model on a specific benchmark requires the kind of held-out replication GSM1k performed — not an assumption.

This is also why the reference-set discipline in this series’ image world is not decorative. A benchmark whose questions are public and stable is a benchmark that gradually enters training data through search results, forum posts, and reproductions of the questions themselves. Treating a held-out set as something to protect under something like chain-of-custody control — track when it was last refreshed, whether any model could plausibly have seen it, and what fraction of gain a fresh replacement erases — is the operational answer to contamination risk, and it is precisely what GSM1k did as a one-off study.

Cherry-picked subsets and the arithmetic of “wins”

The fourth failure mode requires no methodological error at all — only selective disclosure. A vendor claiming “N of M benchmark wins” has made a claim that depends entirely on which benchmarks appear in M, and whether every competitor being compared against actually reported a number for each one.

A detailed technical critique of Google’s Gemini 3.1 Pro launch materials, published shortly after the February 2026 release, works through exactly this mechanism. Of the sixteen benchmarks in Google’s headline comparison, the reviewer found that a named competitor model had published scores on only two, meaning the remaining fourteen “wins” were credited against an opponent that had simply not disclosed a number in that category — not against a demonstrated loss. The same review reports that Google’s own materials omitted a result in which Gemini trailed a competing model by a wide margin on an enterprise-task benchmark, and separately notes that a leading blind-preference platform — built on the pairwise human-preference methodology that Chatbot Arena formalised, where raters compare two anonymised outputs and rankings are recovered from the resulting win rates rather than from any fixed answer key ^[7] — showed the two frontier systems in that comparison as statistically indistinguishable, “essentially tied,” even while the published chart showed a wide gap ^[13]. None of this requires accusing anyone of fabrication — it only requires noticing that a comparison table built from voluntarily disclosed numbers structurally favours whoever chooses which categories to publish.

This is worth stating plainly, and it is exactly why this series holds Google and Gemini to smaller, evidence-proportionate coverage rather than a parallel deep dive: the strongest, most reproducible evidence available about Gemini’s standing right now is evidence about how its comparisons were constructed, not a settled capability finding, and this article reports it as exactly that. The same caution applies with even more force to Meta’s Llama family and to xAI’s Grok — both appear in this piece only as data points already visible inside sources cited above, such as the self-reported figures collected in OpenAI’s simple-evals table ^[5], because both already have their own dedicated series elsewhere in this corpus, and duplicating deep coverage here would not add evidence, only length.

What a rigorous comparison actually requires

Four things, and none of them are exotic.

The same harness. If scaffold choice alone can move SWE-bench Verified by the kind of margin Anthropic’s own footnote records between its internal number and the public leaderboard, then two models must be run through one shared harness before a gap between them means anything about the models rather than the harnesses.

The same prompts and sampling configuration. Zero-shot versus few-shot, single-sample versus majority vote over many, temperature 0 versus a lab-recommended default — pick one setup and hold it fixed across every system under test. Artificial Analysis’s practice of fixing temperature by model type and aggregating a set number of repeats per task, applied identically across every vendor in its index, is the operational version of this rule ^[11].

Reported variance, not a bare point estimate. A score computed from $n$ independent trials with an underlying success probability $\hat p$ carries a standard error of roughly

and two point estimates whose intervals overlap should not be reported as a ranking. Evan Miller’s statistical treatment of language-model evaluation makes this argument in more general form, framing individual evaluation questions as draws from an unseen larger population and deriving the formulas needed to report genuine uncertainty rather than a single noisy number ^[1]. Epoch AI’s benchmarking hub applies the same discipline operationally: it runs each model sixteen times on GPQA Diamond and Mock AIME, eight times on MATH Level 5, and reports a one-standard-error confidence interval alongside every score ^{[9] [10]}.

Disclosed methodology, including what you do not know. The strongest practice visible in the sources behind this article is not a perfect comparison — none of them is — but a labelled one. OpenAI’s simple-evals table marks competitor rows “unknown” rather than presenting them as equivalent to its own disclosed protocol ^[5]. Google’s Gemini 3.1 Pro methodology note states outright which competitor numbers are self-reported, which reasoning setting was assumed when the vendor did not specify one, and which of its own benchmark implementations were patched and why ^[12]. A comparison that tells you exactly where its own uncertainty lives is more trustworthy than one that reports no uncertainty at all.

A worked example, restated in full

Put the pieces from the two frontier labs given primary coverage in this series back together, because the pattern is the point.

OpenAI published two different SWE-bench Verified numbers for its own GPT-5 reasoning model within a matter of weeks: 74.9% in the model’s launch materials, at the API’s default medium-verbosity setting, and 74% in the formal system card’s preparedness section, at the model’s maximum trained-in verbosity, evaluated on a 477-task subset with an explicit note attributing the difference to the verbosity change ^[4]. Anthropic published two different SWE-bench Verified numbers for the same upgraded Claude 3.5 Sonnet release at effectively the same moment: 49.0% under its own internal scaffold, and a co-reported 45.2% for the contemporaneous public leaderboard entry, with the gap explicitly attributed to scaffolding and prompting differences the company believed it could still improve on ^[3]. And on a second benchmark entirely, Anthropic published two numbers for one model in one table: 59.4% and 67.2% on GPQA Diamond, separated only by sample count and aggregation method ^[2].

None of these six numbers is wrong. Each is a correct report of a specific, named configuration. The mistake would be citing any one of them as “GPT-5’s SWE-bench score” or “Claude’s GPQA score” without the configuration attached — because the other number, from the same lab, about the same weights, is sitting in the same document, and it disagrees by a margin larger than most publicly claimed differences between competing frontier models. If two evaluations of one company’s own model can differ this much on methodology alone, a comparison across companies that does not control for the same five variables is not measuring a capability gap. It is measuring whichever configuration each vendor happened to choose to publish.

The discipline this series will hold to

Every comparison that follows this piece in the series will state, for each system under test: the exact model identifier and date, the harness and its version or source, the sampling configuration including temperature and sample count, the prompt template, and — wherever the underlying data permits it — a variance estimate rather than a bare percentage. Where a vendor’s own disclosed number cannot be reproduced under a shared harness, that number will be reported as a vendor claim, attributed and flagged as such, not folded into a ranking beside independently reproduced figures. Where two credible evaluations of the same models disagree, as they demonstrably do above, the series will describe the disagreement and its likely source rather than adjudicate a winner from it. And per the editorial rule that opened this piece: OpenAI and Claude will receive detailed, primary comparative treatment throughout the series, because they are where the best-documented, most reproducible evidence currently exists; Gemini will receive coverage sized to the evidence actually available, as it received here; Llama and Grok will appear only in brief, clearly labelled mentions, because both already have dedicated series elsewhere in this corpus that this one has no reason to duplicate.

Predictions, with what would falsify them

Horizon: 12 August 2028.

One. Independent, harness-controlled benchmarking groups — in the mould of Epoch AI and Artificial Analysis — will grow in citation weight relative to vendor-published system-card numbers for cross-vendor comparisons, because the discrepancies documented in this piece are structural, not a one-off embarrassment. Disconfirmed if major technology press coverage in 2028 still primarily cites vendor self-reported headline numbers for cross-vendor rankings without independent verification.

Two. At least one of the frontier labs discussed here will begin publishing confidence intervals alongside headline benchmark numbers in its own system cards by default, following the practice Epoch AI and Miller’s statistical framework already establish as available and low-cost. Disconfirmed if system cards published by these companies in 2028 still report point estimates only, with no interval or repeat-count disclosure, on their primary headline benchmarks.

Three. SWE-bench-style agentic coding benchmarks will be superseded or heavily revised specifically because of scaffold sensitivity, in the same way earlier static benchmarks were retired after contamination made them unreliable. Disconfirmed if SWE-bench Verified, run under materially the same scaffolding-dependent protocol, remains a headline benchmark in 2028 system cards without a standardised, shared reference scaffold.

What to take away

A leaderboard number is not evidence about a model. It is evidence about a model evaluated under one specific configuration, on one date, by one organisation with one set of incentives about which numbers to publish. The four failure modes worked through here — mismatched harnesses, unstated or mismatched sampling, contamination of the reference material, and selectively disclosed benchmark subsets — are not edge cases found by digging for scandal. Three of the six numbers cited in the worked example above came from the evaluated labs’ own official documents, disagreeing with themselves. Anyone who wants to compare two frontier models honestly has to fix the harness, fix the sampling, report the variance, and say exactly what was measured — and anyone reading a comparison that skips one of those four should treat the missing piece as the most important unstated fact in the chart.