Measuring Frontier Models: Contamination, Variance, and What a Score Can Support

A score is an estimate, and estimates have designs

Assaying is an old discipline with a hard-won habit: never report a value without reporting how the sample was taken, what the reference material was, and how many determinations were made. Language-model evaluation has largely skipped that habit, and the consequence is a literature of single numbers that cannot be reproduced, compared, or audited.

Reframe the object. A benchmark score $\hat{s}$ is an estimator of a population quantity $s$ — the model’s success rate over some distribution of tasks $\mathcal{D}$ that somebody hopes resembles the work you actually have. Every property that makes an estimator trustworthy applies:

The number is only as good as three things: whether $\mathcal{D}_{\mathrm{bench}}$ resembles $\mathcal{D}$, whether the $x_i$ are genuinely held out, and whether the indicator is measured with enough repetition to characterise its spread. All three fail routinely, and they fail in different directions.

Variance: one run is a reading, not a measurement

Generation is stochastic. Even at low temperature, a long agentic trajectory branches, and the same configuration on the same task can succeed and fail across runs. A single-run score has a standard error that is rarely reported and, on the small benchmark suites common in this field, is often larger than the differences being celebrated.

For a suite of $n$ independent items with true rate $p$, the standard error of the estimate is

which for $p \approx 0.5$ and $n = 200$ is about 3.5 percentage points — before adding any run-to-run generation variance, and before accounting for the fact that benchmark items are not independent. A reported two-point difference between two systems on such a suite is not evidence of anything.

There is a second variance source specific to agentic evaluation: budget. Reasoning-capable models spend a variable number of tokens per instance, and the current API surface exposes that as an explicit control — model guidance as verified on 8 August 2026 lists reasoning_effort values from none through max ^[10]. Two runs at nominally identical settings can consume very different compute if the model’s own deliberation policy responds to instance difficulty, a behaviour that reinforcement learning on verifiable outcomes is known to induce ^[8]. Reporting a mean accuracy without a mean and spread of tokens consumed describes half the result.

Minimum standard: at least five runs, report the mean with a dispersion measure, and report the compute consumed alongside the accuracy.

Contamination: the reference material is not clean

The sharpest empirical result on this is also the most carefully designed. Zhang and colleagues commissioned GSM1k, a fresh set of grade-school arithmetic problems built to mirror GSM8k on the dimensions that matter — style, difficulty, human solve rate, number of solution steps, answer magnitude — and then measured the same models on both. Accuracy dropped by as much as 8% on the fresh set, with the worst-affected families falling close to 10%, and a model’s gap correlated with its measured probability of generating GSM8k examples verbatim ^[1].

Two things about that result deserve emphasis because both are usually lost.

First, it is not universal. Several families showed little or no gap. Contamination is a property of a particular data pipeline, not an inevitable consequence of training on the web, which means it is a thing vendors can and do control for — and therefore a thing worth asking about.

Second, the methodology is the contribution. A fresh benchmark constructed to match an old one on measurable properties is the only clean way to separate memorisation from capability. Comparisons that merely check for verbatim overlap miss paraphrase, translation, and the far more common case of training on solutions rather than problems.

The structural problem is that a benchmark’s usefulness decays from the moment it is published. It enters the corpus, gets discussed, gets solved in tutorials, and eventually gets trained on. The instrument dissolves in the sample. This argues for treating benchmark suites the way a laboratory treats reference standards — with expiry dates and periodic recertification against freshly prepared material.

Construct validity: what is the quantity, actually?

Suppose the sampling is sound and the set is clean. There remains the question of whether the benchmark measures the thing its name claims.

MMLU established the multi-subject knowledge format that dominates general capability reporting, covering a wide span of academic and professional subjects in multiple-choice form ^[2]. It measures what it measures well. It does not measure whether a model can carry out a task over an hour, recover from its own errors, or refuse when it should — and no amount of improvement on it establishes those.

SWE-bench moved decisively toward realism by drawing tasks from actual GitHub issues in real repositories and scoring by whether the repository’s own tests pass ^[4]. That is a much stronger construct, and it introduces its own confound: the score now measures a whole system — model, harness, tool interface, retrieval, budget — rather than a model. Two labs reporting on the same suite may be measuring quite different systems.

HELM addressed the problem from the opposite direction. Rather than proposing a better single test, it taxonomised the space of scenarios and desiderata and then measured seven metrics — accuracy, calibration, robustness, fairness, bias, toxicity, and efficiency — across sixteen core scenarios for thirty language models under standardised conditions, achieving full metric coverage in 87.5% of scenario–metric cells ^[3]. That design is the correct shape for the problem, and it carries an unavoidable cost: it produces a matrix rather than a rank, and matrices do not make headlines.

Human-preference evaluation, exemplified by pairwise arena comparison at scale ^[5], measures something real and different again: which output people prefer when shown two side by side. This is genuinely useful and it is not a proxy for correctness. Preference rewards fluency, structure, confidence, and length — the same surface properties that make an incorrect answer persuasive.

The honest summary is that no single construct covers general capability, and combining them is the only defensible practice.

Judges, and the circularity problem

Because human grading does not scale, model-graded evaluation has become standard. It has a specific pathology that is easy to state and easy to forget: the judge shares failure modes with the generator.

Process supervision shows what a well-founded verifier looks like. Lightman and colleagues trained a reward model on human labels applied to each step of a solution rather than only to the final answer, reaching 78% on a representative subset of the MATH test set and releasing the 800,000 step-level labels behind it ^[6]. The reason it works is not the architecture. It is that the labels came from outside the system.

Where labels come from inside, the guarantee is gone. A judge model rewards the surface features it was itself trained to produce, and optimising a generator against it moves both toward a shared blind spot. This compounds with the faithfulness problem: Turpin and colleagues showed models produce coherent reasoning that systematically supports a biased answer without ever mentioning the actual cause ^[7]. A judge reading that reasoning sees a well-argued case. It has no access to the fact that the argument is not why the answer was given.

Practical rule: a model judge is acceptable for screening and unacceptable as the final arbiter of anything you would not be willing to be wrong about. Calibrate it against human labels on a stratified subsample, report the agreement rate, and re-check that agreement whenever the generator changes.

Aggregation hides more than it reveals

Even a well-designed suite loses most of its information the moment it is averaged. A single figure across a heterogeneous set of tasks conceals three distinct patterns that have completely different operational meanings.

Consider two systems reporting the same aggregate. The first is uniformly competent: it succeeds on roughly the same fraction of every subcategory. The second is excellent on half the categories and near-useless on the other half. Averaged, they are indistinguishable. In deployment they are entirely different propositions, and which one you want depends on whether your workload happens to sit in the good half.

A third pattern is worse because it is invisible by construction. If a suite contains many easy items and few hard ones, aggregate movement is dominated by the easy majority. A system that has genuinely improved on the difficult tail while regressing slightly on the trivial head will report as flat or worse. Since the difficult tail is usually where the economic value sits, aggregate reporting is systematically biased against exactly the improvements that matter.

The remedy is the one HELM argued for: report the matrix, not the scalar ^[3]. Break results out by task category, by difficulty stratum, and by failure mode. If a single number is required for communication, publish it as a summary of a table that is also published, never as a substitute for one.

There is a related trap in comparison. When many systems are evaluated on many suites and the best pairing is reported, the reported maximum is biased upward by the selection itself. The correction is elementary and almost never applied: fix the suite and the protocol before running, and report every configuration you ran rather than the one that came out best.

Measuring agents is a different problem from measuring models

Everything above assumes a bounded task with a checkable outcome. Agentic evaluation breaks several of those assumptions at once, and the field has not yet settled a methodology.

The unit of observation is a trajectory, not a response. A trajectory has a length, a cost, a sequence of tool calls, a set of side effects on an environment, and possibly a partial result. Scoring it with a single success indicator discards nearly everything that distinguishes a good run from a lucky one. Two trajectories that both end in a passing test can differ by an order of magnitude in tokens, by whether the agent deleted files it should not have touched, and by whether it verified its own work or simply asserted completion.

Three measurement consequences follow.

Environment state must be part of the score. An agent that passes the test by weakening the test has succeeded on the metric and failed at the task. Any harness that does not diff the environment against an expected change set is measuring something other than what it claims. SWE-bench’s design — real repositories, real issues, the project’s own tests as the oracle ^[4] — makes this tractable precisely because the environment is inspectable, and that is the property to preserve when constructing new suites.

Cost is a dimension, not a footnote. Because per-request compute is now a caller-set parameter, an agent’s success rate is a function of budget rather than a scalar. The honest comparison plots success against spend and asks which curve dominates, or fixes a spend and compares. A single point on an unreported budget conveys almost nothing.

Repeated deployment is the operational question, not one-shot success. Exploration metrics that reward finding at least one success among many attempts answer a research question. An operator needs to know whether the system succeeds on every required run. These are close to opposite quantities, and a system tuned for the first can be poor at the second.

The configuration is part of the result

A published score fixes an outcome and usually leaves the conditions implicit. The GPT-5 system card is explicit that its evaluations were run at high reasoning effort and describes the system as routing between components ^[9] — precisely the disclosure that makes a number interpretable, and precisely what most third-party reporting omits.

Anything less than the following is an anecdote:

• Model identifier, pinned, not an alias.
• Reasoning effort or equivalent budget, plus tokens actually consumed.
• Harness: tool set, retrieval, retry policy, stopping condition.
• Prompt and decoding parameters, verbatim.
• Date of the run, because the served system changes underneath a stable name.
• Number of runs and the dispersion, not just the mean.
• Provenance of the benchmark items and when they were created relative to the training cut-off.

Building the instrument you actually need

The strongest position a buyer can occupy is to hold a private evaluation set that no vendor has seen. It does not need to be large; it needs to be representative of your work and genuinely withheld.

Three properties make such a set worth the effort of building. It should be drawn from your own task distribution rather than an academic proxy, because the whole failure of public benchmarks is a mismatch between $\mathcal{D}_{\mathrm{bench}}$ and $\mathcal{D}$. It should carry a checkable oracle wherever one can be constructed — an expected output, an invariant, a schema, a test — so that most of it can be re-run automatically. And it must never be published, pasted into a hosted model as an example, or committed to a public repository, because any of those actions ends its usefulness permanently.

Refresh it on a schedule. A private set contaminates too, not through training but through your own overfitting: after enough iterations of prompt tuning against it, you have optimised for the set rather than the task. Rotating a fraction of the items each quarter and keeping a small never-touched holdout is the cheapest available defence.

Re-run it on every model change, every effort-setting change, and on a calendar interval regardless — because the served system moves under a stable name. When a number shifts, the private set tells you whether the shift is real before anyone has published an opinion about it.

A protocol worth adopting

Assayers converged on their habits because the cost of a wrong number was borne by whoever acted on it. The same is true here, and the same remedies apply.

Split the sample deliberately: stratify by task type and difficulty rather than taking whatever the suite happens to contain. Prepare fresh reference material: hold back a set built after the model’s cut-off and never publish it. Run the determination more than once and report the spread. Separate the impurity: when a result looks strong, test the same capability on a matched fresh set before believing it. Calibrate the instrument against an outside standard: human labels on a subsample whenever a model is doing the grading. And record the conditions, because a determination whose conditions are unknown is not a determination.

The goal is not a better leaderboard. It is a number that still means something to someone who reads it six months later, on a system that has changed twice since.