The Hardest Unsolved Problems in Frontier AI Model Comparisons

A scoreboard implies a shared instrument

A leaderboard, a model card’s benchmark table, and a press release’s percentage-point comparison all share one visual grammar: numbers in a column, ranked. That grammar borrows the authority of measurement — the implicit claim that the same ruler was laid against every model, in the same units, and that what differs is only the thing being measured. Almost none of that is true of frontier AI model comparisons as currently practiced, and it is worth being precise about why, because the gap is not a matter of sloppy reporting that better discipline would fix. It is a set of open methodological problems, several of which nobody currently knows how to solve even in principle.

This article works through six of them: the absence of a shared unit for “capability,” benchmark contamination that cannot be verified from outside a closed laboratory, human-preference leaderboards that measure a gameable social process rather than ability alone, cost and latency comparisons that depend on an operating point nobody holds fixed, multimodal benchmarks that average across incompatible input pipelines, and safety postures disclosed through incompatible, self-defined vendor taxonomies. Each is illustrated with a specific, attributed finding from a model card, an independent evaluator, or a benchmark methodology paper. None of it is assembled into a ranking of which frontier model is “best,” because the argument of this article is precisely that the evidence available today cannot support one without hiding the seams.

The same six problems recur, with varying evidence available to the public, across every major frontier lab — OpenAI and Anthropic have published the most detailed model cards and are used here as the primary illustrations, but Google’s Gemini family, Meta’s Llama family, and xAI’s Grok family face the identical structural problems, typically with less independent, reproducible third-party evaluation coverage rather than a different set of problems.

Problem one: there is no shared unit of capability

“Capability” is not a scalar quantity waiting to be measured more precisely. It is a label applied to scores from benchmarks that test qualitatively different constructs: multiple-choice recall of specialized knowledge, step-by-step mathematical derivation, code that must actually compile and pass hidden tests, long-horizon agentic task completion, and human raters’ subjective preference between two conversational replies. Averaging or ranking across these is a modeling choice, not a physical fact about the models being compared, and different reasonable choices produce different orderings.

Stanford’s HELM project made this explicit rather than papering over it: instead of collapsing evaluation into one headline number, it deliberately benchmarked dozens of language models across sixteen core scenarios and seven metrics — accuracy, calibration, robustness, fairness, bias, toxicity, and efficiency — under standardized conditions, reporting the resulting matrix rather than a single score ^[7]. That design decision is itself an argument: the HELM authors concluded that a single aggregate number would misrepresent what the underlying scores actually show, so they refused to produce one. A reader who wants “the” HELM ranking has to supply their own weighting across seven incommensurable axes, and different weightings can reorder the field.

The opposite approach — build one unified scale anyway — is being tried too. Epoch AI’s benchmarking hub combines scores from dozens of separate benchmarks into what it calls a single general-capability index, explicitly intended to track long-run trends in frontier progress across models and time ^[11]. That is a useful trend line for the specific purpose of asking “is frontier capability still advancing, and how fast” — a research question about the field, not a procurement question about two specific systems. Treating an aggregate trend index as a device for ranking model A against model B on a specific task is a different use than the one it was built for, and nothing about the index’s construction validates that different use.

A third candidate unit sidesteps aggregation entirely by choosing a construct with an external, human-legible meaning. METR, an independent nonprofit evaluator, defines a task-completion time horizon: the duration of task, measured by how long a skilled human would need, at which a model succeeds with a stated reliability threshold ^{[6] [12]}. METR reports that this horizon has been roughly doubling every seven months since 2019, with a frontier reasoning model reaching approximately fifty minutes of human-equivalent task length in the study’s data ^[6]. That is a genuinely well-defined unit — it converts model performance into a quantity with a real-world meaning, “the length of task a domain expert would need this long to finish” — but it is a unit built for one narrow construct, long-horizon autonomous software and agentic tasks, and it says nothing about factual recall, multimodal grounding, or conversational quality.

The underlying reason none of these approaches resolves the problem is structural. A published score for model $i$ on benchmark $b$ can be decomposed, at least conceptually, as

where $\theta_i$ is some unobservable underlying ability vector for model $i$, $g_b$ is the mapping from ability to expected score that is specific to benchmark $b$ and not shared across benchmarks, $\delta_{i,b}$ is a contamination or leakage term specific to that model-benchmark pair, and $\varepsilon_{i,b}$ is sampling and decoding noise. Because $g_b$ differs across benchmarks by construction — a multiple-choice knowledge test and a pairwise human-preference vote are not measuring the same projection of $\theta_i$ — there is no aggregation operator that turns a vector of $\hat{s}_{i,b}$ values into a single number without an additional, unverified assumption about how the $g_b$ functions relate to one another. HELM’s response is to refuse the aggregation and publish the matrix. Epoch’s response is to aggregate anyway, for a stated and narrower purpose. Neither is wrong; neither solves the general problem, because the general problem — finding the true, benchmark-independent $\theta_i$ from observed $\hat{s}_{i,b}$ — remains open.

Problem two: contamination nobody outside the lab can verify

Even a single benchmark’s score is compromised if the term $\delta_{i,b}$ above is nonzero: if some of what looks like reasoning is actually recall of a benchmark item, or an item closely resembling it, that appeared in the model’s training data. Public benchmarks such as GSM8k have been circulating on the open web for years, meaning they, and countless worked solutions and discussions of them, are plausible components of any large modern pretraining corpus.

Zhang and colleagues investigated this directly for grade-school arithmetic by constructing GSM1k, a fresh benchmark deliberately built to match GSM8k’s style, topics, and difficulty distribution without reusing any of its actual items, then measuring the accuracy gap between the two on the same models ^[8]. They found accuracy drops of up to eight percentage points for some models moving from GSM8k to the fresh, matched-difficulty GSM1k — evidence consistent with partial memorization of the older, more widely circulated set — while reporting that the frontier models in their study showed comparatively little degradation and continued to generalize to the novel problems ^[8]. That finding should be read carefully and narrowly: it is evidence of contamination for some models on one benchmark family, using one specific method for constructing a matched-difficulty replacement, not a general audit of any specific vendor’s entire benchmark suite.

The deeper, unsolved part of the problem is that this kind of test is the best tool currently available, and it has a structural limitation: it can be run only after the fact, on a fixed pair of benchmarks, by researchers who happen to build and publish a matched replacement. For a closed frontier model, nobody outside the laboratory that trained it can inspect the pretraining corpus directly, so contamination can only ever be inferred indirectly — through held-out replacement benchmarks like GSM1k, through membership-inference probes, or through behavioral quirks. None of these methods can certify a benchmark as clean going forward, either: a held-out set that is queried repeatedly through a public API, or published once a study cites specific items from it, has begun the same slow process of becoming part of some future model’s training data. Sealing a test set delays contamination; nothing currently available prevents it indefinitely.

Problem three: human-preference leaderboards measure a different, gameable construct

A second, popular family of comparisons sidesteps fixed-answer benchmarks entirely by asking humans which of two anonymized model outputs they prefer. Chatbot Arena, built by Chiang and colleagues, collects pairwise votes from a broad, self-selected pool of users and derives a ranking from the aggregated comparisons, reporting that its rankings correlate well with the judgments of expert evaluators on the prompts it has collected ^[4]. This is a genuinely different construct from benchmark accuracy — it measures which response a rater preferred, which conflates task correctness with tone, formatting, verbosity, and stylistic fit to what raters expect a “good” answer to look like.

That conflation would already limit how much a preference leaderboard tells you about capability in general. Singh and colleagues, in a 2025 study examining roughly two million battles across 243 models and 42 providers over sixteen months, documented a further, structural problem: providers do not have equal access to the evaluation process itself ^[5]. Their analysis found that undisclosed private testing let some vendors trial many variants before choosing which one to expose publicly — the paper reports Meta testing approximately 27 private model variants ahead of the Llama 4 release — and found stark asymmetries in how much arena data different providers had received, with Google and OpenAI’s models each receiving roughly a fifth of all arena battles while 83 open-weight models combined received under a third ^[5]. Most strikingly for the aggregation problem above, the study reports that additional exposure to arena-style data was associated with performance gains of up to 112% on the platform’s own metric — a result the authors interpret as evidence that some of what the leaderboard rewards is adaptation to the platform’s specific sampling and selection process, not general quality ^[5].

None of this means a human-preference leaderboard, such as the one now operated at Arena.ai, is uninformative ^[10]. It means the number it publishes answers a narrower and more contingent question than “which model do people prefer”: it answers “which model did people prefer, among the specific variants providers chose to expose, sampled under this platform’s specific selection and disclosure rules, over this specific period.” Treating that number as a portable measure of general ability — the way a wind-tunnel drag coefficient is treated as a property of the airframe rather than of the particular wind tunnel — is not something the evidence above supports.

Problem four: cost and latency comparisons have no fixed reference frame

Comparisons that move past raw capability into cost and latency face a different version of the same underlying issue: there is no single operating point at which “model A versus model B” is a well-posed question. A frontier system’s price per output token, tokens-per-second throughput, and time-to-first-token all depend on the reasoning-effort or “thinking budget” setting selected for the request, the batching policy and hardware generation the vendor happens to be running that request on, and the length of the prompt and response — none of which is a fixed property of the underlying weights.

Independent trackers such as Artificial Analysis attempt to hold this still by measuring quality, price, and speed directly against live provider APIs across hundreds of models from OpenAI, Anthropic, Google, Meta, and others, rather than relying on vendor-reported figures ^[13]. That is a real improvement over trusting self-reported benchmark tables, but it only fixes one moment in a fast-moving series of snapshots: prices and serving configurations change frequently enough that any single comparison is a photograph of a point that may no longer exist by the time it is read, and the underlying “quality” component of such an index inherits every aggregation problem described above.

More fundamentally, even a perfectly executed, continuously updated tracker cannot make the comparison one-dimensional, because quality at a given cost and latency is a function of a whole vector of settings — reasoning effort, context length used, modality, batching — and two systems compared at one point in that vector can reverse their relative standing at another point:

A model that leads at high cost and maximum reasoning effort is not guaranteed to lead at the low-cost, low-latency point a production deployment actually needs, and a comparison that quotes only one of those two points without saying which is not wrong so much as silently incomplete. No published methodology currently reports a full efficient frontier across this vector for arbitrary vendor pairs; what exists are snapshots at whichever points the comparing party chose to query.

Problem five: multimodal comparison across incompatible input types

Comparisons get harder still once inputs stop being text. The MMMU benchmark, built by Yue and colleagues, tested multimodal models against roughly 11,500 college-level questions spanning six academic disciplines that require both image understanding and expert reasoning; at the time of its publication, the strongest multimodal systems evaluated managed only 56% and 59% accuracy against a much higher expert ceiling, showing that even models built specifically to handle combined text-and-image reasoning were far from solving the construct MMMU was designed to test ^[9].

That headroom is itself informative, but it understates a harder problem underneath it: frontier systems do not agree on what “multimodal” means as an input contract. Some models accept image and text natively in one forward pass; others reach audio or video understanding through a separate transcription or frame-sampling pipeline bolted in front of a text model; still others handle only a subset of these combinations at all. A benchmark score built from a fixed mix of modality tasks — some fraction image, some fraction audio, some fraction video — embeds a hidden value judgment about how much each modality should count, chosen by the benchmark’s designers rather than by the comparison’s users, and a model with no native audio pathway is not measured on the same footing as one built around it merely because both received an overall “multimodal score.” There is currently no accepted exchange rate between a point of accuracy on an image-reasoning task and a point of accuracy on an audio-reasoning task, so weighting them into one number is an editorial choice dressed as a measurement.

Problem six: safety posture and refusal behavior aren’t on a common scale

The final comparison problem is arguably the least tractable, because it concerns policy as much as capability. OpenAI’s Preparedness Framework describes a structured process for tracking frontier risk across four categories — cybersecurity, chemical/biological/radiological/nuclear, persuasion, and model autonomy — and states that only models scoring “medium” or below on post-mitigation risk in each category may be deployed, with models scoring “high” restricted from further development until safeguards are in place ^[3]. The GPT-5 system card describes the safety evaluation regime applied to that specific model family, again in OpenAI’s own stated terms ^[1].

Anthropic’s system card for Claude Opus 4.5 describes a different structure entirely: safeguards testing, honesty evaluations, agentic-safety testing, a comprehensive alignment assessment, a model welfare report, and evaluations mandated by Anthropic’s own Responsible Scaling Policy ^[2]. These are both, in every instance above, vendor assertions about the vendor’s own internal process — they should be read and attributed as such, not treated as independently verified measurements, because in both cases the laboratory that built the model is also the party running and disclosing the safety evaluation of it.

Even taken entirely at face value, the two frameworks do not share a taxonomy, a threshold vocabulary, or a public crosswalk between them. “Medium” risk in OpenAI’s framework and whatever internal threshold triggers a given Responsible Scaling Policy tier in Anthropic’s framework are not defined against a common external reference, so a claim that one vendor’s model is “safer” than another’s cannot currently be verified by comparing their published levels — the words happen to look comparable without being calibrated to the same scale. Refusal rate itself compounds this: a model that refuses more requests is not straightforwardly safer, because over-refusal (declining benign requests) and under-refusal (complying with harmful ones) trade against each other, and no public methodology assigns a shared cost function to that trade-off across vendors with different risk tolerances, different legal exposure, and different target user bases. Where the two labs’ approaches genuinely disagree — on how much weight to give model welfare assessments, for instance, which appears in Anthropic’s card and not in OpenAI’s published framework — the honest description is that they disagree, not that one is more rigorous than the other; nothing in the public record settles that question.

Why aggregate leaderboard scores can actively mislead

Put the six problems together and a single leaderboard number becomes actively dangerous in one specific way: it hides which problem is driving any observed gap between two systems. A ten-point difference on a public benchmark could reflect superior underlying ability, a contamination advantage from problem two, a stylistic fit to human raters from problem three, an unmatched cost or effort setting from problem four, an easier modality mix from problem five, or some combination of all of them — and from the number alone, a reader outside the laboratories involved cannot tell which.

This is a specific instance of a well-documented general pattern: once a metric becomes a target that career incentives, funding decisions, or procurement contracts are attached to, the measured proxy tends to diverge from the underlying quality it was meant to approximate. The Leaderboard Illusion study’s finding of performance gains of up to 112% from additional platform-specific data exposure is direct empirical evidence of exactly this divergence occurring on a real, widely cited leaderboard, not a hypothetical concern ^[5]. A number that can be moved substantially by optimizing for the test rather than the underlying ability has stopped functioning as a measurement of that ability, whatever its face value suggests.

What is being tried, and why none of it is a full solution

It is worth being fair to the field’s own attempts, because several of them are genuine partial progress rather than window dressing, and it would overstate the case to say nothing has improved. HELM’s refusal to aggregate at least stops a false single number from being manufactured, at the cost of pushing the weighting decision onto the reader ^[7]. Epoch’s capability index gives researchers a defensible way to track the pace of frontier progress over time, for the specific purpose of watching a trend line, provided it is not repurposed as a procurement ranking tool ^[11]. METR’s time-horizon metric supplies a genuinely externally meaningful unit for one narrow, important construct — autonomous task duration — without pretending to generalize beyond it ^{[6] [12]}. Independent cost and latency trackers such as Artificial Analysis remove vendor self-reporting from at least the price and speed components of a comparison, even though the quality component they combine it with inherits every problem above and the whole snapshot ages quickly ^[13]. Each of these is a real improvement on a narrower problem than “which model is best,” and treating any one of them as having solved that broader question would be a misreading of what its designers built it to do.

Practical rules for reading a comparison

Four rules follow directly from the six problems above, and they are the practical content of this article.

Ask which construct is being measured before treating two numbers as comparable. Task-completion accuracy, pairwise human preference, and long-horizon time-to-completion are different constructs measuring different things; a gap on one says nothing certain about the others.

Demand the operating point, not just the model name. A comparison that does not state the cost, latency budget, context length, and reasoning effort at which both systems were run is a comparison at an unstated and possibly unmatched point, and the ordering can reverse elsewhere on that same vector.

Treat a human-preference leaderboard score as a statement about that platform’s sampling and disclosure rules, not about ability in general — particularly where, as documented for Chatbot Arena, disclosed and undisclosed access to the evaluation process is known to differ sharply across providers ^[5].

Treat vendor safety-posture claims as self-reported policy positions unless independently reproduced. A system card describes what its own authors chose to test, measure, and disclose; it is evidence about the vendor’s process, not a measurement on a scale shared with any other vendor’s card.

Predictions, with the observations that would falsify them

These are forecasts, clearly separated from the sourced analysis above, offered with the assumptions and disconfirming evidence that would show them wrong.

One. Horizon: 2029. At least one independent evaluator will begin publishing a standing, periodically refreshed contamination-audit score — a held-out benchmark whose measured overlap with public training data is disclosed alongside the score itself — that becomes a routine companion citation to raw benchmark numbers in the way GSM1k-style replacement sets are used today. Assumption: contamination remains a reputational liability labs want to be seen addressing. Disconfirmed if by 2029 major system cards still report only static public-benchmark scores with no accompanying contamination disclosure of any kind.

Two. Horizon: 2028. Mainstream press coverage and procurement documentation will shift away from citing single-number leaderboard rankings as the default comparison format, toward citing matched-operating-point comparisons (fixed cost, latency, and context length) as the credible standard, specifically because the current format has been shown to be manipulable. Disconfirmed if a single Elo-style leaderboard number remains the primary citation in mainstream technology reporting with no widely adopted matched-point alternative in use by then.

Three. Horizon: 2027. No cross-vendor standardized safety-risk taxonomy will exist; each major laboratory will continue publishing its own framework and threshold vocabulary without a jointly agreed crosswalk between them, because harmonizing thresholds touches competitive and liability considerations that a voluntary crosswalk does not resolve. Disconfirmed if two or more frontier labs jointly publish an explicit equivalence mapping between their respective risk-tier vocabularies.

Four. Horizon: 2029. Aggregate single-scale capability indices will proliferate rather than converge on one accepted standard, because index designers serving different purposes — tracking research trends, informing procurement, triaging safety review — will keep choosing different benchmark baskets and weightings for those different purposes. Disconfirmed if the field converges on one dominant capability index used across all three purposes by then.

What to take away

Every comparison of frontier models currently in public circulation is a comparison at an unstated point along at least six separate axes of ambiguity: what construct was actually measured, whether the test set was clean, whether the platform’s own selection process shaped the result, what cost and latency point the numbers were taken at, which modalities were and were not covered, and whose internal safety taxonomy is being invoked. None of these six problems has a general, agreed solution today, and the honest response to that fact is not a cleaner single number — it is a discipline of stating, every time a comparison is offered, which construct, which operating point, and which unresolved uncertainty travels along with it. A comparison that cannot state those three things has not yet earned the authority a ranked list borrows by default.