From Origins to Frontier: A History of Frontier AI Model Comparisons

A comparison is a record, not a fact

Every method described in this article was built to answer the same question — is one system better than another, and at what — and every one of them was eventually judged inadequate and replaced. That pattern is the actual subject here. “How do we compare frontier models” is not a solved methodological question with a current best answer; it is the latest position in a sequence of positions, each occupied because the previous one had failed in a specific, documented way. Treating today’s leaderboards and arenas as a stable endpoint misreads the history that produced them.

This account starts before language models existed, because the shared-task infrastructure they inherited — a fixed test set, a pooled human judgment, a scoreboard — was built for a different problem first, and its assumptions still shape what a “benchmark” is taken to mean.

Before a leaderboard existed: pooled judgment at NIST

The Text REtrieval Conference began in November 1992, co-sponsored by the National Institute of Standards and Technology and the Defense Advanced Research Programs Agency, to address two gaps at once: no standardized evaluation existed across information-retrieval research groups, and no test collection was large enough to be realistic. Twenty-five participating groups worked against roughly two gigabytes of newspaper, newswire, and government text, and the organizers introduced pooled relevance judgment — human assessors judging a merged pool of each system’s top-ranked documents, since no one could judge every document against every query ^[1]. Systems were scored on the resulting judgments using recall and precision, the two quantities every later single-number metric would try to compress into one figure.

The most durable compression is the F-measure, still standard in information extraction and structured-prediction evaluation:

the harmonic mean of precision $P$ and recall $R$. The harmonic mean, rather than the arithmetic mean, was the substantive choice: it penalizes a system that trades one quantity away for the other, so a system cannot inflate its score by returning everything (maximizing recall while destroying precision) or almost nothing (the reverse). That is a real methodological assumption — that both errors matter and neither should be free — and it is worth stating plainly because later single-number benchmarks inherited the habit of compression without always inheriting this discipline about what the compression should punish.

Comparison research through the 1990s ran on this shared-task model: NIST or a similar body fixed a test collection and a scoring procedure, and participating systems were compared against a common gold standard rather than against each other directly. Statistical parsing evaluation used the Penn Treebank in the same spirit — a hand-annotated set of bracketed sentence structures against which a system’s output could be checked line by line. The unit of comparison was intrinsic and structural: did this system’s parse match this annotation, not did a rater prefer this system’s output in general. That distinction — comparing structured output against a fixed key, rather than comparing free-form output against a rater’s preference — is the axis this whole history moves along, and it moves along it exactly once, from one end to the other, for a reason that only becomes visible several sections later.

GLUE: one number, and how fast it stopped working

Large neural language models changed what needed comparing. A single pretrained representation could, after light fine-tuning, be applied to many distinct tasks — sentiment, entailment, similarity, question answering — and a fair comparison across research groups needed a suite that held the tasks fixed while letting the underlying representation vary. The General Language Understanding Evaluation benchmark, introduced in 2018, assembled nine such tasks behind one public leaderboard and one aggregate score, explicitly framed as encouraging general-purpose learning rather than narrow, task-specific engineering ^[3].

GLUE succeeded at its stated goal almost immediately, which created a new problem. Within roughly a year, models were matching or exceeding the human baselines the authors had established for comparison, and the aggregate score compressed toward a ceiling with little room left to distinguish genuinely stronger systems from noise. The authors’ own response was direct: they built a successor, SuperGLUE, explicitly because “GLUE has outlived its usefulness as a metric for progress,” replacing several of the original tasks with harder ones and adding new task types the original suite had not covered ^[4]. This is the first clean instance of a pattern that recurs through the rest of this history — a benchmark’s success, measured by models converging on its ceiling, is also the event that ends its useful life as a comparison instrument. A benchmark does not fail by being too hard. It fails by being solved.

Restoring headroom: broad multi-task suites

SuperGLUE bought time, but the underlying pressure — models improving faster than any fixed nine- or ten-task suite could measure — did not go away, and the response to it changed shape rather than just difficulty. Instead of harder versions of the same kind of task, two efforts in 2020–2022 changed the axis entirely, asking not “can this model perform this kind of language task well” but “how much of recorded human knowledge does this model actually have.”

Measuring Massive Multitask Language Understanding assembled 57 subjects spanning elementary mathematics to professional law and medicine, scored as multiple-choice questions, explicitly designed to test breadth of acquired knowledge rather than depth on any one linguistic skill; the paper reported that most contemporaneous models performed near random chance while the largest model tested showed a meaningful gap above it, and that even the strongest model fell well short of expert-level accuracy across the full subject range, with especially poor calibration on ethically loaded subjects like law and morality ^[5]. Beyond the Imitation Game pushed further still, assembling 204 tasks contributed by more than 450 authors across over 130 institutions specifically to probe for capabilities existing suites might be missing; its authors reported that most tasks improved smoothly with scale while a smaller number showed discontinuous, breakthrough-like jumps once models crossed a size threshold, and that social bias in ambiguous contexts tended to increase with scale unless specifically counteracted ^[6].

Both suites solved the immediate headroom problem. Both also reintroduced, at larger scale, the aggregation problem the harmonic mean had been built to manage in a narrower setting: a single reported average across dozens or hundreds of heterogeneous tasks tells a reader almost nothing about where a model is strong and where it is weak, unless the per-task breakdown is examined directly. Holistic Evaluation of Language Models later made this critique explicit and structural, reporting that the LM evaluation literature it surveyed had, on average, evaluated any given pair of models on only about 17.9 percent of the same scenario-and-metric combinations, and proposing dense, standardized evaluation across a fixed set of scenarios and multiple metrics — accuracy, but also calibration, robustness, fairness, bias, toxicity, and efficiency — precisely so that “which model is better” would stop being answerable by a single number at all ^[10]. That is an analytical claim about method, not a ranking of any specific vendor’s models, and it is the strongest institutional statement in this history against exactly the kind of single-axis cross-model ranking this article is obligated not to construct.

Documenting what was measured, and on whom

A parallel strand of this history is not about what gets measured but about what gets disclosed alongside the measurement. Model Cards for Model Reporting, published in 2019, proposed a short, standardized document to accompany a released model: intended use, out-of-scope use, and — centrally for the history of comparison — benchmarked performance broken out across relevant demographic and operating conditions rather than collapsed into one headline number, illustrated in the original paper with a smiling-detection model and a toxicity classifier scored separately across subgroups ^[7]. The proposal mattered less as a specific document template than as an institutional norm: that a performance number without a stated population, condition, and limitation attached is an incomplete claim. The system cards that accompany today’s frontier model releases are the direct descendant of that norm, and every specific number a vendor publishes in one — a score on a named benchmark, at a named date, under named conditions — is a vendor assertion in the strict sense used throughout this publication: a claim made by the party being evaluated, worth recording, and not equivalent to an independent measurement until someone outside that vendor reproduces it.

The contamination crisis

The broad suites of the early 2020s shared one design assumption with GLUE and SuperGLUE before them: that a benchmark’s questions and answers, once published, would function as a held-out test set indefinitely. That assumption broke once web-scale pretraining corpora began to include the same public internet the benchmark questions had been published to. A model could score well on a benchmark not because it reasoned well but because a paraphrase, or an exact copy, of the question and its answer had appeared somewhere in its training data.

The clearest statement of the resulting methodological crisis is a 2023 position paper arguing that data contamination causes a systematic overestimation of a model’s real capability relative to an uncontaminated model, that the scale of modern training corpora makes contamination difficult to rule out by inspection alone, and that the field needed standardized, per-benchmark contamination measurement — automatic where possible, semi-automatic where not — rather than an occasional footnote ^[8]. The practical detection methods that paper surveys are mostly variations on the same idea: check for verbatim or near-verbatim overlap, at the level of matching word sequences, between a benchmark’s published text and a sample of the training corpus, and treat a high overlap rate as grounds to discount the corresponding score. This is a real methodological gap between vendor assertion and independent verification: a lab that has not audited its own training corpus against a public benchmark cannot, by construction, rule out contamination on that benchmark, no matter how the resulting score is reported.

Two structural responses followed. One was to build benchmarks explicitly resistant to casual leakage — GPQA assembled 448 multiple-choice science questions difficult enough that PhD-holding domain experts working within their own specialty reached only 65 percent accuracy, and skilled non-experts with unrestricted internet access for over 30 minutes reached only 34 percent, a difficulty bar intended to make the benchmark useful for what the authors call scalable-oversight research even as general capability rises ^[9]. The other, older response, restated with new urgency, was Chollet’s argument that benchmarks scored by accuracy on a fixed, learnable task distribution measure “skill,” not “intelligence” in any transferable sense, and that a benchmark meant to measure generalization has to be built from tasks that cannot be prepared for by memorizing prior examples, however many are supplied — the design principle behind the Abstraction and Reasoning Corpus he introduced specifically to separate the two ^[2]. Contamination resistance and generalization-testing are not the same design goal, but the contamination crisis is what forced both into mainstream benchmark design at once, several years after either argument was first made.

When the answer stopped being checkable: the rise of the judge

Every mechanism up to this point compared a system’s output to a fixed key: a relevance judgment, a bracketed parse, a multiple-choice answer. That comparison stops working once a model’s typical output is an open-ended paragraph, a multi-turn conversation, or working code, none of which has a single correct string to check against. Exact-match scoring cannot tell a helpful, well-reasoned answer from an unhelpful one when both are grammatical and neither matches a reference string.

The response, arriving in 2023, was to stop checking against a fixed answer and start recording a preference between two outputs instead. Chatbot Arena built an open platform in which users submit a prompt, receive responses from two anonymized models side by side, and vote for the one they prefer, without knowing which model produced which response; its authors reported collecting more than 240,000 such votes from a broad user base and validated that crowdsourced preferences agreed well with those of a separate panel of expert raters ^[12]. Pairwise votes of this kind are naturally converted into a single continuous rating using a Bradley–Terry-style model — the same functional form used for chess Elo ratings — in which the probability that a rater prefers system $A$ over system $B$ is modeled as a logistic function of the difference between two scalar ratings:

That formula encodes a specific, testable assumption: that pairwise preference between any two systems can be explained by each system’s position on one shared scale, and that preferences are therefore transitive — if raters prefer $A$ to $B$ and $B$ to $C$, they will prefer $A$ to $C$. This is a real point of methodological disagreement rather than settled fact. Preferences over open-ended text plausibly depend on the task, the rater, and the specific pairing in ways a single scalar rating cannot capture, and a persistently observed intransitive preference cycle would be direct evidence against the model, not just noise around it. The Arena’s own published methodology treats this as an ongoing statistical concern to be monitored, not a solved problem.

A parallel line of work replaced the human rater with another language model. MT-Bench used multi-turn questions scored by a strong LLM acting as judge, and its authors reported that GPT-4 acting as judge agreed with human preferences over 80 percent of the time, comparable to the rate of agreement between two independent human raters — while explicitly documenting the failure modes that agreement rate was masking, including position bias, where the judge favors whichever response is shown first or second regardless of content, and a documented preference for longer, more verbose answers independent of their quality ^[11]. AlpacaEval adopted the same automatic-judge approach for rapid model comparison, and by 2024 its maintainers had confirmed the verbosity problem was severe enough to require a structural fix: Length-Controlled AlpacaEval explicitly models and removes the effect of response length on the judge’s preference, reporting that doing so raised the benchmark’s correlation with Chatbot Arena’s human-preference rankings from a Spearman correlation of 0.94 to 0.98 ^[13].

A further, more specific bias followed once judges and the systems they judged began to be drawn from the same small set of frontier model families. A 2024 study found that language models acting as evaluators could often distinguish their own generations from a human’s or another model’s, and that the strength of a model’s self-preference bias — its tendency to rate its own output more favorably — correlated linearly with how well it could recognize that output as its own ^[14]. That is a mechanism-level finding, not a claim about any one vendor’s current products: it says that using a frontier model to judge a field that includes its own outputs, or close relatives of them, introduces a specific, measurable distortion that a naive reading of the judge’s score will not reveal.

Where this leaves comparison today

Restated end to end, the sequence is not a series of improvements converging on a correct method. It is a series of substitutions, each one solving the specific failure of its predecessor while introducing a new failure of its own: pooled human judgment against a fixed key, until the tasks worth judging stopped being small and enumerable; a single aggregate leaderboard score, until models saturated it; broad multi-task suites restoring headroom, until their sheer breadth made the aggregate number opaque and their public text became trainable data; contamination audits and harder held-out suites, restoring a trustworthy floor under narrower conditions; and, for the open-ended outputs none of the above could score at all, human-preference arenas and LLM judges, which introduced position bias, verbosity bias, and self-preference bias in place of the problems they solved.

No stage in this history supports building a single ranked list of “the best frontier model” from whatever benchmark happens to be fashionable at the time it is consulted. Every mechanism above was built for a specific kind of claim, under specific conditions, and every mechanism above was later shown to admit a specific kind of distortion when used outside those conditions or past its own useful life. A score reported without its benchmark’s contamination status, its judge’s identity, and its suite’s aggregation method is not a fact about relative capability. It is, at best, a record — exactly the kind of record the scoring desk at the start of this history produced, on paper, in 1992, before either “leaderboard” or “arena” existed as a word for what it was doing.

Predictions, and what would falsify them

These are forecasts, separated explicitly from the sourced history above. Horizon: August 2029.

One. Contamination-resistant design — dynamically refreshed question sets, held-out private splits, or tasks requiring live tool use that cannot be memorized in advance — will become the default expectation for any newly introduced capability benchmark, rather than an optional extra a minority of benchmark authors adopt. Disconfirmed if a majority of widely cited new benchmarks introduced after this date still ship as static, fully public question-and-answer sets with no contamination controls.

Two. Reported LLM-as-judge results will routinely disclose the judge model’s family relationship to the models under evaluation, given the documented self-preference mechanism. Disconfirmed if, in 2029, leading leaderboards using model judges still omit which specific model served as judge.

Three. No single scalar leaderboard will emerge as the field’s accepted authority the way GLUE briefly was for its narrow moment; multi-metric, multi-scenario reporting in the style HELM proposed will remain the norm for serious comparison, precisely because the aggregation problem this article traces has no one-line fix. Disconfirmed if a single-number public leaderboard becomes the primary citation used to justify frontier procurement decisions across the industry, the way one general aggregate score arguably did in 2018–2019.

Four. Human-preference arenas will persist as a distinct, non-substitutable evaluation channel rather than being fully replaced by automatic judges, because the intransitivity and rater-dependence concerns raised in this article about the Bradley–Terry model are unlikely to be fully resolved by better automatic judging alone. Disconfirmed if the major human-preference arenas active today are discontinued in favor of automatic-judge-only leaderboards with no active human-vote channel.

What to take away

Frontier model comparison has never been a fixed method quietly getting more accurate. It has been a sequence of specific instruments, each built to fix a specific, documented failure of the one before it, and each eventually shown to have a specific failure of its own — saturation, opacity, contamination, or judge bias. Reading a comparison today without asking which instrument produced it, under what conditions, and what that instrument is already known to get wrong is the same mistake the field has made and corrected roughly half a dozen times since 1992. The corrections are the actual history. The number on any given leaderboard, on any given day, is only ever the latest position in it.