A number is not a measurement
“Claude scored X on benchmark Y” reads like a fact about a model. It is closer to a fact about a harness. The score depends on which variant answered, what tools it had, how many attempts it was given, whether the task text had already appeared somewhere in its training data, who graded the transcript, and on what date the run happened. None of those five things travels with the number when someone repeats it in a slide deck or a comparison table, and each one can move the score by a margin that dwarfs the gap between models the number is supposedly ranking.
This article is not a scoreboard. It does not rank Claude against other model families, because the sources it draws on were not designed to support that ranking and using them for one would misrepresent them. It is instead an account of how Claude’s capability evaluations are actually built: what Anthropic’s own published methodology tests and why, how that testing is gated to a formal policy rather than run at the company’s discretion, what independent government and research bodies check from outside, where benchmark contamination quietly inflates a score before anyone runs it, and why the research groups closest to this work keep publishing warnings against collapsing their own results into one number.
The throughline is methodological, and it matters beyond Anthropic specifically: every frontier lab publishes capability numbers, and the same five hidden variables apply to all of them. Claude is simply the case with the most complete public paper trail — a Responsible Scaling Policy, dated system cards, a joint pre-deployment report from two national safety institutes, and an outside evaluation group that publishes its disagreements with the lab it evaluates. That paper trail is what makes it possible to show, concretely, what “a capability evaluation” actually consists of underneath the headline number.
What a system card actually tests
A Claude system card is not one test. Anthropic’s pre-deployment evaluation suite for a released model runs across several distinct categories: capability evaluations that measure skill on a range of tasks; safety and alignment evaluations covering reward hacking, sabotage capability, evaluation awareness, and model welfare; and a separate set of dangerous-capability evaluations mandated specifically by the company’s Responsible Scaling Policy [3]. These categories answer different questions and are not substitutes for one another — a model can score well on a capability benchmark while a safety evaluation in the same document reports a concerning behaviour at low but non-zero rates, and both facts belong in the same release.
The Opus 4.5 system card also makes explicit something that rarely survives into a headline: the specific configuration under which a score was produced. Evaluations there were run with a defined 64,000-token thinking budget, interleaved reasoning scratchpads, a 200,000-token context window, and default (high) reasoning effort — a fixed operating point, stated once, rather than left implicit [3]. Anthropic also reports actively working to reduce “evaluation awareness” — the degree to which a model can tell it is being tested and may behave differently as a result — by removing training data and tasks judged likely to increase it, describing the resulting effect as modest [3]. That single admission does more to explain why a system card is not a leaderboard entry than any disclaimer could: the company that writes the test is also actively trying to stop the subject of the test from recognising it as one, and reports that the fix only partly works.
Capability and safety evaluation are also more entangled than the category labels suggest. Agentic coding evaluations, for instance, are reported as capability results, but they feed directly into the autonomy assessments the Responsible Scaling Policy requires before a model can be released without additional safeguards [3]. A single evaluation run can therefore answer a marketing question and a governance question at once, using the same transcript.
The policy that decides when a threshold matters
None of this testing happens at the company’s unstructured discretion. Anthropic’s Responsible Scaling Policy defines a small number of “Capability Thresholds” — for example, an AI-research threshold framed around a model’s ability to compress roughly two years of 2018–2024-era AI research progress into a single year, and a CBRN threshold framed around whether a model could substantially uplift the weapons-development capabilities of a moderately resourced state programme — and ties crossing each one to a mandatory upgrade in safeguards, such as the ASL-3 security and deployment standards [1]. A model that trips one of these thresholds is not simply flagged in a report; specific, pre-committed technical and operational controls are required before it can be trained further or deployed at all.
The policy also governs its own testing cadence, and its own history of revising that cadence is instructive. Evaluation intervals were originally set at three months; Anthropic’s own compliance record for the policy describes extending that to six months specifically “to avoid lower-quality, rushed elicitation” — an explicit trade-off between measuring often and measuring well [1]. Models that cross the more serious AI-research threshold trigger an additional requirement: an “affirmative case” identifying the most immediate and relevant misalignment risks before proceeding, rather than a general safety sign-off [1]. To keep the whole arrangement legible to outsiders rather than self-certifying, Anthropic commits to publishing Risk Reports and Frontier Safety Roadmaps documenting how each model was assessed against the thresholds and what safeguards followed [1].
The upshot is that “Claude was evaluated for dangerous capabilities” is not one test either. It is a specific, dated, threshold-referenced procedure whose cadence, evidentiary bar, and required response were fixed by a public policy document before the model in question existed.
When the test has already been seen
Anthropic’s own methodological writing pre-dates most of this infrastructure and remains unusually candid about where it breaks down. A 2023 post from four members of Anthropic’s policy and alignment staff walks through concrete failure modes the company had hit while building its own evaluations, and it is worth treating as primary evidence about evaluation practice rather than as marketing [2]. On MMLU specifically, the authors describe four separate problems: models may have already encountered a given question during training; small formatting changes — spacing, parenthesis style — shift measured accuracy by roughly five points; different labs implement the same nominal benchmark inconsistently; and the benchmark itself contains mislabelled or unanswerable questions [2]. On BBQ, a bias-measurement benchmark, the authors report that correct implementation took one engineer a full week, and that a model which simply refuses to answer questions can score as though it holds no bias at all — a result the authors describe as easy to overinterpret as progress when it is really a refusal rate in disguise [2].
The first of those four problems — prior exposure — is now a distinct research area of its own, because it can inflate a score without anyone involved intending it to. A model does not need to have been deliberately shown a benchmark’s answer key; it is enough for the questions, or close paraphrases of them, to have appeared somewhere in a training corpus scraped from the public internet, where evaluation sets are routinely posted, discussed, and copied. A 2024 survey of the contamination literature frames the mechanism plainly: evaluation-benchmark material leaks into training data, the model partially memorises it, and the resulting score measures recall rather than the general capability the benchmark was built to test [11]. The same survey’s proposed remedy — moving toward dynamic, continually refreshed evaluation material rather than static, reusable test sets — is a direct response to the fact that any fixed, public benchmark has a shelf life before its answers begin leaking back into the training pipelines of the models it is meant to test [11].
The MMLU benchmark itself is a useful illustration of how a good-faith, carefully constructed test still becomes vulnerable to this over time. Hendrycks and colleagues built it specifically to be hard: 57 subjects spanning elementary mathematics to law, requiring “extensive world knowledge and problem solving ability,” at a time when the strongest available model improved on random chance by only about 20 percentage points on average, and the paper’s authors noted that “on every one of the 57 tasks, the best models still need substantial improvements before they can reach expert-level accuracy” [12]. That was a genuinely discriminating test in 2020. A benchmark’s difficulty is not a fixed property; it decays as its contents propagate through the training corpora of every model built after it was published, which is exactly the dynamic the contamination survey is describing at scale [11].
Measuring against a human clock
One response to a decaying, potentially contaminated benchmark is to stop asking a model to answer a fixed multiple-choice set at all, and instead measure something that is much harder to have memorised: how long a task takes. METR, an independent nonprofit evaluation group, has built its central capability metric around exactly this idea. The group defines a “time horizon” as the length of task — measured in the time a human domain expert would need — that a model can complete at a given success rate, most commonly 50% [7].
The method is a curve fit, not a lookup table, and stating it plainly exposes the assumption it rests on. For a task of human-expert duration
with
METR’s own published finding at the time of the paper’s release was that “current models have almost 100% success rate on tasks taking humans less than 4 minutes, but succeed less than 10% of the time on tasks taking more than around 4 hours,” with Claude 3.7 Sonnet’s 50%-success time horizon measured at roughly fifty minutes, and frontier time horizons across the field doubling on a roughly seven-month cycle since 2019 [7]. METR maintains a live public tracker of this metric across model families, and the tracker itself is a useful corrective to treating even this more careful metric as a settled ranking: it reports both 50% and 80% success horizons rather than one figure, flags that “measurements above 16 hrs are unreliable with our current task suite,” and states its own scope limits directly — the underlying tasks are drawn mostly from software engineering, machine learning, and cybersecurity work, are run in a “low-context” setting unlike a real job, and the tracker’s own frequently-asked-questions text asks and answers, in essentially these words, whether a long time horizon means a model can automate an entire job, and answers no [8]. A methodology built specifically to be harder to contaminate than a multiple-choice test still requires four caveats before its own authors will let it stand as a single number.
Who checks the checker
A lab’s own testing, however methodologically careful, is not independent verification, and Anthropic’s evaluation record includes several instances where outside parties checked the work directly rather than reviewing a summary of it after the fact.
In November 2024, the UK AI Safety Institute and the US AI Safety Institute conducted what both described as the first joint pre-deployment evaluation of a frontier model by the two national bodies, testing an upgraded Claude 3.5 Sonnet across four domains — biological capabilities, cyber capabilities, software and AI-development capabilities, and the practical effectiveness of Anthropic’s safeguards — using four distinct methodologies: graded question-answering, agent tasks run inside virtual environments, qualitative expert probing, and red-team attempts to jailbreak the model [6]. As part of their own methodology the two institutes compared the model’s performance against a small set of reference systems, including the model’s own prior version; that comparison was a design choice made by the evaluators for their own report and is described here as their methodology, not as a ranking this article is constructing. Their reported findings included that the model’s biological-capability performance matched their reference models and sat below expert-human baselines on its own, but sometimes exceeded that baseline when paired with bioinformatic tools; that it solved 32.5% of the public cyber challenges used by the US institute and 36% of “apprentice-level” tasks used by the UK institute; and that both institutes were able to routinely circumvent the model’s safeguards through jailbreaks, a finding they noted was consistent with what they had observed testing other AI systems [6].
A second, distinct form of outside checking involves risk assessments Anthropic authors itself. In October 2025, Anthropic published a pilot “Sabotage Risk Report,” explicitly framed as practice for the kind of report its Responsible Scaling Policy will eventually require on a regular basis, assessing whether Claude Opus 4 posed a risk of misaligned autonomous action contributing to catastrophic outcomes; the report’s own conclusion was that this risk was “very low, but not completely negligible” [5]. Rather than publishing that conclusion unchecked, Anthropic gave METR access to additional evidence beyond the public report — deployment information, evaluation results, and safeguard descriptions among them — specifically so METR could review the reasoning and evidence independently [5]. METR’s review broadly endorsed the risk level Anthropic reported, but the two organisations did not agree on everything: METR’s stated primary disagreement concerned the sensitivity of Anthropic’s alignment assessment, flagging a specific risk that its results could be weakened by the same evaluation-awareness problem Anthropic had separately reported trying, with only modest success, to reduce [5, 3]. That is a genuine, unresolved methodological disagreement between the lab and its outside reviewer, reported by both parties rather than smoothed into a single verdict, and it is a more informative fact about the state of frontier-safety evaluation than either organisation’s headline conclusion on its own.
A third channel is safeguard-specific rather than model-specific. In September 2025, Anthropic described US CAISI and UK AISI red-teamers stress-testing its Constitutional Classifiers safeguard system across Claude Opus 4 and 4.1, with access to pre-deployment prototypes, deliberately unprotected model variants, and internal safeguard documentation; the red-teamers found several categories of working bypass, including prompt injection that exploited false annotations, obfuscation-based universal jailbreaks, and automated attack-refinement pipelines, findings that Anthropic states led it to restructure part of its safeguard architecture [4]. Anthropic frames this kind of access as a deliberate policy rather than a one-off: working with independent outside experts to find vulnerabilities is described in the announcement as “a core part” of its safety approach [4].
Passing is not the same as working
Even a well-designed, decontaminated, expert-calibrated evaluation measures something narrower than deployed usefulness, and the clearest evidence for this gap comes from METR examining its own results a second time. In an August 2025 research update, the group re-examined coding tasks that a Claude 3.7 Sonnet-based agent had already passed by the normal grading standard — the automated test cases attached to each task — and had a human reviewer separately judge whether the resulting code was actually production-ready. The agent’s automated pass rate on the underlying tasks was 38%. On human review, none of the passing submissions were judged mergeable as submitted; the average estimated fix time for a “passing” submission was about 26 minutes, roughly a third of the original task’s duration, and the most common defects were inadequate test coverage, present in all of the passing runs reviewed, and missing or incorrect documentation and formatting problems, each present in about three-quarters of them [9].
METR was explicit that this finding has its own limits rather than presenting it as a final word: the repositories used carry unusually strict, mature-project quality bars that may exceed a typical codebase; the reviewed issues were selected partly for having extensive existing test coverage, which likely overrepresents well-scoped problems; and the agent scaffold used was deliberately basic rather than optimised, leaving open how much of the gap would close with better tooling [9]. Read together with the time-horizon result above, the finding sharpens rather than contradicts it: a task that a benchmark scores as completed and a task that a human would accept as finished are measurably different events, and the gap between them is itself something a research group had to run a second, separate evaluation to see.
Why one leaderboard number fails
The final failure mode is not about any single benchmark’s design but about what happens once many models are compared on the same public leaderboard, and it is best documented outside Anthropic’s own evaluations entirely, in an empirical study of Chatbot Arena — a widely cited human-preference leaderboard that includes Claude alongside models from many providers. Examining roughly two million battles across 243 models from 42 providers over sixteen months, the study’s authors identify systemic distortions built into how such leaderboards are populated [10].
Two of their findings matter most for evaluating what a leaderboard rank actually means. First, providers do not all have equal access to pre-release testing: the study reports that Meta tested roughly 27 private variants ahead of its Llama-4 release, and more generally that the two providers with the greatest data access received an estimated 19.2% and 20.4% of all Arena battle data respectively, while 83 combined open-weight models collectively received only 29.7% [10]. A provider that can preview many silent variants and quietly withhold the underperforming ones before choosing which to submit publicly is not competing on the same terms as one that submits once. Second, the leaderboard can be gamed in a way that does not generalise: the authors report that fine-tuning specifically on Arena-style prompts produced up to a 112% relative improvement on a benchmark correlated with Arena rankings, while producing no improvement — and occasionally a slight decline — on a general-knowledge benchmark like MMLU, which the authors interpret as overfitting to the leaderboard’s specific dynamics rather than genuine capability gain [10]. A model can be optimised to look better on the leaderboard without becoming more capable by any broader measure, and the leaderboard’s single ranking number cannot distinguish the two cases from the outside.
This is not a criticism specific to one leaderboard. It is the general form of every objection raised earlier in this article, restated at the point where scores get aggregated into a rank: unequal access to the testing process, and a metric that can be improved in ways that do not track the underlying thing it claims to summarise. METR’s own tracker draws the same conclusion about its own metric when it insists on reporting two thresholds rather than one and stating directly what a long time horizon does not imply [8]. The pattern across every methodology examined here — contamination-checked benchmarks, human-calibrated time horizons, algorithmic-versus-holistic code review, and preference leaderboards alike — is the same: the people closest to producing these numbers are the ones most consistently warning against reading a single one of them as a complete answer.
Predictions, with the observations that would falsify them
These are forecasts, clearly separated from the sourced analysis above. Horizon: 12 August 2029.
One. System cards will disclose more of the hidden evaluation configuration over time — reasoning-effort level, context window, sampling settings, and contamination-screening method — rather than less, because the alternative already reads as incomplete against the current disclosure baseline. Disconfirmed if major labs’ system cards in 2029 report headline capability numbers with less configuration detail than Anthropic’s 2025 cards provide.
Two. Independent, government-run pre-deployment evaluation of frontier models will become a standing arrangement covering most major releases rather than an occasional joint exercise. Disconfirmed if, by 2029, joint government pre-deployment testing of frontier models remains as infrequent as the single 2024 US–UK joint evaluation identified here.
Three. The gap between automated benchmark pass rates and human-judged deployment readiness, of the kind METR measured for coding tasks, will narrow for narrow, well-specified domains faster than it narrows for open-ended ones. Disconfirmed if published algorithmic-versus-holistic gaps in 2029 are as large in narrow coding domains as the 2025 measurement found.
Four. Preference leaderboards will move toward disclosed, audited data-access rules for all providers rather than continuing with undisclosed private pre-testing. Disconfirmed if a 2029 successor to the Leaderboard Illusion study finds comparable or worse data-access asymmetry than the 2025 study reported.
None of these predictions requires a capability discontinuity. They follow from a trend already visible in every source examined above: evaluators, including Anthropic’s own alignment staff, keep publishing the specific ways their own numbers can mislead, and each such disclosure tends to become a baseline the next report is measured against.
What to take away
A Claude capability evaluation is a compound object: a category of test (capability, safety, or RSP-mandated dangerous-capability), a fixed operating configuration, a contamination-screening step that is never perfectly clean, a comparison against a human baseline that itself required careful calibration, and — for the evaluations that matter most — an outside reviewer whose disagreements are published alongside the headline conclusion rather than absorbed into it. Every one of those components can shift a reported score without the underlying model changing at all.
Treat a single benchmark number the way this article’s sources treat their own: as a data point requiring the configuration, the date, and the acknowledged limitations that produced it, never as a self-sufficient verdict. Ask which evaluation category produced the number, whether the material could have been seen in training, who checked it from outside, and what the same research group has said elsewhere about that number’s limits. If those four answers are not available, the number in front of you is not yet a measurement.
