What Open Weights Actually Let You Verify

Two ways of being told a number

When a vendor publishes a benchmark score for a model served only behind an API, an outside party can interrogate the claim from exactly one angle: send prompts, read outputs, and infer. The weights that produced the answers are not available to inspect, hash, or rerun; the serving stack behind the endpoint can change on a date nobody announces; and a model name can silently refer to a different checkpoint next month. Verification, in that setting, means statistical inference about a moving target from the outside.

Open weights change what is available to check, though not in the way the marketing around “open” usually implies. They do not make a benchmark claim self-evidently true. What they do is narrow the space of possible explanations for a discrepancy, because one entire source of disagreement — is this actually the model in question — becomes a fact that can be settled with a hash comparison rather than a fact that has to be taken on trust. Once the weights themselves stop being a variable, whatever gap remains between a vendor’s reported number and an independent lab’s reproduced number has to live somewhere else: in the evaluation harness, the prompt format, the decoding settings, or the training data the weights were fitted to in the first place.

The Llama release history is an unusually complete record of this mechanism actually operating — cases where the gap was chased down and closed, and at least one case where it was chased down and found to be exactly what it looked like. This article works through that record, then asks what independent verification still cannot reach even with the weights sitting on a local drive.

What a reproduction actually holds constant

Treat a published benchmark score as the output of a function of two arguments: the weights being scored, and the procedure used to score them.

Here $\theta$ denotes the model’s parameters and $H$ denotes everything about the evaluation procedure that is not the model itself — the exact prompt template, the number of few-shot examples, whether a chat template is applied, how the model’s free-text output is parsed into a scored answer, and the decoding configuration. A vendor publishes one value of $s$. An independent lab computing its own $s'$ is implicitly testing whether $f(\theta, H) = f(\theta', H')$, and a mismatch could in principle come from either argument.

With a closed, API-served model, both arguments are partly opaque: $\theta$ is asserted rather than shown, and $\theta'$ — whatever the endpoint is actually running today — cannot be confirmed to equal $\theta$. A reproduction attempt is chasing two unknowns with one equation. With an open-weight release, $\theta$ is a downloadable file, and a downloaded file can be checksummed against the one a vendor states it evaluated. That does not resolve the equation, but it removes a whole term from it: if the checksums match, $\theta = \theta'$ is no longer a hypothesis, it is a verified fact, and any gap between $s$ and $s'$ has to be attributed to $H \neq H'$. That is a strictly smaller, more tractable problem — and, as the next three sections show, it is a problem the field has repeatedly been able to solve, which is itself informative about how much of a published number is model and how much is procedure.

The thirty-point gap that was never about the model

The clearest illustration predates the current benchmark-fudging controversies and involves no allegation of misconduct at all. When Hugging Face’s team investigated why models on the original Open LLM Leaderboard did not match their own papers’ reported numbers, they focused on MMLU and the original LLaMA family as the test case. The original implementation, evaluated with topic context supplied in the prompt and comparing the model’s probability mass over the four answer letters, scored one 65-billion-parameter LLaMA checkpoint at 0.637 on MMLU. The EleutherAI evaluation harness, running the same weights with a different prompt structure — dropping the topic line, adding “Question:” and “Choices:” labels, and computing log-likelihood over full answer strings rather than single letters — scored the identical checkpoint at 0.488, a roughly 30 percent relative difference on a benchmark whose name and headline number were, on paper, the same thing ^[12].

Stanford’s Holistic Evaluation of Language Models project generalized the finding rather than treating it as one dataset’s idiosyncrasy: prior to HELM’s standardization effort, models being compared in the literature had on average been evaluated on only 17.9 percent of the same core scenarios as each other, so headline rankings across papers were frequently comparing different tests wearing the same benchmark name. HELM’s contribution was to force dense, standardized coverage — 96.0 percent of core scenarios shared across 30 models under one harness — specifically so that a reported gap between two models could be attributed to the models rather than to which subset of a benchmark suite each one happened to have been run against ^[5]. Read together with the MMLU case, the lesson is not that any particular number was fabricated. It is that “the MMLU score” was never a single well-defined quantity, and open weights are what let an outside party discover that empirically rather than take it on faith — because the same $\theta$ could be run under both procedures and directly compared.

A number that came back once the procedure matched

Meta’s own Llama 3.1 model card reports GSM8K performance of 84.5 for the 8-billion-parameter instruction-tuned model, using an 8-shot chain-of-thought configuration matching the format established by Wei and colleagues, and states plainly that the company also built and released “an eval reproduction recipe that demonstrates how to closely reproduce the Llama 3.1 reported benchmark numbers using the lm-evaluation-harness library” ^[2], with the underlying evaluation transcripts published as a Hugging Face collection ^[1]. That harness — EleutherAI’s lm-evaluation-harness — is itself open infrastructure, cited in hundreds of papers and used as the backend for several public leaderboards precisely because it fixes $H$ as a shared, versioned, publicly inspectable artifact rather than a private in-house script ^[4].

The offer to reproduce was tested directly. When an independent evaluator ran Llama-3.1-8B-Instruct through the harness using what looked like the natural default settings, GSM8K came back at approximately 76.95 — nearly eight points below Meta’s reported 84.5, a gap large enough to read, on first glance, as an inflated vendor claim ^[13]. It was not. The discussion thread on the model’s own Hugging Face repository traces the resolution: the default run had not applied the chat template the instruct model expects, and had not treated the few-shot examples as multi-turn exchanges the way Meta’s internal harness configuration does. Once the evaluator added the --apply_chat_template and --fewshot_as_multiturn flags — matching, rather than merely approximating, Meta’s documented procedure — the reproduced score rose to roughly 85.4, within a fraction of a point of the published number, using a community-contributed harness configuration file built specifically to align with Meta’s methodology ^[13].

That is what a working reproduction ecosystem looks like when it functions as designed: a public claim, a public harness, a visible discrepancy, and a traceable resolution that ends with $H = H'$ rather than a shrug. No hash comparison was needed here because nobody doubted the weights; the entire dispute and its resolution lived inside the harness argument, exactly where the earlier equation says it should when $\theta$ is fixed and known.

The variant that was never for sale

The Llama 4 release in April 2025 produced the opposite outcome, and it is worth being precise about what, specifically, independent scrutiny caught — because the mechanism is different from a harness-flag disagreement, and it depended on the same underlying fact: an outsider could get hold of the actual released weights and run them.

Meta launched Llama 4 Scout and Maverick on April 5, 2025, and a checkpoint labeled “Llama-4-Maverick-03-26-Experimental” briefly ranked second on LMArena’s crowdsourced human-preference leaderboard, trailing only Gemini 2.5 Pro. Developers comparing that leaderboard-topping variant against the model available for download noticed it behaved differently — longer, more heavily formatted responses laced with emoji, a style known to score well with human raters — from the plain “Llama-4-Maverick-17B-128E-Instruct” that anyone could actually deploy ^[8]. Meta confirmed the experimental variant was real: a spokesperson described it as “a chat optimized version we experimented with that also performs well on LMArena,” distinct from the public release ^[8]. LMArena’s own account was less neutral. The platform stated that “Meta’s interpretation of our policy did not match what we expect from model providers,” and announced it would tighten its submission rules to require clearer disclosure when a variant tuned for one evaluation differs from what ships ^[8].

The concrete number is what makes this a case about verification rather than a stylistic quibble. Once LMArena added the actual public release to its board and ran it through the identical crowdsourced comparison used for the experimental variant, the vanilla Llama-4-Maverick-17B-128E-Instruct settled at 32nd place — below OpenAI’s GPT-4o, Anthropic’s Claude 3.5 Sonnet, and Google’s Gemini 1.5 Pro, all models that were, by April 2025, already a generation or more behind the frontier ^[9]. The gap between second place and thirty-second place is not the kind of thing an evaluation-harness flag explains. It is the kind of thing that only becomes checkable at all because the actual released $\theta$ could be independently obtained and run through the same public arena the showcased variant had been run through — an outsider held the released weights, and holding them was what made “which model actually scored this” answerable rather than assumed.

Meta did not fully concede the point at the time, framing the episode as legitimate experimentation with custom variants. Nine months later, in January 2026, Meta’s outgoing chief AI scientist Yann LeCun told the Financial Times that the published Llama 4 benchmark results “were fudged a little bit” — describing a practice in which the team evaluated multiple internal variants of Maverick and Scout, selected whichever variant scored highest on each individual benchmark, and reported those selected scores as though they described one consistent model, rather than the standard practice of evaluating a single fixed checkpoint across an entire suite ^[10]. That account should be read for what it is: a first-person recollection from a departing insider, reported by one outlet, describing internal decisions this article’s sources cannot independently audit from the outside. It is not of the same evidentiary kind as the LMArena ranking, which is a directly observable, independently reproduced measurement. Both belong in the record, and they should not be blended into one fact.

What independent evaluators found once the dust settled

A separate line of independent evaluation, run by the benchmarking group Artificial Analysis outside the LMArena controversy entirely, illustrates a second and more mundane way a vendor number and an independent number can diverge even with identical weights and good faith on both sides. Their initial pass at Llama 4 Scout and Maverick produced an aggregate “Intelligence Index” of 36 and 49 respectively — placing Maverick ahead of Claude 3.7 Sonnet but behind DeepSeek V3 — while their multiple-choice benchmark results, on tests like MMLU Pro and GPQA Diamond, came in materially below the numbers in Meta’s own model card ^{[11, 3]}.

The cause, once diagnosed, was procedural rather than substantive: Scout and Maverick frequently answered multiple-choice questions in a free-text format — “The best answer is A” — that the evaluator’s automated scoring script did not initially recognize as a valid answer, so correct responses were being marked wrong purely on parsing grounds. Once Artificial Analysis revised its scoring to accept that answer format, Scout’s Intelligence Index rose from 36 to 43 and Maverick’s from 49 to 50, closing most of the gap to Meta’s claims without any change to the weights, the prompts, or the underlying capability being measured ^[11]. This is the harness-flag story from the GSM8K section again, playing out in public with a commercial evaluator rather than a community forum thread — and it argues for the same conclusion: a raw discrepancy between a vendor’s number and an independent number is a starting point for investigation, not a verdict, and open weights are precisely what make the investigation possible instead of speculative.

Not every discrepancy in that same evaluation round resolved so cleanly. Independent long-context testing found Llama 4 Maverick’s accuracy falling to roughly 28 percent at 120,000 tokens of context and Scout to roughly 16 percent, despite Scout’s headline specification advertising a 10-million-token context window ^[11]. That gap is not a parsing artifact; it is a documented case where independent testing and vendor specification simply disagreed about a claimed capability, with no procedural reconciliation on record. Both outcomes belong in an honest account of what verification found: some gaps are the evaluation’s fault, and some are not.

What open weights do not hand you

None of the above should be read as open weights solving evaluation transparency generally. Three separate limits survive even a fully successful weight-level reproduction.

The first is that Meta’s own headline numbers are not produced by the public harness at all. The Llama 3.1 model card states directly that “for all the evaluations, we use our internal evaluations library,” with the lm-evaluation-harness recipe offered afterward as an approximate, independently reproducible path to similar numbers rather than as the tool that generated the original figures ^[2]. The precise internal prompts, exact scoring code, and any preprocessing applied to model outputs before scoring are not published. What is published is close enough that independent parties have been able to land within a point of the headline numbers, as the GSM8K case showed — but “close enough to reproduce with effort” and “fully disclosed” are different claims, and only the weaker one is actually true here.

The second is contamination, and it does not respect the open/closed boundary at all. A benchmark question can leak into a training corpus regardless of whether the resulting model’s weights are later released openly, because the leak happens upstream, in the data collection pipeline, not in the release decision. Hugging Face’s own leaderboard history is direct evidence of the scale of the problem: the original six-benchmark Open LLM Leaderboard was retired in 2024 specifically because years of public benchmark text circulating on the open web had been absorbed into training corpora broadly enough that top models were approaching scores that looked like memorization rather than generalization, and the leaderboard was rebuilt around six harder, less-exposed benchmarks — IFEval, BBH, MATH, GPQA, MUSR, and MMLU-Pro — for exactly that reason. A dedicated statistical contamination detector, ConStat, was applied directly to Open LLM Leaderboard entries and reported “high levels of contamination in multiple popular models including Mistral, Llama, Yi, and the top-3 Open LLM Leaderboard models” of its evaluation round, using a method built to distinguish genuine capability from benchmark-specific memorization rather than merely checking for verbatim string overlap ^[7]. A broader survey of the contamination literature confirms this is now a structural feature of the field rather than an isolated finding: detection methods split into white-box approaches requiring access to token-level probabilities, gray-box approaches using partial signals, and black-box approaches limited to model outputs, and all three families exist because no single check reliably rules contamination out ^[6]. Weights being open tells you nothing about whether the data that produced them included the test.

The third limit is that Meta has not released Llama’s pretraining corpus, so an independent party cannot run a direct overlap check between Llama’s training data and any given benchmark even with the weights fully in hand — they can only run indirect statistical detectors of the kind ConStat represents, which infer contamination from performance patterns rather than observe it directly in the data. That is a materially weaker position than a release that publishes both weights and training data together would put an evaluator in, and it means “the weights are open” and “the training process is auditable for contamination” are two different claims that happen to get compressed into the same word.

The ecosystem that makes checking routine

None of the individual reproductions above happened because one motivated engineer decided to spend a weekend on it. Each depended on standing infrastructure that exists specifically to make this kind of check repeatable rather than a one-off act of diligence.

The lm-evaluation-harness is the clearest example: a single, versioned, open-source implementation of dozens of benchmark tasks, used as the backend for public leaderboards and cited across the field, whose entire value proposition is that two different labs using the same version and the same task configuration are running the same $H$, so any gap left over is attributable rather than mysterious ^[4]. HELM performs a related but distinct function: rather than fixing one harness, it forces broad, standardized coverage across many benchmarks and many models at once, so that a model cannot be evaluated on a favorable subset of scenarios while a competitor is evaluated on a different, harder subset and the two get compared as though they took the same test ^[5].

Private, held-out evaluation sets address the contamination problem from the opposite direction, by design rather than by chasing leaks after the fact. Scale AI’s SEAL leaderboards score models against curated question sets that are never published, specifically so a benchmark’s questions cannot circulate on the web and be absorbed into some future model’s training corpus — trading the transparency of an openly published test set for the durability of one that cannot leak because it was never exposed in the first place ^[14]. That is a direct, structural answer to the contamination limit described above, though it comes at its own cost: a private benchmark cannot itself be independently reproduced by a third party, only trusted or distrusted as an institution, which is a different kind of verification problem than the weight-level one this article has mostly been about.

LMArena’s course correction after the Maverick episode belongs in the same category, imperfect as it was. A crowdsourced human-preference arena is, in principle, one of the harder benchmarks to game with a static answer key, precisely because there is no fixed test set to contaminate — but it is not immune to a vendor submitting a variant tuned specifically to win human approval rather than representative of what ships. The fact that the mismatch was caught, named publicly, and used to tighten submission rules is evidence the check functioned, not evidence it is unnecessary; a benchmark that can be gamed and was not caught would look identical from the outside to one that was never gamed at all.

Reading a benchmark claim after all this

Put the pieces together and a usable discipline falls out, and it is worth stating plainly because it differs from “trust the leaderboard” and from “distrust every vendor number” in equal measure.

A vendor-reported score is a claim, not a fact, regardless of whether the model behind it is open or closed — that much does not change. What open weights add is a specific, bounded capability: the ability to fix $\theta$ as a checked, shared constant and then interrogate $H$ until a gap is either explained or confirmed. The GSM8K and Artificial Analysis cases show that capability working exactly as intended, turning an apparent discrepancy into a diagnosed and closed procedural gap. The Llama 4 Maverick case shows what happens when the gap is not procedural: the same capability that resolves harness mismatches also has nowhere left to hide a genuinely different model, which is precisely why that episode ended in a public correction rather than an unresolved dispute.

What open weights do not add is a guarantee about the evaluation set, the training data, or the internal tooling that produced the original number — those are separate disclosures, independently withheld or granted, and a release can be maximally open on weights while remaining opaque on all three. Reading a Llama benchmark claim carefully means asking four separate questions rather than one: is this the checkpoint that was actually scored, was it scored with a disclosed and matchable procedure, could the relevant benchmark plausibly have leaked into training, and was the evaluation set itself ever exposed to the open web. Only the first two of those four are what weight access actually settles.

Predictions, with the observations that would falsify them

These are forecasts, separated from the sourced analysis above. Horizon: August 2029.

One. Vendor model cards for major open-weight releases will increasingly publish exact evaluation-harness configuration files rather than prose descriptions of methodology, because the GSM8K-style reconciliation process demonstrated here is cheaper to run when the configuration is a file rather than a paragraph to reverse-engineer. Disconfirmed if major releasers in 2029 still describe evaluation procedure only in narrative form without a runnable configuration artifact.

Two. Crowdsourced preference arenas will adopt mandatory identity verification for submitted checkpoints, of the kind a hash comparison already provides for weight-level benchmarks, given that the Maverick episode’s core defect was that a submission’s provenance could not be checked at the time. Disconfirmed if leading arenas in 2029 still accept anonymous or unverified checkpoint submissions without provenance attestation.

Three. The gap between statistically inferred contamination detection and direct training-data audits will remain open for frontier open-weight models, because releasing full pretraining corpora carries copyright and competitive costs most vendors will continue to decline even when they release weights freely. Disconfirmed if a majority of major open-weight releases by 2029 include full pretraining corpus disclosure alongside weights.

What “open” actually settles

Open weights turn one specific kind of dispute — is this the model — from a matter of trust into a matter of arithmetic, and the Llama record shows that conversion mattering in both directions: it closed a genuine discrepancy in one case and exposed a genuine one in another. That is a real, checkable gain over a closed model’s benchmark claim, and it is also the entire gain. Everything downstream of “these are the correct weights” — the procedure that scored them, the data that trained them, the question of whether the test had already been seen — remains exactly as open or closed as the vendor separately chooses to make it, and no amount of weight access substitutes for asking about each of those directly.