The Hardest Unsolved Problems in Open-Weight AI

Four problems that have outlasted every fix proposed for them

“Open-weight AI is risky” and “open-weight AI is safe” are not the claims examined here. Both are values-laden summaries of a much longer argument, and reasonable people weigh the same evidence differently. This article asks a narrower question: which specific technical problems in open-weight release remain unsolved, in the plain sense that a proposed fix has been tried and has a documented failure mode, or that the people building the mitigation say so themselves?

Four qualify. A released checkpoint cannot be recalled, and the safety behaviour installed on top of it can be removed cheaply and permanently by anyone holding a copy — not a hypothetical, but a demonstrated bench procedure. Releases marketed as open routinely withhold the training data and exact recipe needed to reproduce them, a gap the open-source research community’s own audits keep finding rather than closing. Benchmark scores can be inflated by training on rephrased or reinforcement-learned variants of test data in ways current detectors do not reliably catch, a problem serious enough that a widely used leaderboard shut down rather than keep publishing numbers it no longer trusted. And the tension between open access and dangerous-capability proliferation has no agreed way to measure which risk is larger, a gap the most recent international expert synthesis explicitly declines to close.

None of these four is a debate about values. Each is a problem where a fix was proposed, tried, and documented not to fully work — which is the operational meaning of “unsolved” this article uses throughout.

The permanence problem

Software has always had irreversible releases in principle — a bug fixed in version two still exists on every machine that never upgrades. Open-weight AI sharpens this into something categorically different, because the artefact being distributed is not a program with a version number but a fixed set of numbers that can be copied at zero marginal cost, run offline indefinitely, and modified by anyone with a single accelerator card.

The clearest illustration involves the model family this article’s series is named for. Meta announced restricted access to the original LLaMA weights on 24 February 2023, limited to vetted researchers under a noncommercial licence. Within roughly a week, a user posted a BitTorrent link to the full weight set on the imageboard 4chan; researchers mirrored it on GitHub and Hugging Face faster than Meta’s takedown requests could catch up, developers had it running on consumer laptops within days, and a Discord bot built on the leaked weights was soon being prompted to produce content the original release was never meant to permit ^[15]. Stanford’s Alpaca, a fine-tuned derivative, was announced about two weeks after the leak; the Stanford team withheld their own trained weights while they worked out the implications, which is a telling asymmetry — the restraint of one downstream lab did nothing to un-leak the base model everyone else already had ^[15].

RAND’s 2024 report on securing frontier model weights treats this as an established class of event rather than a one-off: the authors write of “at least one known case (and arguably two) in which one of the most capable models of its time was irreversibly leaked,” and build their entire security framework around preventing a repeat rather than around any method of undoing one after the fact ^[3]. The report’s recommendations — reducing the number of people with weight access, hardening interfaces against exfiltration, insider-threat programs, defense-in-depth, confidential computing — are all upstream controls. None of them, and the report does not claim otherwise, does anything for a model whose weights are already circulating ^[3].

Anthropic’s July 2026 position statement makes the same point from the releasing lab’s side of the decision, stated as plainly as a technical claim can be stated: “once weights are released they cannot be withdrawn” ^[1]. The company is not a neutral party in this argument — it does not release frontier weights itself, and saying release is irreversible supports its own posture — but the sentence is a factual claim about file distribution, not a values claim, and nothing in the counter-arguments examined later in this article disputes it.

What makes the problem specifically about safety rather than only about access is how thin the layer sitting on top of the weights turns out to be. Lermen, Rogers-Smith, and Ladish took the safety-tuned Llama 2-Chat 70B model — a release that had gone through Meta’s own alignment process — and used quantised low-rank adaptation to fine-tune it on a single GPU, for under two hundred US dollars, in a matter of hours. The refusal rate on two harmful-instruction benchmarks fell to roughly one percent, while general capability benchmarks were largely unaffected ^[5]. Safety fine-tuning, in other words, is not fused into the model in any sense that resists modification; it is a coat of paint that a few hundred dollars of compute strips back to bare metal, and the paint cannot be reapplied to a copy the licensor no longer controls.

Put the two facts together and the shape of the unsolved problem is precise. It is not “open weights are dangerous” — that is a separate, contested claim examined below. It is that the field has no method, proposed or implemented, for making a release reversible, and every mitigation on offer — better licences, use-restriction clauses, staged access — governs the decision to release, not anything that happens afterward. A simple way to see why after-the-fact governance cannot rescue the situation is to notice how quickly the number of independent copies dominates any per-copy probability of control. If $N$ copies exist and each copyholder independently retains theirs with probability $q$ close to one, the probability that every single copy is eventually deleted or brought back under control is $q^{N}$, so the probability that at least one copy survives indefinitely is

This is a deliberately simplified model — real retention decisions are not independent, and $q$ is not a single fixed number — but it isolates the right variable. Because copying an already-downloaded checkpoint costs approximately nothing, $N$ grows with every re-share, and $P(\text{at least one surviving copy})$ climbs toward one almost as soon as distribution starts, for any $q$ short of certainty. This is a restatement, in one line of algebra, of what NTIA’s 2024 report and RAND’s 2024 report both conclude in prose: monitoring and mitigation are the available tools, and recall is not among them ^{[2, 3]}.

The reproducibility gap

A second problem hides behind the word “open” itself. Running a model and reproducing how it was built are different capabilities, and a release can grant the first while withholding everything needed for the second.

The Llama 3 technical report is a useful benchmark for how much a leading developer discloses even in an unusually detailed release. It documents an approximate data mixture for the 405-billion-parameter flagship model — on the order of half general-knowledge tokens, roughly a quarter mathematics and reasoning tokens, and the remainder code and multilingual text, trained on 15.6 trillion tokens in total — and describes the curation and filtering pipeline in aggregate, qualitative terms ^[6]. What it does not do is publish the corpus, a dataset card sufficient to reconstruct it, or the exact filtering thresholds applied at each stage. A reader can learn the proportions of the recipe without ever learning the ingredients list, and this is true of essentially every major “open” release, not a criticism specific to one developer.

The LLM360 project frames this as a field-wide pattern rather than an isolated gap, and it is worth reading their claim as a limitation the authors are stating about the state of the art they are trying to improve on, not as an outside critique: “most LLMs have only released partial artifacts, such as the final model weights or inference code,” while “technical reports increasingly limit their scope to high-level design choices,” a combination they argue “hinder[s] progress in the field by degrading transparency into the training of LLMs” ^[7]. Their own release — full training code, data, checkpoints, and logs for two 7-billion-parameter models — exists specifically to demonstrate that the gap is not technically necessary, only commercially and legally inconvenient to close at frontier scale ^[7].

The Data Provenance Initiative’s large-scale audit supplies the quantitative version of the same finding, at the level of the datasets these models are built from rather than the models themselves. Auditing more than 1,800 text datasets across the platforms researchers actually use, Longpre and colleagues found licence omission rates above 70 percent and error rates above 50 percent on popular hosting sites — meaning that for a majority of widely used training datasets, a downstream user cannot reliably determine even the legal terms under which the data was gathered, let alone its provenance or composition ^[8]. That figure describes the raw material feeding into releases across the industry, open and closed alike; it is the reason “the training data is proprietary” and “the training data is genuinely undocumented, including by the people who assembled it” are frequently the same underlying situation described two different ways.

None of this means openness is meaningless — running a model without an API dependency, inspecting its weights directly, and fine-tuning it locally are all real capabilities that a weights-only release grants and a closed API does not. The unsolved problem is narrower and more specific: scientific reproducibility, in the sense of an independent team rebuilding a substantially equivalent model from what was published, remains the exception rather than the norm even among releases the industry and press describe as open, and the gap is documented by the same community trying to close it rather than only alleged by outside critics.

The contamination arms race

The third problem is closer to a genuine arms race than to a static gap, because open weights sharpen both sides of it at once. Anyone holding a full offline copy of a model can test it against a benchmark exhaustively, for free, with no rate limit and no provider watching — which is exactly what makes contamination easier to detect, and exactly what makes it easier to manufacture and hide.

The clearest evidence that this has already gone wrong at scale is institutional rather than academic. Hugging Face’s original Open LLM Leaderboard evaluated models against six benchmarks — including MMLU, HellaSwag, ARC, and GSM8K — that had, by mid-2024, become both saturated and contaminated: models were reaching near-human scores in ways that tracked memorization more than capability, and some entries showed clear signs of having been trained on data resembling the test sets ^[13]. Hugging Face’s response, in June 2024, was not a patch but a wholesale replacement with six harder benchmarks chosen specifically to resist saturation and contamination ^[13]. That fix did not hold indefinitely either: in March 2025, the Open LLM Leaderboard team retired the project outright, stating that it was “slowly becoming obsolete” and risked encouraging developers to “hill climb irrelevant directions in the field” after evaluating more than 13,000 submissions ^[12]. A leaderboard that its own maintainers shut down twice, in two different ways, inside a single year is strong direct evidence that the underlying measurement problem was not fixed by better benchmarks alone.

The detection side of the arms race has real, published methods, and each comes with a documented limit rather than a solved status. Shi and colleagues’ Min-K% Prob, presented at ICLR 2024, detects whether a given text was likely seen during pretraining by checking whether its lowest-probability tokens under the model are unusually confident — a method that needs no access to the training corpus itself and works from the model’s own output probabilities, validated on copyrighted-book detection and contaminated-benchmark detection alike ^[9]. But detecting verbatim or near-verbatim contamination does not catch a text that has been paraphrased. Yang and colleagues showed that simple n-gram string-matching decontamination — the method most pretraining pipelines actually use — is defeated by paraphrasing or translating test data before training on it, and that a 13-billion-parameter model deliberately trained this way could reach scores on MMLU, GSM8K, and HumanEval comparable to a much larger frontier model, without the string-matching filters ever flagging the overlap ^[10]. The gap between what a filter catches and what a rephraser can produce is not a corner case; it is the central finding of the paper.

The newest result closes off the most promising remaining fix. Reasoning models are trained with an additional reinforcement-learning stage on top of supervised fine-tuning, and Wang and colleagues report that this stage does something detection methods did not anticipate: “even a brief GRPO training can markedly conceal contamination signals that most detection methods rely on,” tracing the effect to the importance-sampling and clipping mechanics common to an entire family of reinforcement-learning algorithms, not a quirk of one method ^[11]. Contamination introduced before reinforcement learning, in other words, can be laundered by the reinforcement-learning stage itself, and the paper’s own framing is that evading current detectors is now “alarmingly easy” ^[11].

A simple decomposition makes clear why this is a measurement problem and not merely an inconvenience. Let $s_{\text{obs}}$ be an observed benchmark score, $s_{\text{true}}$ the underlying capability the benchmark intends to measure, and $c$ the fraction of test instances the model has effectively memorized, whether verbatim or through a rephrased variant. If a memorized instance is answered correctly with certainty, then approximately

Every published leaderboard number implicitly assumes $c \approx 0$. The methods above show that assumption is not something the field can currently verify for a given open-weight release, only something it can occasionally catch violated in specific, published cases — which is a different and much weaker guarantee than a general bound on $c$.

The access-versus-proliferation tension

The fourth problem is the one most often mistaken for a resolved disagreement, and the clearest evidence that it is not comes from the most recent attempt to resolve it.

Two claims are frequently treated as opposed. The first: open weights meaningfully raise the risk that a dangerous capability reaches someone who could not otherwise obtain it, because deployment-side controls — rate limits, monitoring, refusal behaviour — apply only to a hosted model and do nothing once weights are downloaded. The second: open weights matter less than the counterfactual availability of the same capability through other means, so restricting them trades a real, certain cost against a speculative, unmeasured benefit.

Documented incidents support the first claim’s premise, if not its full conclusion. In July 2023, SlashNext researchers disclosed WormGPT, a tool built on top of GPT-J, an openly released language model from EleutherAI, specifically fine-tuned and marketed on underground forums for generating business-email-compromise phishing content, sold on a subscription basis to buyers with no coding or hacking background required ^[14]. This is not a hypothetical dual-use scenario; it is a documented instance of an open-weight model being adapted, distributed, and monetized specifically to remove the safeguards a hosted product would have carried, precisely the mechanism the permanence section above describes in the abstract.

What resists resolution is not whether such incidents happen — they demonstrably do — but how they compare in scale to the counterfactual, and neither side of the debate currently has a way to measure that comparison. NTIA’s 2024 report, produced under a marginal-risk framework specifically designed to weigh open-weight-specific harm against harm achievable through existing tools, concluded that the U.S. government “should not restrict the wide availability of model weights for dual-use foundation models at this time,” while explicitly recommending that the government build new capacity to monitor for risks that might change that conclusion later — a recommendation that only makes sense if the report’s authors regard the current picture as provisional rather than settled ^[2].

Anthropic’s more recent position complicates any reading of that debate as closed. Writing in 2026 — a lab that, again, releases no frontier open weights of its own — Amodei argues the marginal-risk framing itself may not transfer cleanly to every capability domain, worrying specifically that “biology will have a strong attacker-defender asymmetry, where sufficiently capable models may be able to quickly weaponize pandemic-level viruses,” in contrast to domains like cybersecurity where the same openness that helps an attacker also helps a defender find and patch the resulting vulnerability ^[1]. This is presented as an open empirical question about which domains behave which way, not as a settled finding, and it is a materially different worry than “open models are dangerous in general.”

The most recent attempt at a neutral synthesis declines to resolve the disagreement rather than adjudicating it. The International AI Safety Report 2026, coordinated across more than thirty countries and authored by over a hundred experts, states plainly that open-weight access “facilitates research and innovation, including in AI safety,” while also noting such models “can also pose risks, for example by facilitating malicious or misguided use that is difficult or impossible for the developer of the model to monitor or mitigate” ^[4]. Both halves of that sentence are stated as established facts. Which one dominates for a given model, capability, and time horizon is exactly the question the report does not claim to answer, and the fact that the highest-effort international attempt to date presents both sides rather than a verdict is itself the strongest evidence that a genuine, unresolved tension — not a manufactured one — sits underneath the disagreement.

Why these four are unsolved, not merely contested

It is worth being precise about what distinguishes these four problems from the broader open-weights debate, because the distinction is the whole argument of this article. A contested question is one where reasonable people, given the same evidence, would still disagree — whether the benefits of accessibility outweigh the proliferation risk is a contested question, and this article has not tried to answer it. An unsolved problem, in the sense used here, is one where a specific technical or institutional fix was proposed, implemented, and then documented — by its own proponents, not only its critics — as not fully working.

Run back through the four. Licence restrictions and staged release are the proposed fix for irreversibility, and RAND’s own security framework is built entirely around preventing the initial leak because it has no answer for what happens after one, a gap the report does not paper over ^[3]. Model cards and disclosed data-mixture percentages are the proposed fix for reproducibility, and the organisation trying hardest to demonstrate a better standard says outright that most releases fall short of it ^[7]. Harder benchmarks and probability-based detectors are the proposed fix for contamination, and the newest detector paper reports that a training technique already in common use conceals the signal the detector relies on ^[11]. Marginal-risk analysis is the proposed method for resolving the proliferation tension, and the framework’s own authors recommend continuous monitoring precisely because the analysis is not treated as final ^[2]. In each case, the fix exists, has been tried, and has a documented hole in it. That is a stronger and narrower claim than “people disagree,” and it is the one this article has tried to support with evidence rather than assert by tone.

Predictions, with the observations that would falsify them

These are forecasts, kept separate from the sourced analysis above. Horizon: 12 August 2029.

One. No general-purpose technical method for making a weight release reversible will exist by the horizon date — meaning no released frontier open-weight model’s safety behaviour will be restorable against a determined fine-tuning attacker who holds the weights. Disconfirmed if a published, independently reproduced method demonstrates safety behaviour that survives adversarial fine-tuning of an already-released open-weight model at frontier scale.

Two. Fully reproducible releases in the manner LLM360 and comparable projects define it — weights, data, training code, and recipe together — will remain a minority of significant releases by count, even as more developers publish detailed data-mixture percentages without releasing the underlying corpus. Disconfirmed if a majority of releases from the largest five developers include the training corpus or a certifiably sufficient substitute.

Three. Benchmark contamination detection will continue to lag behind contamination-generation techniques rather than closing the gap, because each new detector targets the previous generation of evasion rather than the next one. Indicator: whether a published detector explicitly designed against reinforcement-learning-concealed contamination is demonstrated to hold up against a subsequent, independently developed evasion technique. Disconfirmed if such a detector is published and survives at least one independent adversarial attempt to defeat it.

Four. No jurisdiction or international body will publish an agreed quantitative threshold separating acceptable from unacceptable open-weight proliferation risk by the horizon date; qualitative, case-by-case marginal-risk assessments will remain the dominant tool. Disconfirmed if a government or the International AI Safety Report process publishes a specific, operationalised risk threshold that determines release decisions rather than informs them.

What is left standing

Strip away the parts of the open-weight debate that are genuinely about values, and four specific, documented gaps remain: a release that cannot be recalled and a safety layer that comes off for a few hundred dollars; a disclosure norm that stops well short of what its own advocates call reproducible; a benchmark ecosystem whose maintainers have twice concluded they could not trust their own scores; and a risk calculus that the field’s best current synthesis states honestly still has two live sides. None of these four is closed by picking a side in the openness argument, because each would still be a problem under either answer to it. They are the parts of the picture that stay the same size no matter which way the larger debate is decided, and that is what makes them worth treating separately from it.