A checkpoint is a beginning, not an endpoint
An open-weight release is usually narrated as a single event: a blog post, a licence, a download link, a benchmark table. The file arrives, the table is published, and the story appears to end there. It does not end there. A released checkpoint is the first link in a chain of forking, fine-tuning, relabeling, redistribution, and re-scoring that runs largely without the releasing lab’s visibility and mostly without anyone’s active supervision, and that chain has its own, separately documentable failure modes.
This article sets out ten of them, each backed by a primary source, a peer-reviewed study, a security advisory, a court record, or contemporaneous reporting — not by inference from how the ecosystem probably behaves. None of the ten is an argument that open weights are a mistake; several of the underlying problems (fine-tuning safety away, benchmark gaming, malicious payloads) have documented instances against closed, API-gated systems too. What is specific to the open-weight case is that the failure, once it occurs, propagates through an artefact anyone can copy, and the propagation itself is where the harder problems live.
1. Safety training removed within weeks, at low cost, in public
The clearest documented case is Llama 2. Meta released Llama-2-Chat on 18 July 2023 with safety fine-tuning built into the released weights. By 31 October 2023 — a little over three months later — Lermen, Rogers-Smith, and Ladish had published a paper showing that low-rank adaptation fine-tuning on a single GPU, for under two hundred US dollars, reduced the 70-billion-parameter chat model’s refusal rate on two harm benchmarks to roughly 1%, while leaving general capability intact [1]. Their explicit conclusion was that evaluating fine-tuning risk needs to be part of any decision to release weights, because the cost of undoing safety training was already, at that point, trivial relative to the cost of training the model in the first place.
A second and more general result explains why that finding was not a one-off. Arditi and colleagues showed, across thirteen open-weight chat models up to 72 billion parameters, that refusal behaviour is mediated by a single direction in the residual stream: subtracting that direction from the model’s weight matrices — a one-time linear-algebra operation requiring no gradient descent at all — disables refusal broadly while leaving other capabilities largely intact [2]. The technique, nicknamed “abliteration” by the practitioner who first popularised it, turned safety removal from a fine-tuning exercise measured in GPU-hours into a weight edit measured in minutes, and derivative “abliterated” versions of major open releases have circulated on model hubs since. Community uncensored variants of Llama 2 were already being discussed within about two weeks of its release, using conventional fine-tuning on a modified instruction dataset rather than the later, cheaper technique. The mechanism changed; the outcome — a safety-trained release circulating in a safety-stripped form within the same season it shipped — did not.
2. Ordinary fine-tuning silently eroding safety, with no bad intent
The failure above is loud: someone sets out to remove refusal and says so. A quieter version is more consequential precisely because nobody notices it happening. Qi and colleagues showed two things in the same study. First, that GPT-3.5 Turbo’s safety guardrails could be jailbroken by fine-tuning on just ten adversarial examples for under twenty US cents through a hosted API — a result about a closed model, cited here because it establishes the mechanism cleanly. Second, and more relevant to the open-weight case, that fine-tuning on entirely benign, commonly used datasets also measurably degraded safety alignment, without any adversarial intent on the part of whoever ran the job [3]. On an API, a provider can in principle monitor for this. Once weights are local, there is no equivalent checkpoint: a team fine-tuning an open-weight base model on ordinary support tickets or internal documents has no external signal telling them their refusal behaviour degraded, no comparison baseline, and often no safety evaluation step in their pipeline at all. The failure here is not that safety can be removed — it is that it can be lost as a side effect, unmeasured and unannounced, by people who never intended to touch it.
3. License terms that cannot be verified from outside the licensor
Meta’s Llama 4 Community License requires that a licensee whose products reach more than 700 million monthly active users in the month preceding release “must request a license from Meta, which Meta may grant to you in its sole discretion,” requires prominent “Built with Llama” attribution, and requires that “Llama” appear at the beginning of the name of any AI model created, trained, or fine-tuned using Llama materials and then distributed [4]. Each clause is coherent as a business term. None of them is independently verifiable by anyone outside Meta. Monthly active user counts are self-reported; there is no public registry of which derivative models exist, let alone which comply with the naming and attribution requirements; and no publicly confirmed case of Meta invoking the scale threshold against a named licensee was found in the course of researching this piece. That absence is the failure mode, not an exception to it: a restriction that can only be enforced by the party best positioned to selectively enforce it, against violations nobody outside that party can observe, is a restriction whose actual compliance rate is unknown by construction.
The closest thing to independent measurement of licence compliance across the ecosystem comes from a different angle entirely. Laufer, Oderinwale, and Kleinberg mapped the fine-tuning lineages of roughly 1.86 million models on Hugging Face and found that “licenses counter-intuitively drift from restrictive, commercial licenses towards permissive or copyleft licenses, often in violation of upstream license’s terms” [5]. That is a documented, quantified finding of systematic licence non-compliance across the fork ecosystem — just not the specific clause Meta’s licence is best known for, and not something any licensor is positioned to police at the scale the same study measured.
4. Fine-tuning lineages with no canonical version
The same study is also the strongest evidence for a distinct failure: that “open” releases fragment into sprawling families with no single authoritative descendant. Laufer and colleagues describe family trees of adapters, quantisations, and merges branching from base checkpoints, and report a counterintuitive structural finding — that “mutations are fast and directed, such that two ‘sibling’ models tend to exhibit more similarity than parent/child pairs” [5]. In plain terms: two forks released around the same time, by different people, for different purposes, often resemble each other more than either resembles the base model both came from, because fine-tuning converges on the loudest recent trend (instruction format, chat template, popular dataset) rather than preserving lineage. A base weights release does not have one “real” successor. It has a distribution of them, diverging in ways that track fashion in the fine-tuning community more than fidelity to the original.
5. A model fine-tuned and re-released as if it were original work
In June 2024, a Stanford-affiliated team released a multimodal model called Llama3-V. Within days, the maintainers of MiniCPM-Llama3-V 2.5 — a model built by Tsinghua University and OpenBMB — documented, in a GitHub issue that drew wide attention across Hugging Face and the broader research community, that Llama3-V’s directory structure, configuration files, and behaviour closely matched their own, including an idiosyncratic recognition capability for Tsinghua Bamboo Slip characters that had only ever appeared as an experimental, undisclosed feature of MiniCPM-Llama3-V 2.5 [6]. That behavioural fingerprint — a quirk with no plausible independent origin — is what made the allegation stick rather than remain a dispute about superficial similarity. The Llama3-V authors deleted the GitHub issue when it was first raised, then deleted their repositories and hid their Hugging Face project page once the evidence spread; two of the three authors subsequently issued a public apology. The mechanism generalises beyond this one case: a fine-tune or a re-packaging of someone else’s derivative work can be presented as original research, and the open-weight ecosystem’s provenance signal — a model card someone wrote themselves — has no independent verification behind it unless a wronged party does the forensic work publicly, as happened here.
6. Training data too undisclosed for compliance to be checkable, even in litigation
Thirteen authors, including Richard Kadrey and Sarah Silverman, sued Meta alleging that Llama’s training pipeline drew on their copyrighted books obtained through shadow libraries including Library Genesis and a later compilation, Anna’s Archive; discovery in the case established that Meta had downloaded at least 666 copies of books the plaintiffs held copyright over [7]. On 25 June 2025, Judge Vince Chhabria granted summary judgment to Meta on the fair-use question, finding the training use “highly transformative” and the plaintiffs’ evidence of market harm insufficient to defeat the defence [7]. The ruling matters here less for its fair-use holding than for what had to happen to establish even the basic facts: it took a federal lawsuit and discovery to get a specific, sourced count of which books were downloaded from which archive. No public, pre-litigation documentation from Meta made that information available to a downstream user trying to assess licensing exposure before building on Llama. An open-weight release routinely discloses none of the specifics needed to check copyright compliance — not the source datasets, not a manifest, not even an aggregate provenance summary — and the training-data opacity is uncorrelated with how permissively the resulting weights are licensed. Two releases can carry an identical, generous weights licence while one’s training data was scraped under terms the other’s was not; nothing about the weights or the licence tells a user which is which.
7. Vulnerabilities independent audits find in the code that loads the weights
Weights are inert data; something has to parse them, and that code is itself an attack surface distinct from the model. Cisco Talos disclosed CVE-2024-23496 in llama.cpp’s GGUF file parser on 26 February 2024, a heap-based buffer overflow in the gguf_fread_str function, rated 8.8 out of 10 in severity, exploitable by a specially crafted .gguf file leading to arbitrary code execution [8]. That finding was one of a cluster of similar GGUF-parsing overflows disclosed the same season, all in one of the most widely embedded pieces of open-weight inference software, used by desktop tools, mobile apps, and server deployments that never touch Meta or any model publisher directly. The failure here is orthogonal to everything about the model’s own quality, licence, or training data: it is that “download the weights and run them” implies trusting a parser that reads untrusted, attacker-influenceable input by design, and that trust has been misplaced at least once in a way independent researchers, not the tool’s own maintainers, had to find.
8. Malicious weights distributed as if they were legitimate models
A file that loads without crashing is not the same as a file that loads safely. In February 2024, JFrog’s security research team reported finding roughly 100 machine-learning models on Hugging Face carrying real, harmful payloads, exploiting Python’s pickle serialisation format via its __reduce__ method to execute arbitrary code the moment a model was loaded; one identified model established a reverse shell to a specific external address, granting an attacker control of the loading machine while the victim believed they were loading an ordinary checkpoint [9]. This is distinct from the parser vulnerabilities above: those require a flaw in trusted software; this requires only that a user download and load a file from an untrusted uploader, using software working exactly as designed. Model hubs host uploads from anyone, model names can imitate legitimate releases, and unlike a software package registry with decades of supply-chain tooling behind it, model-weight distribution had no comparable norm of provenance verification until security vendors began building it after incidents like this one were found.
9. Benchmark and leaderboard scores gamed by the model’s own publisher
On 5 April 2025, Meta released Llama 4, and a variant labelled Llama-4-Maverick-03-26-Experimental appeared on the LMArena leaderboard with an Elo rating of 1417, ranking near the top of the board [10]. That variant was not the model anyone could download. When the publicly released Llama-4-Maverick-17B-128E-Instruct was subsequently benchmarked on the same leaderboard, it ranked 32nd, below GPT-4o, Claude 3.5 Sonnet, and Gemini 1.5 Pro [11]. LMArena’s own account of the episode stated plainly that “Meta’s interpretation of our policy did not match what we expect from model providers,” and that Meta should have made clear the experimental submission was “a customized model to optimize for human preference” rather than the model being released [10]. LMArena published more than 2,000 head-to-head battle records for independent review and tightened its disclosure rules afterward. This is benchmark gaming through leaderboard-specific optimisation rather than test-set contamination, and it is worth distinguishing from the second, quieter version of the same failure: Zhang and colleagues constructed GSM1k, a benchmark deliberately matched in style and difficulty to the widely used GSM8k grade-school-math benchmark but built by human annotators with no LLM involvement to guarantee it was never in any training set, and found that several open-weight model families — notably Phi and Mistral — showed accuracy drops of up to 13 percentage points moving from GSM8k to GSM1k, consistent with the original benchmark’s answers having leaked into training data [12]. One case is a publisher choosing which variant to submit for evaluation; the other is a publisher’s training data overlapping, deliberately or not, with the test set the world is using to judge it. Both produce a headline number that does not describe the model anyone can actually download.
10. A stable model name silently resolving to a different model
Open weights are usually assumed to be immune to the silent-update problem that closed APIs have — a downloaded file does not change under you. The distribution layer around those files can still change what a fixed name points to. When DeepSeek released the R1 family in January 2025, the smaller checkpoints were explicitly named as distillations — DeepSeek-R1-Distill-Llama-8B, DeepSeek-R1-Distill-Qwen-32B, and so on — correctly reflecting that they were existing Llama and Qwen models fine-tuned on R1’s reasoning traces, not the 671-billion-parameter model itself. Ollama’s model library collapsed that distinction, listing the distilled variants under the bare tag deepseek-r1. Simon Willison documented the consequence directly after DeepSeek’s May 2025 update: “Ollama have mixed the two new models into their existing deepseek-r1 label. Last week running ollama pull deepseek-r1:8B from Ollama would get you DeepSeek-R1-Distill-Llama-8B… today that same command gets you DeepSeek-R1-0528-Qwen3-8B, a completely different base model” [13]. Two users running the identical command a week apart received checkpoints with different base architectures, different training lineages, and materially different behaviour, under a name that gave no indication anything had changed. The weights themselves were static; the resolution of a name to a specific file was not, and the layer where that resolution happens sits outside any model publisher’s control.
What the ten share
None of these ten is exotic. Each is a specific, dated, sourced instance of a general property: an open-weight release multiplies the number of places a given property of the model can fail, without multiplying the number of parties responsible for checking it. A safety-trained closed model has one deployment surface a provider monitors; a safety-trained open model has as many deployment surfaces as there are people who downloaded it, and monitoring responsibility does not travel with the file. If a base model preserves a given desirable property — genuine safety training, license compliance, unmodified provenance — with independent probability
Predictions, with the observations that would falsify them
These are forecasts, kept separate from the sourced findings above. Horizon: August 2029.
One. Provenance and scanning tooling for model weights (comparable to software supply-chain tooling) will become standard practice at major hubs before licence-compliance tooling does, because malicious payloads are technically detectable while licence violations require information the hub does not have. Disconfirmed if a major hub ships automated licence-lineage enforcement before it ships default-on malicious-weight scanning for all uploads.
Two. At least one major open-weight lab will publish, or be compelled to publish, a specific dataset manifest or provenance disclosure exceeding what is currently standard, following continued litigation pressure of the kind seen in Kadrey v. Meta. Disconfirmed if, by the horizon date, no top-ten open-weight releaser has published training-source-level disclosure beyond a high-level description.
Three. Benchmark leaderboards will move toward requiring reproducible, hash-verifiable model identifiers for any submission used in public rankings, following the Llama 4 Maverick episode. Disconfirmed if major leaderboards in 2029 still accept unverifiable, publisher-labelled “experimental” submissions into headline rankings without an audit trail.
Four. The gap between how safety-trained a released open-weight checkpoint is and how safety-trained its most popular community fork is will remain unmeasured at the ecosystem level, because no actor has both the incentive and the standing to measure it across hubs. Disconfirmed if a recurring, independent, cross-hub audit of derivative-model safety behaviour exists and is regularly published by 2029.
What “open” does not include
Every failure catalogued here happened after a release, not instead of one. That distinction matters, because the natural response to this list is to read it as a case against open weights, and the evidence does not support that reading — most of the underlying mechanisms (fine-tuning away alignment, gaming a benchmark, shipping a malicious artefact, misattributing provenance) have documented instances against closed systems too, and a closed system’s opacity is its own, differently shaped failure mode, not an absence of one. What is specific to open weights is that “the release” was never actually the finish line the announcement implied. The weights leaving the building are the beginning of an audit trail that mostly does not get audited, by parties who mostly are not positioned to do it, using tooling that mostly does not exist yet. Reading a release announcement as though it settles what a downstream user will actually get — an aligned model, a compliant model, an unmodified model, a stable-named model, a model whose score means what the leaderboard implies — is the single error underneath all ten items on this list.
