An archive, not a scoreboard
“The model has safety training.” “We red-team our systems.” “The classifier caught it.” Each of these sentences is usually offered as though it settled a question, the way a pass/fail stamp settles one. The published record does not support that reading. Independent security researchers, red-teamers and the AI developers themselves have documented, separately and repeatedly, that deployed safety and misuse-control systems fail in specific, recurring, causally distinct ways — not as isolated bugs to be patched once, but as structural consequences of how refusal training, content classifiers, red-team programs and monitoring pipelines are actually built.
This article catalogues ten such failure modes as a post-mortem archive rather than a report card: not “is this system safe” but “which of the ten documented ways can this kind of system fail, and what does the paper trail for each one actually say.” None of the ten is specific to one vendor. Several have been demonstrated against multiple providers’ systems independently; all are drawn from peer-reviewed research, red-team papers or the AI developers’ own safety-framework filings, cited individually rather than summed into a single verdict.
1. Jailbreak techniques that bypass safety training entirely
The clearest demonstration that refusal is not an unbreakable property of a trained model is Zou and colleagues’ work on universal, transferable adversarial suffixes. Their method runs an automated, greedy coordinate-gradient search over an open-weight surrogate model — Vicuna, in their experiments — to find a short token suffix that, appended to almost any harmful request, maximises the probability that the model begins its answer affirmatively rather than with a refusal [1]. The result that matters is not that an open model can be attacked; open weights invite exactly that. It is that the same suffixes, optimised entirely against the surrogate and never touching the target’s weights, transferred with measurable success to closed commercial systems including ChatGPT, Bard and Claude, alongside further open models [1]. Safety training installed in one model’s weights was not sufficient to stop an attack string discovered by attacking a completely different model. That transfer property is the finding: the vulnerability is not particular to a checkpoint, and defeating it in one place does not defeat it everywhere it might reappear.
2. Downstream fine-tuning that strips trained refusals
A refusal habit installed by reinforcement learning from human feedback is, mechanically, a further round of training layered on top of a capable base model. Qi and colleagues showed that a comparably small amount of further training can remove it. Fine-tuning GPT-3.5 Turbo through the ordinary customer-facing API on just ten adversarially chosen examples, at a total cost under twenty cents, was enough to jailbreak the model’s safety guardrails on the majority of harmful categories tested, and the paper’s more unsettling finding is that even ordinary, benign fine-tuning datasets degraded safety alignment measurably, without any adversarial intent on the part of whoever ran the job [3]. Lermen, Rogers-Smith and Ladish demonstrated the open-weights version of the same result: low-rank adaptation fine-tuning applied to Llama 2-Chat 70B, for under two hundred dollars on a single GPU, drove the model’s refusal rate on two safety benchmarks down to roughly one percent, while the model’s general capabilities on standard evaluations were essentially preserved [4]. Read together, the two papers say the same thing from opposite sides of an API boundary: trained refusal behaves like a thin, separable layer sitting on top of capability rather than something woven irreducibly through it, and the training operation that removes the layer looks almost exactly like the training operation that installed it.
3. Classifiers with high false-negative rates on adversarial paraphrase
A second, structurally independent line of defence is an external classifier that scores a request or response without depending on the underlying model’s own trained judgment. Chao and colleagues’ PAIR method shows how readily such a classifier can be walked around by iterative rewording alone. An attacker language model is placed in a loop with the target: it proposes a candidate jailbreak, observes the target’s response, and revises the phrasing, mimicking social-engineering persuasion rather than any gradient-based search — and the paper reports that this loop typically needs fewer than twenty queries to produce a working jailbreak, with competitive success rates against GPT-3.5, GPT-4, and other closed and open models [2]. Nothing about the underlying harmful request changes across those iterations; only its surface phrasing does. A parallel result in content moderation, rather than jailbreaking specifically, comes from Hartvigsen and colleagues’ ToxiGen work, which built an adversarial generation procedure explicitly designed to produce implicitly toxic statements — text that carries the same hateful content as an overt slur without using one — and found that existing toxicity classifiers, trained on more overtly worded examples, missed a substantial share of it [5]. Both results describe the same underlying gap: a classifier trained on one distribution of phrasing is, by construction, evaluated out of distribution the moment an adversary reworks the surface form, and neither paper’s target classifiers were built to anticipate that.
4. Red-team findings that never reach a shipped mitigation
Red-teaming is frequently invoked as the industry’s answer to the first three failure modes, but Feffer and colleagues’ review of how it is actually practised complicates that answer considerably. Surveying public red-teaming literature and case studies, they found substantial inconsistency in what organisations even mean when they say they “red-teamed” a system — differences in scope, adversary sophistication, and reporting depth wide enough that the term covers activities with little in common beyond the name [6]. Their sharpest point is about the gap between conducting an exercise and acting on it: public announcements of red-teaming efforts, in their sample, routinely stopped short of stating whether the vulnerabilities found were fixed, deprioritised, or simply logged, and they characterise the invocation of red-teaming as a general-purpose reassurance — a “panacea for every possible risk” — as tipping into security theatre precisely where that specificity is missing [6]. This is not a claim that red-teaming is worthless; the paper is explicit that it can be genuinely valuable when conducted rigorously and acted on internally. It is a claim that the public record, as currently reported across the field, usually cannot distinguish a red-team exercise that changed the shipped product from one that produced a slide in an internal deck.
5. Safety training that generalises only to tested phrasings
Yong, Menghini and Bach supplied one of the cleanest demonstrations that safety training’s coverage tracks its training and testing distribution rather than a model’s underlying capability. Translating the same harmful prompts from English into low-resource languages such as Zulu or Scots Gaelic and submitting them to GPT-4 succeeded in eliciting actionable harmful content 79% of the time on their benchmark, a rate they report as comparable to, or in some categories exceeding, established jailbreaking techniques targeting the model in English — while the same translated prompts into high- and mid-resource languages showed markedly lower success [7]. The model’s ability to understand and respond fluently in the low-resource language was not in question; its safety training simply had not been exercised there, likely because the red-teaming and reinforcement data that shaped its refusals was itself concentrated in a handful of well-resourced languages. This is the mismatched-generalisation pattern from the jailbreak literature made visible at the scale of an entire language rather than a single clever prompt: whatever a model’s safety training covers, it covers only where it was tested, and everywhere else is an open question rather than an inherited guarantee.
6. Misuse classifiers with documented disparate false-positive rates
A classifier’s errors are not evenly distributed across the population it scores, and this has been measured directly rather than inferred. Davidson, Bhattacharya and Weber examined widely used hate-speech and abusive-language datasets and the classifiers trained on them, and found that tweets written in African-American English were flagged as abusive at substantially higher rates than comparable text in Standard American English, a pattern traced to how the underlying training data was itself labelled [8]. The practical consequence they identify is a disproportionate false-positive burden landing on exactly the community such tools are typically framed as protecting. Hartvigsen and colleagues’ ToxiGen paper reports the same shape of failure from a different angle: toxicity systems that “falsely flag text that contains minority group mentions as toxic,” regardless of the actual content of the sentence, meaning mere reference to a demographic group raises a classifier’s score independent of hostility [5]. A misuse-control system with a demographically uneven false-positive rate is not a hypothetical risk to be modelled in advance; it is a measured property of specific, deployed classifier families, documented well before large language models made automated content moderation ubiquitous.
7. Automated safety classifiers manipulable via prompt injection
As classifiers and moderation systems increasingly are language models themselves — used as automated judges to score outputs, rank candidate responses, or select what gets shown — they inherit language models’ own vulnerability to injected instructions, and Shi and colleagues built an attack that demonstrates this directly against LLM-as-judge systems. Their method, JudgeDeceiver, formulates the search for an injected sequence as an optimisation problem: embedded into an attacker-controlled candidate response, the sequence causes an LLM judge to select that response for an attacker-chosen question regardless of what the other candidates contain [9]. The paper tested the attack against practical use cases including retrieval ranking, reinforcement learning from AI feedback, and tool selection, and reports it substantially outperforming both existing manual prompt-injection attempts and jailbreak techniques adapted to the same setting; the authors further tested three published defences — known-answer detection, perplexity filtering, and windowed perplexity filtering — and found all three insufficient against the optimised attack [9]. The safety irony is direct: a classifier deployed specifically to catch manipulated or harmful content is, when that classifier is itself an LLM, a further attack surface for the same class of manipulation it exists to detect.
8. Safety measures degrading helpfulness disproportionately
Meta’s own account of building Llama 2-Chat documents this trade-off as an engineering problem the team had to design around, not as a rumour about training. Citing earlier work on the tension between helpfulness and harmlessness objectives, the paper states plainly that the two “sometimes trade off,” which “can make it challenging for a single reward model to perform well on both” [10]. Their response was structural rather than cosmetic: rather than training one blended reward model, they trained two separate reward models, one for helpfulness and one for safety, and used both during reinforcement learning specifically because a single combined objective had not served both goals well [10]. The paper also reports that, given sufficient helpfulness training data, an additional stage of safety-specific mitigation did not measurably degrade helpfulness performance in their setup [10] — a genuinely reported mitigation, and one worth reading for what it implies rather than dismisses: that outcome required a deliberate architectural fix precisely because the naive approach of optimising helpfulness and safety together, in the same objective, was the version that cost helpfulness. The alignment tax is not a fixed toll every safety measure charges; it is a cost that shows up by default and has to be engineered away, one training decision at a time.
9. Inadequate monitoring to catch novel jailbreak patterns after deployment
Shen and colleagues gave the clearest evidence that the gap between a jailbreak technique existing and a deployed system catching it can be measured in months, not hours. Collecting 1,405 jailbreak prompts circulating across Reddit, Discord, dedicated prompt-sharing sites and open datasets between December 2022 and December 2023, they found the earliest prompt in their collection had persisted online, in active use, for over 240 days, and that 28 separate accounts had continuously iterated on and refined jailbreak prompts for more than 100 days apiece [11]. When they tested the strongest prompts from this in-the-wild collection against six LLM chatbot services, five achieved a 0.95 attack success rate against both GPT-3.5 and GPT-4, and the authors conclude directly that the platforms’ safeguards “cannot adequately defend jailbreak prompts in all scenarios” [11]. These were not novel zero-day discoveries requiring specialist skill; they were techniques openly discussed, refined collaboratively, and left running in public communities for weeks or months at a stretch before, or without, being caught by whatever monitoring the deployed systems had in place. A misuse-control pipeline that only inspects individual exchanges in isolation has no natural way to notice that pattern; noticing it requires watching the outside world where attacks are actually developed, which is a different and much less well-instrumented job than scoring a single request.
10. Safety commitments that are not independently verifiable
The furthest-upstream failure mode covers all nine of the others: an outside party’s ability to check whether any of them have actually been addressed. Casper and colleagues’ argument is that the access most external evaluators actually get to a closed frontier system — sending queries and reading responses, so-called black-box access — is structurally insufficient for a rigorous audit of the kind that would let someone outside the company confirm a safety claim rather than take it on trust [12]. They argue for a graduated model of deeper access: white-box access to weights, activations and gradients enables far stronger adversarial testing and interpretability work than input-output probing alone can, and “outside-the-box” access to information about training data and deployment context enables auditors to check claims about a system’s development that no amount of querying the finished product could ever surface [12]. Their point is not that any particular company’s safety claims are false. It is narrower and more durable than that: without some version of the deeper access they describe, a published safety commitment and an unverified assertion are, from an outside auditor’s vantage point, indistinguishable, and the industry’s current default — self-reported system cards and voluntary disclosure — is the version of AI safety oversight that asks the reader to trust the party being graded.
What the ten share
None of these ten is a coding error waiting to be patched once. Each follows from a specific, identifiable design decision: refusal is trained into weights and weights have adversarial examples; a classifier scores a distribution of phrasing and paraphrase moves the input out of that distribution; a red-team program produces findings and someone still has to route them to engineering; a judge built from a language model inherits that model’s own injectability; monitoring that only inspects single exchanges cannot see a pattern spread across a public forum over months; and an audit built only on querying the finished system cannot see what happened during training. Fix any one of the ten and the other nine remain exactly as documented.
The failure modes built on repeated querying — jailbreak search, paraphrase evasion, mismatched-language generalisation — share a further structural property worth stating precisely, because a single strong per-attempt success rate is often reported as though it settled the question of robustness on its own. Suppose a safeguard blocks a fixed, independently generated attack attempt with probability
At
Predictions, with the observations that would falsify them
These are forecasts, kept separate from the sourced findings above. Horizon: 12 August 2029.
One. Published red-team reports will increasingly disclose a specific routing outcome for each finding — fixed, mitigated, accepted as residual risk, or unresolved — rather than an aggregate hours-and-severity summary, because the current format cannot answer Feffer and colleagues’ critique. Disconfirmed if major system cards in 2029 still report red-team engagement hours and counts without a per-finding disposition.
Two. At least one further language-coverage or dialect-coverage gap in a deployed safety system, comparable in kind to the low-resource-language and African-American-English findings cited here, will be documented in a peer-reviewed paper against a currently deployed frontier system before the horizon date, because safety training data collection has not caught up with the number of languages and dialects these systems are fluent in. Disconfirmed if no such gap, in any language or dialect, is documented against a 2027-or-later frontier model.
Three. Structured-access AI audit regimes — granting vetted outside researchers some form of access beyond ordinary API queries — will expand from voluntary pilot programs to a formal requirement in at least one major jurisdiction’s AI regulation, because Casper and colleagues’ argument has no persuasive rebuttal on the public record. Disconfirmed if by the horizon date no jurisdiction with a binding AI-safety statute requires anything beyond black-box evaluation for frontier-scale systems.
Reading a safety claim after this
A refusal, a passed red-team exercise, a low false-positive rate on a benchmark, and a published safety framework are each, individually, evidence about exactly one of the ten failure modes catalogued here and silent about the other nine. None of them, alone or together, constitutes proof that a deployed system is safe in some general sense, because “safe” is not the kind of property this evidence can settle — it is ten separately measurable properties, each with its own documented failure rate, its own attribution, and its own open question about whether anyone outside the company that built the system has actually checked. The useful habit is the one a post-mortem archive forces by its structure: for any safety claim, ask which of the ten drawers it belongs in, what specifically was measured, who measured it, and which of the other nine drawers it says nothing about at all.
