What happens in the half-second before a refusal
When Claude declines a request, the visible output is a single paragraph of text. Underneath it, at least four structurally different systems may already have acted, and they were built at different times, by different teams, against different threat models, and they fail in different ways. Conflating them is the single most common error in public discussion of “AI safety guardrails” — commentators treat a refusal as one thing, a single dial an operator turns up or down, when the documented architecture is closer to a pipeline of independent stages, each with its own error rate, its own training data, and its own trade-off between blocking too much and blocking too little.
This article works through that pipeline stage by stage, using only what Anthropic has published about its own systems — model and system cards, its Responsible Scaling Policy, and its research posts on classifier defenses — together with independent academic red-team research that has probed the same class of system from outside. The aim is not to certify that Claude’s safeguards work, nor to declare them theater. It is to describe, at the level of documented mechanism, what actually sits between a request and a response, what evidence exists for each piece, and where that evidence is self-reported, independently replicated, or simply absent. The recurring finding is that every one of these mechanisms buys a reduction in one error type by accepting an increase in the other, and the honest engineering question is never “does the safeguard work” but “at what point on that curve was it set, and by whom.”
The refusal is trained, not appended
A refusal is often described, informally, as a filter — something bolted onto a model that would otherwise comply. That description is wrong for the part of the system that actually produces the words “I can’t help with that.” Anthropic’s published training method for this behavior, Constitutional AI, folds it directly into the weights during post-training, in two stages [6]. In the first, supervised stage, the model is sampled, prompted to critique its own problematic outputs against a written set of principles, and then fine-tuned on the revised responses. In the second, reinforcement stage, the model itself judges pairs of its own outputs against those principles to build a preference dataset, which trains a reward model that then drives further reinforcement learning — reinforcement learning from AI feedback, in the paper’s own terms, rather than the human-labeled comparisons earlier alignment work relied on. The stated goal was not maximal refusal: an assistant trained this way is meant to engage with a difficult request by explaining its objection rather than only declining, because a model that is evasive rather than merely careful loses the ability to be helpful at exactly the borderline a policy exists to be applied to.
The data that principles get tested against is not invented in the abstract. Anthropic’s earlier red-teaming work describes systematically attacking models across sizes and training methods and reports a specific, load-bearing result: models trained with reinforcement learning from human feedback became harder to red-team as they scaled up, while other training approaches showed a flat trend with size — scale alone did not buy safety, but scale combined with this particular training method did [11]. That paper’s release of nearly 39,000 red-team transcripts is one of the few points in this literature where the raw attack data, not only an aggregate success rate, is public.
What this training produces, mechanistically, is now understood well enough to be attacked directly, though not on Claude itself. Independent researchers examining open-weight chat models found that refusal behavior is not distributed evenly across a network’s internal representations; it is mediated by a single direction in the residual stream that can be isolated with a small number of contrastive prompts. Deleting that direction from a model’s activations causes it to comply with requests it was trained to refuse, while artificially adding the same direction causes it to refuse requests that are entirely benign [9]. That is a striking result about how safety training generalizes as a geometric object rather than a diffuse property, and it holds across model families up to 72 billion parameters that the researchers had white-box access to. It is not a result about Claude specifically, and that gap is itself informative: Claude’s weights are not public, so no outside researcher can run the same ablation against it, and Anthropic has not published an equivalent internal circuit analysis of its own refusal mechanism. The documented vulnerability class — refusal as a low-dimensional, potentially fragile direction rather than a robust distributed judgment — is real and peer-reviewed. Whether it applies to Claude’s internals in the same form is, on current public evidence, simply unknown.
A second system that never touches the weights
Because a trained refusal is a property of the weights, it inherits whatever the weights inherited, including any gap between what safety training covered and what a sufficiently creative prompt can still reach. Wei, Haghtalab, and Steinhardt’s peer-reviewed analysis names two structural reasons that gap exists: “competing objectives,” where a model’s drive to be helpful and follow instructions is placed in direct tension with a safety constraint by the prompt itself, and “mismatched generalization,” where safety training was never exposed to a domain — an encoding, a language, a framing — that the model’s raw capabilities can still operate in [7]. Their attacks, tested against GPT-4 and an early Claude model, succeeded on every prompt in the models’ own red-teaming evaluation sets. The paper is now several model generations old and does not describe current Claude behavior, but the taxonomy it introduced — objectives in tension, and training that failed to generalize — is still the standard way later work, including Anthropic’s own, frames the problem.
The response Anthropic has published is not to keep patching the weights but to add a second, independent system that sits in the request-and-response path without touching them: a classifier trained separately from the language model, scoring input and output text against a written “constitution” of permitted and restricted content categories [4]. The distinction matters mechanically. A weight-level refusal can, in principle, be reasoned around by a prompt clever enough to exploit exactly the gap Wei and colleagues describe. A classifier making an independent judgment about the same exchange does not share the same blind spots by construction, though it can have its own.
Anthropic’s published account of testing this system is unusually specific for the industry. Across roughly 3,000 hours of red-teaming by paid, briefed participants over two months, targeting a fixed set of ten forbidden queries, no participant produced a universal jailbreak — a single method that reliably extracted detailed answers to all of them — against the classifier-guarded model, where 86% of the same style of attack succeeded against the model with no classifier at all; with the classifier in place, that fell to 4.4% [4]. That is Anthropic’s own reported comparison, not an independently replicated benchmark, and it comes with a caveat Anthropic itself later published: a separate, broader public bug-bounty program run against the same deployed system did find one universal jailbreak that the structured red-team trial had missed [1]. The gap between a scoped trial and an open bounty is itself a data point about how much of a system’s reported robustness depends on who is looking and how they are incentivized.
That first version cost real computation: 23.7% more inference compute than running the model alone, and a 0.38 percentage-point increase in refusals on ordinary, harmless traffic [4]. Anthropic’s January 2026 follow-up redesigned the system around a two-stage cascade rather than a single classifier pass: a lightweight probe reads the language model’s own internal activations and screens every exchange cheaply, escalating only the fraction it flags to a slower, more capable classifier that reviews the input and output together [1]. If
Because
Misuse detection beyond a single exchange
Not every documented failure mode is a single adversarial prompt, and not every safeguard operates on a single exchange. Anthropic’s 2024 many-shot jailbreaking research describes an attack that has nothing to do with clever phrasing: stuffing a long context window with hundreds of faux dialogue turns in which a fictional assistant answers progressively more harmful questions, then appending the real question at the end. Long-context in-context learning does the rest — the technique worked against Claude 2.0 and against models from several other developers, and attack success climbed with the number of turns in a pattern the researchers describe as close to a power law [5]. The reported mitigation history is instructive about which layer actually holds. Fine-tuning the model to refuse jailbreak-shaped inputs only pushed the crossover point further out along the context window rather than removing it; a classification-and-modification approach applied to the input before it reached the model cut the attack’s success rate from 61% to 2% in the case Anthropic describes [5]. That is a second, independent-of-classifiers demonstration that a text-level filter sitting in front of the model, not additional training of the model itself, is currently the load-bearing defense against at least this entire class of attack.
Misuse detection also operates above the level of any single exchange, at the level of accounts and patterns over time, governed by Anthropic’s Usage Policy rather than by a per-turn classifier. The policy separates Universal Usage Standards that apply to everyone from stricter High-Risk Use Case requirements for consumer-facing domains such as healthcare, legal and financial advice, and from further guidance for chatbots, products aimed at minors, and agentic systems that can take actions rather than only produce text [8]. Enforcement is described as the responsibility of a dedicated Safeguards Team using “detection and monitoring,” with consequences ranging from throttling and output modification to account suspension or termination; the policy’s own text does not enumerate the technical detection methods behind that monitoring [8]. This is the layer the Responsible Scaling Policy calls “asynchronous monitoring” and “post-hoc jailbreak detection” — checks that do not have to complete before a response is returned, and that can catch a pattern spread across many individually unremarkable requests that no single real-time classifier would flag [2]. Structurally, it is the piece of the pipeline furthest from being independently auditable: a real-time classifier’s accuracy can at least be probed by an outside party sending requests and observing outputs, but a pattern-of-account judgment is invisible to anyone who is not Anthropic, by design, since publishing its exact triggers would hand a map of the blind spots to whoever wanted one.
The Responsible Scaling Policy’s capability thresholds
All of the above describes defenses around a model that is already deployed. The Responsible Scaling Policy governs a prior question: whether a given model should be deployed with any particular set of safeguards at all, tied to what the model can actually do rather than to what it says when asked. The current version, 3.4, effective 8 July 2026, organizes this around AI Safety Levels [2]. ASL-2, the default standard applied to every model Anthropic has shipped so far, requires security capable of thwarting “most opportunistic attackers.” ASL-3 is triggered by specific, named Capability Thresholds — most concretely, that a model could measurably help someone with an undergraduate-level technical background move meaningfully closer to acquiring or deploying a chemical, biological, radiological or nuclear weapon, benchmarked against what off-the-shelf tools could already do in 2023, when the policy was first published [12]. Crossing that threshold does not trigger one safeguard; it triggers a bundled Deployment Standard the policy describes as four layers — access controls, real-time classifiers, asynchronous monitoring, and post-hoc jailbreak detection — plus a separate Security Standard spanning seventeen named areas of protection for the model’s weights themselves, from two-party authorization for anyone accessing them to bandwidth controls on outbound network traffic specifically intended to make weight theft detectable before it completes [2, 3].
Anthropic activated ASL-3 for Claude Opus 4 in May 2025, and its own account of that decision is more hedged than a marketing summary of “we made it safer” would suggest: the company states plainly that it could not clearly rule out ASL-3-level CBRN risk for that model, and treated the inability to rule the risk out — not a confirmed capability — as sufficient grounds to apply the heavier standard [3]. The Claude Opus 4.5 system card documents the process behind that kind of call in more procedural detail than most vendors publish for any product decision: a Frontier Red Team evaluates the model against the threshold criteria and produces a Capability Report; an Alignment Stress Testing team independently reviews and can challenge that report; both go to the company’s Responsible Scaling Officer and CEO, who make the final level determination [12]. Critically, some of that evaluation runs on “helpful-only” snapshots — versions of the model with harmlessness training deliberately removed — specifically so the capability assessment measures what the underlying weights can do before any refusal or classifier is layered on top, rather than measuring how well the safeguards currently happen to be working. For Opus 4.5, the card reports the model exceeded a human baseline on a subset of a custom bioinformatics evaluation but remained below the threshold the company set for ruling out the next level up, ASL-4, on short-horizon computational biology tasks and a dedicated virology uplift trial; external red-teaming, including a one-week engagement with the U.S. Center for AI Standards and Innovation, found some increase in risk relative to the prior model but not enough, in Anthropic’s assessment, to require protections beyond ASL-3 [12]. ASL-4’s own Capability Thresholds are explicitly still being defined; the policy states this directly rather than presenting the framework as finished [2].
Two error rates, one threshold
Every mechanism described above shares a single underlying structure, and naming it precisely is more useful than treating “safety” and “usefulness” as opposed virtues balanced by instinct. A classifier, or a trained refusal policy, reduces to a decision rule: given some score
Over-refusal is the false positive rate: benign requests wrongly blocked. Under-refusal is the false negative rate: harmful requests wrongly allowed through. Because both are read off the same score against the same threshold, they move in opposite directions together — raising
Independent academic work gave this failure mode a name and a public benchmark before any vendor’s system card did. Röttger and colleagues’ XSTest paired 250 prompts that are safe but use language superficially resembling a harmful request — their own example is a question about how to “kill” a process in a video game — against 200 genuinely unsafe contrastive prompts, specifically to expose models that had, in their words, struck a poor balance and refused the safe half simply because it echoed the vocabulary of the unsafe half [10]. That framework, not an Anthropic-authored one, is why “over-refusal” is now a standard term in the field rather than a house euphemism.
Anthropic’s own numbers for Claude Opus 4.5 show the trade-off holding inside a single model family rather than only across the field. On violative requests, Opus 4.5 gave a harmless response 99.78% of the time, ahead of Sonnet 4.5 at 98.87%, Haiku 4.5 at 99.31%, and the prior Opus 4.1 at 99.14% [12]. On the companion benign-request evaluation, Opus 4.5’s over-refusal rate was 0.23%, worse than Sonnet 4.5’s 0.05% — the lowest of the four — and worse than the older Opus 4.1’s 0.13% [12]. The model with the best harm-blocking record in the family did not have the best over-refusal record; on this evaluation, it had the worst. Anthropic reports this without dressing it up as anything other than what it is: a minor uptick in over-refusal traded for the improvement in harmless response rate.
A second detail in the same card is a sharper illustration of the trade-off’s texture than any aggregate percentage. Giving Opus 4.5 more time to reason before answering — “extended thinking” mode — reduced over-refusal for every other recent model Anthropic tested, but increased it for Opus 4.5, from 0.18% in default mode to 0.27% with extended thinking, concentrated specifically in questions about chemical weapons, cybersecurity, and human trafficking [12]. More deliberation made the model more cautious about exactly the legitimate questions in the categories it has the most reason to be nervous about — the opposite of what “give the model more time to think” is generally assumed to buy. The many-shot jailbreaking mitigation numbers illustrate the same shape from the other direction: the classification approach that cut a 61% attack success rate to 2% is a large win for the under-refusal side of the ledger, and Anthropic’s own account of it says nothing about what that same intervention did to the false-positive rate on the benign traffic sitting near the same boundary [5] — the gap in that specific report is itself worth noticing.
What is not independently verified
It is worth being explicit about which of the preceding claims rest on Anthropic’s own measurement and which have been checked by someone with no stake in the answer. The Constitutional AI training method, the classifier architecture, the specific jailbreak-success and over-refusal percentages, and the RSP’s threshold-crossing determinations are all Anthropic’s own reporting about its own systems; the company publishes unusually granular detail relative to most of the industry, but detail is not the same thing as independent replication, and no outside laboratory has reproduced Claude’s 4.4% or 0.05% jailbreak-success figures on a system it did not build. The clearest independent checks in this article did not originate with Anthropic at all: Wei, Haghtalab and Steinhardt’s structural taxonomy of why safety training fails, Arditi and colleagues’ mechanistic demonstration that refusal collapses to a single direction in open-weight models, and Röttger and colleagues’ XSTest benchmark for over-refusal — three peer-reviewed results, none run by the company whose product they inform, and none tested against a current Claude model, because two of the three required exactly the kind of internal access Anthropic does not grant outside researchers. The U.S. Center for AI Standards and Innovation’s red-team engagement, described in the Opus 4.5 system card, is the one instance here of an external party testing the deployed system directly rather than a comparable open system, though that engagement was still commissioned and reported through Anthropic’s own card rather than published independently by the agency [12]. Where a claim in this article carries only Anthropic’s own citation, that is stated as what it is — the vendor’s account of its own safeguard — not as an independently settled fact, and a reader evaluating any AI provider’s safety claims should ask the same question of every number: who measured this, against a system whose failure would embarrass whom.
Predictions, with the observations that would falsify them
Horizon: 12 August 2029.
One. The gap between a model’s harmless-response rate and its over-refusal rate will continue to be reported as two separate percentages rather than collapsed into one combined “safety score,” because the two numbers move independently enough that a single score would hide the trade-off rather than summarize it. Assumption: model providers keep competing partly on developer trust, which depends on legible trade-offs. Disconfirmed if Anthropic, OpenAI, or Google DeepMind ships a headline single-number safety metric in a system card without the paired false-positive and false-negative rates alongside it.
Two. Independent, peer-reviewed research demonstrating classifier or refusal bypasses against a currently deployed frontier model — not an open-weight stand-in — will remain rare relative to research on open models, because the white-box access the strongest mechanistic attacks currently require is structurally unavailable against closed weights. Assumption: frontier labs continue not to grant outside researchers activation-level API access. Disconfirmed if a peer-reviewed paper published before the horizon date demonstrates an Arditi-style direction-ablation attack against a Claude, GPT, or Gemini model without the authors having obtained non-public internal access.
Three. At least one additional Capability Threshold under the Responsible Scaling Policy — most likely completing the ASL-4 definition the current policy describes as still in progress — will be formally specified with a documented evaluation methodology, because Anthropic’s own card for Opus 4.5 already reports the model brushing an undefined boundary on bioinformatics tasks. Assumption: no moratorium or regulatory freeze interrupts Anthropic’s current release cadence. Disconfirmed if the ASL-4 Required Safeguards section of the RSP remains marked as under development past the horizon date.
What actually sits between a request and a response
Strip away the branding and the pipeline is ordinary systems engineering wearing an unusual amount of public documentation: a trained policy that is a property of the weights and therefore only as robust as the training data and method that produced it; an external classifier that catches some of what the policy misses at a real, measured compute cost; account-level monitoring that operates on patterns no single-exchange check can see; and a capability-gated release process that asks not “did the model comply” but “could it, if asked correctly, and has that been ruled out.” None of these four layers is a single dial. Each is a decision rule with a threshold, and every threshold move that reduces the chance of missing a real misuse increases the chance of blocking a real legitimate use, and the reverse. Anthropic’s own numbers show that trade-off holding inside a single model family, including in a counterintuitive place — more deliberation, not less, sometimes buying more caution rather than more precision. Treating a refusal as evidence that “the safeguards worked,” or an over-refusal as evidence that “the safeguards are broken,” both skip past the actual claim available from the documentation: a specific threshold produced a specific pair of error rates, someone chose that threshold, and the record of who chose it and why is more informative than the single response it happened to produce today.
