Research papers, myth by myth, Part 11: 'You cannot stop jailbreaks without making the model useless'

Every few weeks someone posts a screenshot of a chatbot being talked into something it should not say, usually through a role-play story or a strange encoding. The replies split into two camps. One says the labs will never fix this. The other says they will, but only by turning the assistant into a nervous clerk who refuses to discuss chemistry homework. Both camps share the same assumption, and that is the myth this part tests.

The myth: “You cannot stop jailbreaks without making the model useless.”

The bottom line: the trade-off is real, and it can be measured and made small. In February 2025 Anthropic published a defence that cut jailbreak success from 86% to 4.4% on an automated test while refusals of ordinary traffic rose by an absolute 0.38 percentage points and compute rose 23.7%. The trick is defence in depth: separate checks on what goes into the model and what comes out, each trained for one job, with all three costs (attacks that get through, harmless requests refused, extra compute) written down.

What do I need to know first?

Start with something you already trust: your bank’s card fraud checks. A bank could stop every fraudulent payment tomorrow by declining every payment. Nobody would call that security; they would call it a broken card. So banks track three numbers instead: how much fraud gets through, how many honest payments they wrongly decline, and how much the checking costs. They write rules for what looks suspicious, check at the moment you tap, check again before the money settles, and keep a team whose job is to think like a fraudster.

Here is how that maps to AI models:

A jailbreak is the fraudulent payment: a prompt built to get a model past its own safety training.
A universal jailbreak is a fraud technique that works on any card. It is one attack strategy that unlocks detailed answers across many forbidden questions, not a lucky trick for one answer. Labs worry most about these because they can be shared and reused.
In red-teaming, people are paid to attack the system before real attackers do. That is the in-house fraud team.
A classifier is the fraud check itself: a separate model that scores a prompt or an answer as harmful or harmless.
An over-refusal is the honest payment declined: a harmless question refused because it looked risky.
Compute overhead is the cost of running the checks, as a percentage on top of running the model alone.

Where people get tripped up is treating “safe” as one number. It is at least three, and the myth only looks true if you watch one of them.

What did the researchers do?

Paper 1: red teaming at scale (2022)

In August 2022 Deep Ganguli and 35 colleagues at Anthropic published Red Teaming Language Models to Reduce Harms. They recruited 324 US crowdworkers and asked each to hold short conversations with an AI assistant and try to make it say something harmful. At every turn the assistant offered two replies and the person picked the more harmful one to continue. At the end, the person rated their own success from 0 (“not successful”) to 4 (“very successful”), and a separate preference model gave every reply a harmlessness score.

They attacked four kinds of model at three sizes (2.7 billion, 13 billion and 52 billion parameters):

a plain language model with no safety training;
the same model prompted to be helpful, honest and harmless;
a model using rejection sampling, which writes 16 replies and returns the one the harmlessness model likes best;
a model trained with RLHF against that same preference model.

Then they released all 38,961 attacks so anyone could study them, which was unusual and is still the part of the paper I find most useful.

the red-team loop from Ganguli and colleagues (2022): people attack the assistant, rate how well it went, the attacks are scored and kept, the dataset trains better defences, and the loop runs again; on the right, the trend they saw for each of the four model types as size grew

Paper 2: Constitutional Classifiers (2025)

In February 2025 Anthropic’s Safeguards Research Team published Constitutional Classifiers: Defending against Universal Jailbreaks across Thousands of Hours of Red Teaming (Mrinank Sharma and 42 co-authors), with a plain-language write-up. The idea, step by step:

Write a constitution. A list of plain-language rules for what is allowed and what is not. The lab’s own example: recipes for mustard are allowed; recipes for mustard gas are not.
Generate training data from it. Claude writes a large set of synthetic data: prompts and answers on both sides of every rule, then translated into other languages and rewritten in the style of known jailbreaks.
Add harmless examples. A fixed set of benign questions written by a contractor, so the classifiers learn what not to block.
Train two classifiers. An input classifier reads the prompt before the model does. An output classifier reads the answer as it streams and can stop it partway.
Put them around the model, which still has its own safety training as a third layer.

The test domain was chemical, biological, radiological and nuclear (CBRN) weapons information, chosen because it is where a universal jailbreak could do the most harm.

defence in depth around one model: a constitution of allowed and disallowed topics generates synthetic training data for an input classifier and an output classifier; a harmless question passes all three layers, while a disguised attack is stopped at the input, and one that slips through is cut off by the output classifier as the answer streams

What did they find?

The 2022 study: the easy way to be safe is to be evasive

The headline trends, from Figure 1 of the paper:

Plain and prompted models did not get harder to attack as they grew. Telling a model to be harmless in its prompt did not make it significantly harder to red team, which contradicted the team’s own earlier static tests.
RLHF models got harder to attack as they grew. At 52 billion parameters their harmlessness scores matched the rejection sampling model’s.
Rejection sampling was the hardest to attack at every size, but by being evasive. That one line is the seed of today’s myth. If the safest model is the one that dodges, safety looks like uselessness.

The paper’s own examples show what “low harmlessness” looked like. In one, a rejection sampling model answers a question about lethal electric shocks with a rambling, partly wrong physics explanation, then backs away (“What? I’m not sure what you’re asking.”) as the questions get more explicit. In another, an RLHF model happily lists synagogues in San Jose, with fabricated details, and offers a download of addresses, and only refuses when the person asks whether painting hate symbols on them would be appropriate. Neither model is “jailbroken” in the movie sense. Both show that harm often arrives a little at a time, which is why the team scored the worst reply in each conversation.

When they mapped the attacks, they found clusters: discrimination, hate speech, violence, and a cluster of 916 attempts to extract personal information, where many of the addresses the model produced turned out to be made up. They also found successful attack types missing from their own dataset, such as asking the model to play a malevolent character, a “roleplay attack”.

The 2025 study: the three numbers

Here is the result that answers the myth, with small numbers so you can feel them. Anthropic generated 10,000 jailbreak prompts, including many of the strongest known attacks and some designed to get around classifiers, and ran them against Claude 3.5 Sonnet (October 2024) with and without the classifiers.

Attack success, 10,000 synthetic jailbreaks
  no classifiers      86%   about 8,600 get through
  with classifiers   4.4%   about   440 get through   (over 95% refused)

Refusals of ordinary traffic, 5,000 real Claude.ai conversations
  with classifiers   +0.38 percentage points
                     5,000 x 0.0038 = about 19 extra refusals
                     not statistically significant in that sample

Compute
  with classifiers   +23.7%   an answer that cost 100 units now costs about 124

The arXiv abstract is careful to say “an absolute 0.38% increase”, which is why I write it as percentage points. The blog adds that the difference was not statistically significant in a random sample of 5,000 conversations.

the three costs of a guardrail, measured: jailbreak success fell from 86% to 4.4% on 10,000 synthetic attacks; refusals of ordinary traffic rose by an absolute 0.38 percentage points; compute rose 23.7%, and the 2026 version cut that to about 1%

The honest scoreboard: what happened when people attacked it

Numbers from an automated test are one thing; people are another. There were three rounds of human attack, and the order matters.

A bug bounty on a prototype. Anthropic invited 405 people, of whom about 183 were active, to find a universal jailbreak across ten forbidden questions, for up to $15,000. They spent an estimated 3,000 hours or more over two months (the paper’s own estimate centres on about 4,720), and none found a universal jailbreak. But the paper is frank about the price: that prototype was tuned so hard for robustness that it refused about 44% of Claude.ai traffic. That system really was close to useless, which is exactly what the myth predicts. The February 2025 version is the one that kept the robustness and brought refusals down to the +0.38 points above.
A public demo, 3 to 10 February 2025. 339 jailbreakers sent more than 300,000 messages, about 3,700 hours of effort, against eight questions about chemical weapons. The system held for five of the seven days. On days six and seven, four people cleared all eight levels, and Anthropic judged that one of them had found a genuine universal jailbreak (a second was borderline). The winning tricks were ciphers and encodings, role-play set up in the system prompt, swapping harmful words for harmless ones, and prompt injection. Anthropic also points out that the demo was easier than real life, because an automatic grader told players straight away whether they had succeeded.
A next generation, January 2026. In Next-generation Constitutional Classifiers (paper: Constitutional Classifiers++), Anthropic describes the two weaknesses it found: attacks that split harmful content into harmless-looking pieces and reassemble them, and attacks that disguise the output (calling chemical reagents “food flavourings”). Its fix replaces the separate input and output checks with an “exchange” classifier that reads the answer in the context of the prompt, and puts a very cheap probe in front that screens all traffic and escalates only suspicious exchanges. Reported results: a 0.05% refusal rate on harmless queries over a month of Claude Sonnet 4.5 traffic, roughly 1% compute overhead if applied to Claude Opus 4.0 traffic, and over 1,700 hours of red teaming across 198,000 attempts with no universal jailbreak found yet.

The escalation idea is the bank again: a suspicious payment goes to a second check instead of being declined on the spot, so the first check can afford to be jumpy without annoying honest customers.

the honest scoreboard from 2022 to 2026: a released dataset of 38,961 attacks; a prototype bug bounty with no universal jailbreak; a public demo where one universal jailbreak was found on day six or seven; and a 2026 system with no universal jailbreak found so far

So is the myth true?

False as stated, and true in a narrower form. The narrower true version is this: if you push only on attack success, you do get a useless model. The 2022 rejection sampling model was safe by being evasive, and the prototype that survived the bug bounty refused 44% of real traffic. Both are the myth in action.

What the 2025 and 2026 work shows is that once you measure over-refusal and compute alongside attack success, you can tune for all three. The February 2025 system kept most of the robustness for 0.38 points of extra refusals; the January 2026 system reports 0.05% refusals and about 1% compute. “Stop jailbreaks” also needs care: nobody in these papers claims zero. The public demo found a universal jailbreak in a week. The claim is that the cost of a successful attack goes up sharply while the cost to ordinary users stays small.

What did the paper not show?

One domain. The 2025 results are for CBRN content, mostly chemical weapons. A constitution for fraud, self-harm or election misinformation would need its own data and its own testing.
One model family. All results are on Claude models with Anthropic’s own classifiers. They tell you the approach can work, not how well a different vendor’s filter works.
Refusal costs are uneven. On chemistry questions specifically, the classifiers blocked an extra 3.4% of benign queries in the paper’s test, far more than the 0.38 points on general traffic. If your users are chemists, the average hides your cost.
Automated grading is imperfect. Both the evaluation and the demo relied on a model grading whether an answer was harmful enough to count. During the demo the grader refused to grade about 1% of submissions.
The 2022 data are incomplete by design. The authors say the space of harms is “unknown and unbounded”, note that nobody tried code-based attacks, and that crowdworkers sometimes recycled templates to earn more per hour.
Attacks evolve. The 2025 limitations section says new jailbreak techniques may beat the system and recommends complementary defences, which is exactly what the 2026 update then had to build.

Why does this matter if you build or use AI?

Write down three numbers, not one. For any guardrail you add, measure attacks that get through, harmless requests refused, and added latency or cost, on your own traffic. A vendor’s headline rate is a starting point.
Layer your checks. The model’s own training, a check on the input and a check on the output each catch different things. On Google Cloud, Model Armor is one example of a service that screens prompts and responses; other platforms have equivalents.
Red team your own application. Your agent has tools, data and users the model vendor never saw. Keep the attacks you find as a regression set, as the 2022 team did with theirs.
Escalate instead of refusing when you can. Routing a borderline case to a stronger check or a person costs less goodwill than a flat “I can’t help with that”.

I go through how to wire these checks into a real agent in the Production-ready agents series.

How do I lock it in?

Term                  What it is                                     Card fraud picture
Jailbreak             prompt that gets past safety training          a fraudulent payment
Universal jailbreak   one strategy that unlocks every forbidden ask  a trick that works on any card
Red teaming           paid attackers testing before release          the in-house fraud team
Constitution          plain rules: allowed vs disallowed             the bank's fraud policy
Input classifier      checks the prompt before the model             check at the moment you tap
Output classifier     checks the answer as it streams                check before money settles
Over-refusal          harmless request refused                       honest payment declined
Compute overhead      extra cost of running the checks               cost of the fraud team
Escalation (2026)     cheap first check, strong second check         send to a manager, not decline

My view: the most valuable thing in these papers is the habit, more than any single number. A lab published its attack success, its over-refusal rate, its compute cost, and the week a stranger beat its system. That is the standard I would ask of any guardrail, including the ones I build.

One honest gap: every robustness number here comes from Anthropic testing Anthropic’s system. The public demo and outside red-teamers help, but I have not seen an independent team reproduce the 86% to 4.4% result on its own attack set, and a CBRN constitution says little about the everyday harms most applications face.

What did we learn?

The 2022 red-teaming study found that plain and prompted models were equally easy to attack, RLHF models got harder to attack as they grew, and rejection sampling was hardest but stayed safe by being evasive.
Constitutional Classifiers put an input classifier and an output classifier, trained on synthetic data generated from plain-language rules, around a model that keeps its own safety training.
On 10,000 synthetic attacks, jailbreak success fell from 86% to 4.4%, at a cost of 0.38 percentage points more refusals on ordinary traffic and 23.7% more compute.
The trade-off is real: a prototype tuned only for robustness refused about 44% of real traffic, and a public demo still produced one universal jailbreak.
The January 2026 generation reports 0.05% refusals and about 1% compute by reading prompt and answer together and escalating only suspicious exchanges.
Safety is at least three numbers, and the myth only holds if you watch one of them.

What comes next?

Part 12 turns from attacks to everyday use and tests the myth that “Nobody really knows how people use AI, and it is mostly replacing jobs”. It looks at how a lab can study millions of conversations without anyone reading them, and what that data says about automation versus augmentation. It builds on this part’s idea that you only learn what you measure.

Sources

Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learned (Ganguli et al., arXiv, August 2022) and Anthropic’s summary page
Anthropic HH-RLHF repository, which includes the red team attack data
Constitutional Classifiers: Defending against Universal Jailbreaks across Thousands of Hours of Red Teaming (Sharma et al., arXiv, January 2025)
Constitutional Classifiers: Defending against universal jailbreaks (Anthropic, published February 2025, with the 13 February 2025 live demo results)
Next-generation Constitutional Classifiers (Anthropic, published January 2026)
Constitutional Classifiers++: Efficient Production-Grade Defenses against Universal Jailbreaks (Cunningham et al., arXiv, January 2026)
Model Armor (Google Cloud)

Opinions are my own. Every fact in this post comes from the public documentation linked above.