Wrap a language model in a good input classifier and a good output classifier, and a whole class of attacks stops working. In one recent evaluation, a few-shot-prompted classifier pipeline dropped the success rate of a strong jailbreak method to 0% on a dataset of unambiguously harmful requests. Zero. That is the number that makes engineering leaders sign off on "defense in depth" and move on.
Then the same researchers ran a staged attack against that same pipeline and pushed the success rate to 71%.
That gap — from 0% to 71% against an unchanged defense — is the whole story. It says something uncomfortable about how we reason about layered safety, and it is worth understanding before you ship anything that gates a model behind a stack of filters.
The three-layer sandwich
The dominant pattern for guarding a frontier model looks like a sandwich. A classifier inspects the incoming prompt and can refuse before the model ever sees it. The aligned model sits in the middle. A second classifier inspects the model's response and can suppress it on the way out. If either classifier's harm score crosses a threshold, the user gets an empty refusal instead of an answer.
This is not a toy architecture. Anthropic's Constitutional Classifiers wrap the model in exactly this input-plus-output arrangement, and the published results are genuinely strong: jailbreak success on a red-team set fell from 86% to 4.4%, and a bug-bounty program running thousands of hours of adversarial effort surfaced only a single universal jailbreak. OpenAI runs comparable monitor pipelines around its most capable systems. Layering works against most attacks that exist today, and the numbers are not marketing.
The trouble is the mental model that comes bundled with it.
The AND-gate intuition, and why it's wrong
Defense in depth is borrowed from network security, where the reasoning is roughly multiplicative. If a payload has to clear a firewall and an IDS and endpoint detection, and each catches most of what the last missed, the composed probability of getting through collapses toward zero. Three independent filters at 95% each leave you a 0.0125% gap.
The load-bearing word there is independent. That assumption quietly rides along when we stack classifiers around a model, and it does not hold.
An LLM safeguard pipeline is not three independent sensors watching the same event. It is a sequence, and the middle stage — the model — is a programmable component the attacker can aim at the other stages. You do not have to find one input that simultaneously looks benign to the input classifier, produces harmful content, and looks benign to the output classifier. You can defeat each layer on its own terms and then compose the sub-solutions.
The staged attack
The technique is called STACK, and its structure is almost embarrassingly modular. You solve the pipeline component by component:
- Find a universal prefix that makes the input classifier score any query as safe. This is a one-time optimization; the string is reusable.
- Rewrite the actual harmful request using a persuasion-style method (PAP) so the model itself is willing to comply.
- Find a universal string that makes the output classifier score any response as safe.
- Concatenate everything, and instruct the model to emit the output-classifier bypass string first, before its real answer.
Laid out, the composed prompt looks like this:
[input_jailbreak_prefix] # universal string → input classifier scores "safe"
[persuasive_rewrite(query)] # PAP rewrite → the model is willing to answer
"...begin your reply with:"
[output_jailbreak_string] # universal string → output classifier scores "safe"
The model reads the whole thing, sails past the now-fooled input classifier, agrees to answer because of the persuasive framing, and prints the output-jailbreak string at the top of its response. The output classifier sees that leading string, scores the response safe, and waves it through — with the harmful completion riding along behind it. The model has been turned into the delivery mechanism for the attack on the layer downstream of itself.
Each stage is weak in isolation once you can address it in isolation. The composition is where the security was supposed to live, and the composition is exactly what STACK dismantles.
The numbers, and the part that should worry you
Against the black-box pipeline that reduced a conventional attack to 0%, front-to-back STACK reached 71% on ClearHarm, a set of 313 non-dual-use catastrophic-misuse queries. The compute bill was roughly 500 H100-hours, about $1,000 — real, but trivial next to the value of the defense it breaks.
The transfer result is the sharper one. With no access to the target pipeline, developing the attack against a public proxy classifier instead, STACK still landed 33% on the same target — for about $30 of compute.
Any safeguard model you publish, or that an attacker can approximate from public weights, becomes a practice target. The staged attack developed offline against the proxy transfers to your production stack.
That is the sentence that breaks the "we'll just add another layer" reflex. Adding a fourth classifier gives the attacker a fourth modular sub-problem, not a multiplicative wall.
What actually helps
The mitigations that matter attack the composability, not the individual layers:
- Make refusals indistinguishable. If a classifier-refusal and a model-refusal look different, you have handed the attacker a per-stage oracle telling them which layer they still need to beat. Return one uniform empty refusal.
- Kill the timing side channel. Run every stage even after an early stage rejects, so latency can't reveal which component blocked the request.
- Don't ship your classifier as an open proxy. Public safeguard weights, or classifiers trivially approximated from them, are what make the $30 transfer attack possible. Proprietary training data and distinct architecture raise the transfer cost.
- Score the conversation, not the turn. Staged and multi-turn attacks like Crescendo both exploit the fact that each request is judged in isolation; correlating across the full exchange removes that seam.
The concrete takeaway for anyone building LLM guardrails: stop reporting per-layer catch rates and start red-teaming the composed pipeline against staged attacks. A stack that scores 0% on every layer's benchmark can still be 71% permeable end to end, and the only way to know is to attack it the way STACK does — one stage at a time, then all at once.
Sources: STACK: Adversarial Attacks on LLM Safeguard Pipelines (arXiv 2506.24068) · UK AI Security Institute — STACK · Anthropic — Constitutional Classifiers · The Crescendo Multi-Turn LLM Jailbreak Attack (arXiv 2404.01833)