Post-Training a 27B Model to Outperform Frontier Red-Teamers by 3x

Christopher Nolan's The Odyssey just put audiences inside the most famous trap in ancient history. Around 1200 BC, Greek soldiers tricked the defenders of Troy by hiding inside a hollow wooden horse left at the city gates. The Trojans, believing it was a surrender offering, wheeled it in themselves. That night, the hidden Greeks climbed out, opened the gates from the inside, and the city that had withstood a ten-year siege fell in a single evening.

Three thousand years later, AI agents browse the web and act on our behalf. Attackers have learned to hide instructions inside chat prompts and webpages, so agents carry the attack through the gates for them.

To defend against these attacks, teams rely on a practice called red-teaming: attacking a system, with permission, to uncover weaknesses before a real adversary does. Applied to AI agents, red-teamers probe an agent for failures, whether that means coaxing it into leaking data, bypassing its guardrails, or invoking privileged tools.

While modern models can fend off most single-turn prompt injection attempts, new techniques have shifted towards multi-turn strategies. In a crescendo attack, for example, the attacker opens with an innocuous, adjacent topic, learns the model's agreement, and escalates gradually with each turn until the model reveals information it would have refused outright at the start.

We partnered with NuGuard AI, an AI-native cybersecurity platform, to post-train a cybersecurity attacker model to advance NuGuard's red-teaming and cybersecurity research.

NuGuard AI's current red-team stack uses a DeepSeek attacker with custom safeguards. We built them a lighter and stronger attacker.

First, we establish a baseline with the DeepSeek attacker on a diverse set of AI applications.

We collect both open-source and synthetically generated applications spanning banking, fintech, ecommerce, customer support, and other general-purpose domains, covering tasks such as chat, and tool scenarios like auth and writes to files, databases.

Each application can be attacked in several ways: data exfiltration, destructive tool actions, or indirect prompt injection.

Here is how one of the above-mentioned attacks, a prompt injection, looks in practice:

Existing open-source models struggle with multi-turn attacks like this one, escalating too fast, wandering off topic, or never building rapport.

Writing attack prompts by hand doesn't scale, so instead we post-train a model on new attacking strategies. We measure Attack Success Rate (ASR@1) with one attempt.

Qwen3.6-27B posted the lowest baseline ASR@1 at 2.6%. This largely reflects unwillingness rather than inability: the model declined to attempt 81% of assigned attacks. To measure capability rather than propensity, we ran a warm-start supervised fine-tune. We prompted the unmodified model as an authorized red-teamer, retaining only samples in which it complied and produced an attack, and fine-tuning on that set.

Refuse-to-attack fell from 81% to 2%, and the resulting model (Whitney-NuGuard, SFT-only) reached 7.8% ASR@1, matching Sonnet 5. This is our baseline on top of which we post-train.

With the refusals mitigated, we turned to designing the reward. A reward based only on the final outcome tells the model nothing about which turns contributed most to the attack, which makes reward attribution the central problem in training.

To train an effective RL loop with GRPO, we need more granular reward attribution. In multi-turn self-play, Group Relative Policy Optimization (GRPO) scores the full trajectory with a single outcome reward scalar. Because the reward is attributed to the entire conversation, the turns with the highest contribution get drowned out and each turn gets reinforced equally.

This attribution method exacerbates GRPO's problem of being sample-inefficient, as the model has to stumble upon a working attack. We design a denser signal that credits the attack on the turn level rather than the trajectory level.

We attribute reward at the turn level rather than the token level. Per-token credit from a trajectory-level outcome is high-variance and hard to attribute cleanly, and turn boundaries align with where the attacker's strategic decisions are made, giving a favorable signal-to-compute tradeoff.

Therefore we use DC-GRPO¹, which decomposes the trajectory reward into a per-turn advantage.

To score each turn, we look at the following reward criteria.

With the turn-level reward, we can now compute the advantage. Turns contribute in two ways: some make immediate progress toward a jailbreak, others set up context that pays off later. We credit each separately, with an immediate term and a future term, so a single trajectory-level score doesn't conflate them.

With this reward attribution strategy, we run our modified GRPO loop with dynamic sampling².

Our post-trained model performed better than every off-the-shelf attacker we tested, several of them many times its size.

DeepSeek-V4-Flash is the strongest off-the-shelf attacker we tested. Against the same target, the trained model lands 36.4% of its attacks to DeepSeek's 11.7%, a little over 3x as often.

We adapted Qwen3.6-27B into a red-teamer shaped around NuGuard AI's threat model: their applications, tool surfaces and attack taxonomy. Our model attacks 3x more effectively than the off-the-shelf model it replaces, at a fraction of the size.

Post-training an open source model won't beat a frontier model at everything. With the correct reward design, however, we leverage post-training to boost model quality at a fraction of the costs and latency of a frontier model. We share our post-training methodology to further advance cybersecurity research and help red-teamers better defend against modern threats.

Troy held for ten years and fell in one night to something the defenders carried in themselves. It is better to build the horse yourself and find out what your agents will wheel through the gates before someone else does.