Factory's Droid Shield 2.0 Beats GPT-5.5 at Catching Secret Code Leaks

When an AI agent is writing and committing code autonomously at enterprise scale, no human reviewer can keep up. That's the reality Factory is building for, and it's why secret detection can't be an afterthought. Droid Shield 2.0 is Factory's answer: a learned, context-aware security layer that sits on top of their existing pattern-based scanner to catch what rules alone can't.

The problem with rules-only scanning

Because the scanner is deterministic, it suffers from two failure modes: false positives that fire on placeholders, examples, fixtures, and non-secret identifiers, creating friction and training users to ignore the flag; and false negatives that miss real secrets not matching its fixed pattern set, reducing confidence for fully autonomous usage.

Both failures are costly in different ways. False positives erode trust until engineers start overriding warnings reflexively. False negatives are the silent catastrophe: an API key or database credential slipping into version control history. Users have noted that the original Droid Shield could be overly aggressive, "flagging things that weren't actually secrets but secret adjacent like names of vars."

Two models, two failure modes

Rather than replacing the deterministic scanner, Factory wraps it with two specialized fine-tuned models, each targeting one failure mode. Think of it as a three-stage pipeline: the rule-based scanner in the middle, with one model watching each exit.

• Risk model: Runs when the scanner did not fire, but the changed line still looks secret-bearing within broader context. The catastrophic error is missing a real secret, so this model is optimized for recall, accepting some extra warnings to catch more true positives.
• Downgrade model: Runs when the scanner did fire. Every detected secret candidate is masked before the model sees it, so the model must decide from context alone whether the scanner hit should stay blocked or be cleared as a false alarm.

Each model emits a binary verdict plus a short user-facing reason, and is scored according to the model's block probability via exposed token probabilities around this verdict. This is a tunable threshold that requires no retraining for reconfiguration.

How they were built

Both models are fine-tuned from Qwen 3.6 35B A3B, chosen for its coding strength, reasoning capabilities, and improved cost/latency compared to the frontier. The final adapters are LoRA fine-tunes: a rank-16 adapter for Risk and a rank-64 adapter for Downgrade. The rank difference matters: rank 64 showed significant improvement over lower-rank adapters on the downgrade evaluation set, with ROC-AUC jumping from 0.697 at rank 8 to 0.845 at rank 64.

Training data is shaped from Samsung's CredData, a public benchmark of real source files and annotated credential spans. Factory hand-labeled a batch of unknown samples to improve judging quality, using these as grounded reasoning examples for further data pipeline steps. An LLM judge ensemble (GPT-5.5 as primary, Opus 4.8 adjudicating) assigns training verdicts and reasoning, anchored on the hand-labeled batch to minimize labeling errors.

The Risk training set contains 5,000 rows across passwords, keys, tokens, and secrets, skewing toward passwords since they have no fixed entropy or format. The Downgrade set contains 6,776 rows, skewing toward keys and secrets since those are what the deterministic scanner most commonly catches.

One important privacy note: real sessions are never training rows. The only production signals used are aggregate class priors sampled across enough distinct sessions to prevent any identifying information.

Beating frontier models at a fraction of the cost

The headline result is that these smaller, task-specific adapters outperform GPT-5.5 and Opus 4.8 on both tasks. Here's how the numbers break down:

Task	Model	ROC-AUC
Risk	Fine-tuned LoRA	0.975
Risk	Opus 4.8	0.961
Risk	GPT-5.5	0.948
Downgrade	Fine-tuned LoRA	0.845
Downgrade	GPT-5.5	0.819
Downgrade	Opus 4.8	0.800

Adjusted to the models' best operating points, they are the strongest or tied-strongest classifiers on both tasks compared to the frontier, at a fraction of the cost and with a significant improvement in latency. At the strict false-positive budget (FPR ≤ 0.05), the Risk adapter reaches 0.698 recall versus GPT-5.5's 0.588, with non-overlapping 95% confidence intervals.

What it still can't do

Factory is transparent about the limitations. On the Risk side, the candidate pool is enriched by secret-name keywords, meaning the model sees more deterministic secret-shaped lines than true semantic similarities. On the Downgrade side, because the model has to judge from context where every detected value is masked, there's a hard limit on the signal that can be included in the data. Enriching the code window with non-sensitive value shape and entropy remains future work.

There's also a calibration caveat for the frontier comparison: GPT-5.5 and Opus 4.8 don't expose token probabilities, so they have to be tuned on self-reported confidence rather than the calibrated score read from Factory's own models. This introduces some noise in the comparison.

The real shift: security that understands context

The deeper story here is about what becomes possible when AI agents write code at scale. As developer workflows increasingly rely on autonomous development agents, new risks emerge: prompt injection, sensitive data loss, and unsafe URL access, which, if left unaddressed, can undermine the very benefits these agents offer.

Traditional secret scanners were designed for human-paced commits. A developer pushes a few times a day; a pattern miss is recoverable. Factory's Droids write, refactor, and commit code autonomously at a volume that no human reviewer can completely monitor. At that velocity, a single missed pattern in the scanner's ruleset becomes a systematic vulnerability. Context-aware models that understand whether a string is a real credential or a test fixture are the only way to keep the false-negative rate manageable without drowning engineers in false alarms.

This also signals a broader architectural pattern: rather than asking a frontier model to do everything, Factory is showing that small, specialized adapters fine-tuned for a narrow task can outperform general-purpose giants on that task, while being faster and cheaper to run in a hot path like a pre-commit hook.

How to get it

Both model weights are available on Hugging Face: the Risk adapter (rank-16 LoRA) and the Downgrade adapter (rank-64 LoRA). Both are PEFT LoRA adapters on top of Qwen 3.6 35B A3B and can be loaded with any stack that supports PEFT LoRA, including transformers + peft, vLLM, and TGI:

from peft import PeftModel
from transformers import AutoModelForCausalLM
base_model = AutoModelForCausalLM.from_pretrained("Qwen/Qwen3.6-35B-A3B")
risk_model = PeftModel.from_pretrained(base_model, "factoryai/shield-risk-r16-c15")

The model takes a JSON input with the file extension, a lines window of source code, and a focus_line index. It returns a verdict (S for safe, B for block) plus a plain-English reason. Decoding must be greedy with thinking mode disabled, and the decision threshold is not baked in, letting you tune the recall/FPR tradeoff for your own risk tolerance without retraining.

Droid Shield 2.0 is currently in research preview on the Factory platform. Organizations that want access can reach out to the Factory team directly. The open weights, however, are available to anyone who wants to run, adapt, or build on top of them today.