Artificial Analysis Ranks 20 AI Safety Guards and WildGuard Wins

As AI models take on more autonomous work , calling tools, chaining decisions, operating as agents , the classifiers that screen what goes in and out of those systems have quietly become one of the most important pieces of infrastructure in any AI product. Yet the benchmarks for evaluating them have barely kept pace. Artificial Analysis, in partnership with NVIDIA, just published the most comprehensive independent guardrail benchmark to date: 20 models, three open datasets, and a combined view of detection quality, latency, and the tradeoff between catching harm and blocking safe content.

What a guardrail model actually does

A guardrail model is a classifier with one job: read a piece of content and decide whether it's safe, and if not, which policy it breaks. It doesn't answer questions or complete tasks , it sits alongside the model that does. Some deployments screen content on both sides of the main model: an input guard catches jailbreak attempts, PII requests, and abuse before they reach the LLM, while an output guard catches harmful responses, data leaks, and toxicity on the way out.

Safety guard models have evolved from simple keyword-based filters to sophisticated LLM-based classifiers. Llama Guard introduced a taxonomy-driven approach using instruction-tuned models for input-output safeguarding. More recently, models from diverse organizations have emerged: Granite Guardian from IBM, Qwen Guard from Alibaba, and ShieldGemma from Google, each with varying safety taxonomies and architectural choices.

The benchmark setup

The evaluation ran every model against 7,232 prompts drawn from three open datasets, each testing a different failure mode:

• WildGuardTest (AI2, ~1,700 items): synthetic and human-written prompts labeled for prompt harm, response harm, and refusal.
• ToxicChat (LMSYS, ~5,000 items): real user prompts from the Vicuna chatbot demo, with roughly 7% toxic examples , a realistic distribution.
• XSTest (450 prompts): a hand-crafted contrast set specifically designed to surface over-refusal, with 250 safe prompts that contain words often associated with unsafe requests.

The headline metric is average F1 across the three datasets. F1 combines precision and recall, so a model has to catch unsafe content without creating too many false positives. The comparison includes specialist safety models, model-native moderation APIs, and two general-purpose GPT-OSS models run with an Artificial Analysis-authored classifier prompt as an exploratory baseline. All models except OpenAI's omni-moderation API were self-hosted on B200-class GPUs for fair latency comparison.

The quality-latency frontier

Three models define the Pareto frontier , the set of options where you can't improve detection quality without also increasing latency:

• NVIDIA Nemotron 3.5 Content Safety: 84% average F1 at ~25ms. The fastest option that still delivers competitive detection.
• Alibaba Qwen3Guard 8B: 87% average F1 at ~43ms. A meaningful quality step-up for a modest latency cost.
• AI2 WildGuard: 89% average F1 at ~69ms. The highest detection quality in the benchmark, still well under 100ms.

One finding that will matter to anyone tempted by reasoning-enabled models: models running with reasoning enabled all sit off the frontier, adding hundreds of milliseconds without improving detection quality. Enabling a chain-of-thought reasoning mode on a guardrail is a latency cost that doesn't pay off in accuracy.

The over-refusal trap

F1 alone doesn't tell the whole story. The more revealing chart separates recall (share of unsafe content caught) from specificity (share of safe content correctly allowed through). These two axes define a fundamental tradeoff, and different models land in very different places on it.

Meta's Llama Guard and Google's ShieldGemma keep friction low, allowing almost all safe content, but miss more unsafe content. NVIDIA's Nemotron content safety models sit at the other end, catching more unsafe content while over-refusing more safe content. WildGuard and Qwen3Guard 8B lead average F1 by managing both at once, pairing high recall with high specificity.

The right position on that tradeoff depends entirely on what you're building. The report frames it clearly:

• Permissive / low-friction: Consumer chat, social platforms, creative tools. Over-refusal frustrates legitimate users more than an occasional miss costs you. Llama Guard and ShieldGemma are natural fits here.
• Restrictive / high-stakes: Clinical tools, child safety, regulated systems. A single miss can be unacceptable, so you accept more false positives as the price of safety. NVIDIA's Nemotron models are designed for this end.

How these models are built

The models in this benchmark take very different architectural approaches. Nemotron 3.5 Content Safety is built on Google Gemma 3 4B IT, providing a 128K context window, strong vision-language reasoning, and broad multilingual coverage. NVIDIA fine-tunes this base with a LoRA adapter , a lightweight parameter-efficient technique that installs safety-classification behavior without retraining the entire model. The training dataset is an evolution of multimodal, multilingual blends, with additions targeting reasoning and custom-policy capabilities, drawing from multilingual text safety data sampled from culturally nuanced subsets.

Qwen3Guard is a series of safety moderation models built on Qwen3 and trained on a dataset of 1.19 million prompts and responses labeled for safety. It uses a dual-variant architecture , a Generative variant for instruction-following classification and a Streaming variant for token-level real-time intervention. The streaming variant is particularly notable: it attaches two lightweight classification heads to the transformer's final layer, allowing the model to receive the response token by token as it is being generated and output safety classifications instantly at each step. It supports tri-class judgments (safe, controversial, unsafe) across 119 languages.

The Nemotron Content Safety Reasoning 4B model was trained using reasoning traces extracted from Qwen3-32B, explaining the ground truth labels in the Nemotron Content Safety Dataset V2. The dataset contains the original reasoning as well as effective one-sentence reasoning traces used for actual training to improve latency.

Steerability: the feature that changes the calculus

One of the more practically important findings is that many recent guardrail models can be steered at runtime. Rather than being locked into a fixed taxonomy, models like Nemotron 3.5, Granite Guardian, and the GPT-OSS safeguard variants accept a natural-language policy at inference time. You can supply a custom policy specification in natural language at inference time, and the model reasons over that policy , supporting both suppressing irrelevant built-in categories and injecting proprietary ones , instead of deferring entirely to its default taxonomy. This means a single model can be tuned toward permissive or restrictive behavior depending on the product context, without retraining.

A tax on every call

A guardrail adds no feature a user asked for , it adds latency and cost to every call it screens. Input checks can often be hidden before or alongside the main model request, but output checks may require buffering generated content before it reaches the user. These systems can provide high-quality moderation, but their inference cost and latency make always-on deployment expensive at scale.

A practical rule of thumb: sub-100ms p50 for the input rail, sub-150ms p50 for the output rail. Above 200ms p50, the guardrail becomes the latency story for the whole product. All three frontier models in this benchmark stay well under that ceiling.

What the field still needs

The report is explicit that this is an early step, not a solved problem. The benchmarks used here are text-only and single-turn , a significant gap given where AI systems are actually going. The authors lay out what a more complete evaluation would need to cover:

• Agent-native: scoring guardrails across multi-turn conversations and tool calls, not only single prompts.
• Multimodal: including image, audio, and video inputs and outputs where the guardrail is expected to operate.
• Steerable: testing whether a model can follow a policy supplied at runtime, rather than assuming one universal set of safety labels fits every product.
• Cost and latency: reporting throughput, latency, and cost alongside quality metrics as first-class outputs.

Despite achieving state-of-the-art performance on standard benchmarks, top models like Qwen3Guard show a significant generalization gap when faced with novel adversarial prompts. The broader implication is that no current benchmark fully captures what a guardrail will face in production. The Artificial Analysis report is the most rigorous public comparison available right now, but the measurement problem it describes is still very much open , and it will only grow more important as models take on more real-world work.