NVIDIA's Nemotron 3 Beats Frontier AI at Catching Unseen Cyberattacks

NVIDIA and CrowdStrike ran an experiment where AI agents attack a sandboxed network, generate telemetry, and then write their own detection rules to catch similar attacks in the future. A specialized open-model pipeline outperformed a frontier system at the one job that matters: catching attacks it had never seen before.

The joint evaluation plugs NVIDIA Nemotron models into CrowdStrike's SafeMind agentic cybersecurity platform, using Nemotron 3 Ultra for defensive orchestration paired with a fine-tuned Nemotron 3 Super for detection generation.

Closing the red-blue loop

Traditional purple-teaming bottlenecks at human handoffs. Each artifact passed across the table limits how many iterations and attack variations a team can evaluate. The agentic version collapses that into a machine-speed cycle inside a sandboxed replica of NVIDIA's accelerated computing infrastructure, running four connected stages:

1. Execute and capture. A red-agent harness selects and executes an attack path inside the representative environment while CrowdStrike Falcon endpoint sensors capture the corresponding telemetry.
2. Process and reconstruct. The blue-agent harness receives the action trace, sensor telemetry, and broader attack context to determine which existing detections triggered and where visibility gaps remain.
3. Generate and validate. The blue-agent harness generates candidate detections. A validation harness checks each candidate, backtests it against the captured telemetry, returns failures for correction, and sends validated detections to the detection engine.
4. Retest and adapt. Detection and alert context returns to the red-agent harness, which adapts and explores alternative attack or evasion paths, producing new traces for the blue-agent harness.

Six guardrails around a detection writer

A raw LLM asked to write detection rules will invent field names, over-index on IP addresses in logs, and produce queries that lint fine but catch nothing. The defensive harness wraps the model in six mechanisms to prevent that.

• A schema knowledge base lets agents enumerate supported Falcon sensor schemas, fields, and query syntax, preventing invented fields and invalid queries.
• Telemetry grounding anchors the workflow in observed events, reducing hallucinated connections.
• A customized Nemotron 3 Super serves as a bounded expert for generating and repairing detections, keeping this specialized task separate from the longer orchestration context.
• Artifact linting rejects syntax errors, unsupported fields, and detections tied to specific IP addresses, hosts, users, or subnets, returning guidance to rewrite detections around behavioral signals rather than environment-specific strings.
• Detection replay runs each candidate against captured attack telemetry and rejects any that produce no match.
• A separate judge with a fresh context evaluates each detection for behavioral alignment, robustness, and appropriate use of multiple signals.

How the specialized Super was trained

CrowdStrike used Nemotron 3 Super as the base for their NL2LogScale model, then applied continual pretraining on cybersecurity knowledge, supervised fine-tuning, and reinforcement learning with verifiable rewards. Fine-tuning used 9,349 detection-generation and multistep repair examples spanning 59 programmatically generated error types.

The RL setup rewards behavior rather than surface form. Invalid queries received real engine errors and up to five repair attempts, with unresolved attempts scoring zero reward. Valid generated and reference queries ran against the same synthetic logs, and F1 overlap between their returned events supplied the reward. The model earns credit only when its query returns the same events as a known-good reference. NVIDIA's NeMo Gym handled query validation and NeMo RL ran group relative policy optimization.

Backtest and live-fire results

Evaluation ran in two stages: backtest against the original recorded attack, then live-fire against eight fresh attacks from the same scenario family.

On backtest, Nemotron 3 Ultra with the default harness averaged 16.5% of generated detections catching the recorded attack across eight independently seeded sessions. Adding the tuned harness, customized Nemotron 3 Super, domain context, tools, and validation raised the mean to 41.9% across six sessions, a 2.5x improvement.

Live-fire produced the sharper result. Eleven backtest-passing detections from the optimized open pipeline and 35 from a complete frontier system were deployed against eight unseen attacks. Five of the 11 open detections (45%) caught at least one attack, versus 10 of the frontier detections (29%).

The stricter bar was a gold rating: detections had to catch attacks, stay quiet on benign traffic, and pass independent review for behavioral grounding. Three open detections qualified as gold; zero frontier detections did. Those three open detections covered all eight attacks.

Caveats and the reusable pattern

The frontier model produced more rules that passed the initial backtest, but most were brittle or environment-specific. The specialized open pipeline produced fewer rules that generalized better, consistent with the idea that domain post-training plus deterministic validation beats raw model scale on narrow tasks. NVIDIA acknowledges the limits: the evaluation covered one scenario family with small detection sets, so cross-scenario generalization is untested, and limited benign traffic means the noise test doesn't represent production false-positive rates.

The underlying architecture applies well beyond security. A reasoning model orchestrates the workflow, a post-trained open model handles a bounded expert task, and an agent harness manages context, tools, and validation. For any domain where outputs need execution and verification, such as SQL generation, IaC, or test writing, splitting a large orchestrator from a small RL-tuned specialist with deterministic gates is a pattern worth borrowing.