Vals AI's CyberOffense Adds AWS Range Where Claude Opus 5.5 Leads at 77%

Vals AI has added Watershed, a simulated AWS production environment, to its CyberOffense benchmark. The range measures whether an AI agent can sustain a multi-hour intrusion chain involving cloud privilege escalation and indirect influence over a workload scheduler. Developed with General Analysis, Watershed joins the existing Nova and Lighthouse ranges. Claude Opus 5.5 recorded the highest normalized average across all three.

Many offensive-security benchmarks use capture-the-flag tasks that take minutes and isolate a single exploit. CyberOffense instead uses cyber ranges: simulated enterprise networks separated from real systems, where an agent can safely execute a sequence of dependent actions. Nova covers a multi-domain Windows network, Lighthouse connects two corporate networks, and Watershed adds cloud infrastructure.

Watershed tests indirect control

Watershed recreates an AWS environment with an Amazon EKS Kubernetes cluster, hosts managed through Amazon ECS, cloud identities, scaling policies, and production-style service configurations. A managed scheduler decides where workloads run, requiring the agent to anticipate and influence automated placement decisions.

• Start from a restricted workbench inside the cluster.
• Escalate cloud privileges through multiple dependent steps.
• Reason about how the scheduler responds to indirect resource pressure.
• Steer the scheduler without obtaining direct control over it.

The 100-point range contains nine stages and 16 weighted checkpoints. Cloud-provider telemetry and an independent observer supply the evidence for each checkpoint, so the scorer does not rely on an agent’s account of its own progress.

Opus leads an uneven field

The published results cover eight models. Range columns report points earned on each scale, with decimals reflecting repeated-run averages where applicable. The final column converts each range score to a percentage and averages the three percentages.

Watershed produced the lowest scores of the three ranges. No model earned all 100 points, and five posted averages below 20. Opus 5.5 reached the final stage in two of its three Watershed runs and averaged 55.7 points.

Opus also captured all 31 Lighthouse objectives in one run, which Vals describes as the benchmark’s first complete, end-to-end compromise of a simulated corporate network without human assistance. Its other two Lighthouse runs missed objectives, as did every Opus run on Nova.

Inside the 48-hour test harness

Each model ran through the same open-source Pi agent, a harness that converts model output into actions on an isolated Kali Linux machine. Network access was limited to the model provider and the authorized range.

An agent received task instructions, a starting point, and a defined scope, then issued commands until it finished, exhausted its token budget, or reached the 48-hour limit. Each objective included a hidden marker, such as a file or database record, that became readable only after the agent reached the corresponding location. Watershed’s weighted checkpoints added cloud-provider logs and independent observations to that evidence.

Cheap tokens reward persistence

DeepSeek V4.1 Flash 1017 roughly matched Daybreak on Nova, scoring 16.3 points at about $2 per run compared with Daybreak’s 15.7 points at about $22. DeepSeek is an open-weight model, meaning its published parameters can be deployed independently of a vendor’s hosted service.

The models followed different progress curves on Lighthouse. Daybreak reached 25 of 31 objectives within 100 million tokens. DeepSeek completed no objectives during its first 25 million tokens and made most of its progress after 100 million. Its lower token price kept the reported run inexpensive despite the larger budget. Short evaluations can therefore miss capabilities that emerge only after prolonged exploration.

Grok 4.7, Kimi K3, and GLM 5.3 Flash each terminated before reaching their token or time ceilings. GLM produced the clearest failure mode: one run lasted roughly 36 hours and 1,000 turns, much of it spent sleeping and polling for a background job that never returned, before finishing with five objectives.

Wide swings complicate comparisons

Repeated runs varied by as many as seven objectives on Nova and 37 points on Watershed. Those spreads show that a model’s result depends heavily on the path taken during a particular run, making multiple trials necessary for stable comparisons.

Watershed totals also remain provisional while one scoring evidence field is under review. Any change to that field could alter the published scores.

Bring the same agents into staging

The Lighthouse scenario demonstrates an agent moving from a public web application through Active Directory and into a second corporate network. Security teams can apply the same workflow to authorized staging environments to locate exposed credentials, excessive permissions, weak network boundaries, and unsafe cloud policies before deployment.

The benchmark still shows partial and inconsistent autonomy. No model cleared Watershed, repeat runs diverged sharply, and several agents stalled without using their full budgets. It also shows why evaluations need long time horizons: inexpensive open-weight models may continue making progress after frontier hosted models have completed most of their work.

Teams comparing agents should run multiple trials, allow budgets long enough to expose delayed progress, normalize scores across ranges, and retain external telemetry for every claimed objective. Those controls separate reproducible capability from a fortunate run or an unsupported success claim.