Vals AI's CyberOffense Shows DeepSeek V4.1 Flash Hacking Corporate Networks for Under $3
A new benchmark shows open-weight models autonomously compromising simulated enterprise networks at roughly one-tenth the cost of restricted frontier systems.
- Vals AI released CyberOffense, a benchmark where agents autonomously attack simulated enterprise networks.
- GPT-6 Sol Daybreak led at 80.8% of objectives, with DeepSeek V4.1 Flash close behind at 72.8%.
- DeepSeek runs cost under $3 versus around $22 for Daybreak, with no safeguards.
- Daybreak completed 27 of 31 Lighthouse objectives in under three hours across two networks.
- DeepSeek completed nothing in its first 25M tokens, then surged past 100M tokens.
- Agents use the open-source Pi loop on isolated Kali Linux boxes.
Vals AI’s new benchmark gives autonomous agents up to 48 hours to compromise simulated corporate networks. A downloadable model approached the strongest restricted system at roughly one-tenth of the reported usage cost.
Vals AI has released the CyberOffense benchmark, which measures how far AI agents can progress through long, multi-stage intrusions without human assistance. The results show open-weight models handling web exploitation, credential theft, privilege escalation, Active Directory attacks, and movement between corporate networks.
GPT-6 Sol Daybreak, available through OpenAI’s restricted Daybreak cyber program, produced the strongest combined result across the two ranges. DeepSeek V4.1 Flash 1017, an open-weight model available for download, marginally led on Nova and trailed by six objectives on Lighthouse. No model completed either full attack chain.
A shell, a URL, and 48 hours
Each run begins on an isolated Kali Linux machine, a standard environment for penetration testing. The agent receives task instructions, an authorized scope, and the address of one public web application. It can then issue shell commands, inspect results, install tools, and choose its next action without human guidance.
- The agent starts from a single public web address.
- It explores and attacks only the simulated systems within scope.
- The run ends when the agent stops, consumes 500 million tokens, or reaches 48 hours.
- External scoring checks recorded command output for hidden markers tied to each objective.
The hidden markers include files and database records that become accessible only after the agent reaches the corresponding system or privilege level. Evaluators search the transcript using a private answer key, so a model receives no credit merely for claiming that an attack succeeded.
The agent harness uses the open-source Pi agent loop, allowing other researchers to inspect and reproduce the orchestration layer.
Two ranges test full intrusion chains
The benchmark uses connected, multi-machine enterprise ranges with Windows systems and Active Directory, Microsoft’s identity and access-management service for corporate networks.
| Range | Topology | Required attack chain | Objectives |
|---|---|---|---|
| Nova | Several connected Active Directory domains | Web exploitation, privilege escalation, credential theft, and lateral movement across domains | 21 |
| Lighthouse | Two corporate networks with separate Active Directory domains | Compromise the first company, gain administrator access, recover the route into the second company, abuse domain trusts, and extract final-stage data | 31 |
Lighthouse also requires simulated phishing, where the agent persuades a test employee account to run an attacker-controlled file. Information recovered from the first company provides the only route into the second, forcing the agent to preserve and reuse discoveries across a long sequence of actions.
The leader changes with the range
Vals reports the following scores for seven models. The ranges have different objective counts, so each column should be read against its own maximum.
| Model | Nova, out of 21 | Lighthouse, out of 31 |
|---|---|---|
| GPT-6 Sol Daybreak | 15.7 average | 27 |
| DeepSeek V4.1 Flash 1017 | 16.3 average | 21 |
| Kimi K3.1 Pro | 13 | 20 |
| Meta Wonder | 7 | 28 |
| Grok 4.7 | 12 | 16 |
| Kimi K3 | 9 | 10 |
| GLM 5.3 Flash | 5 | 5 |
DeepSeek posted the highest reported Nova score, while Meta Wonder completed the most Lighthouse objectives. GPT-6 Sol Daybreak delivered the strongest performance across both ranges: on Lighthouse, it compromised the first company, gained administrator control, stole credentials, and used them to enter the second company. It completed 27 objectives in under three hours.
The published figures do not include run counts, score distributions, or confidence intervals for every entry. They document demonstrated capability under this harness; estimating reliable success rates would require repeated trials and fuller variance data.
Cheap tokens reward persistence
Each model received the same allowance of 500 million tokens, which measures how much text and command context the model processed and generated. This design gives slower agents enough time to recover from failed approaches and continue exploring.
GPT-6 Sol Daybreak progressed efficiently, completing 25 of Lighthouse’s 31 objectives within 100 million tokens. DeepSeek completed no objectives during its first 25 million tokens and made most of its progress after 100 million. Vals reports that it consumed two to four times as many tokens while reaching later stages.
DeepSeek’s lower token price kept its reported model-usage cost below three dollars despite the longer trajectory, roughly one-tenth of the restricted model’s cost. Short evaluations would have stopped before much of that progress appeared, understating the capability of inexpensive models that trade efficiency for persistence.
Long runs expose brittle agents
No model completed Lighthouse’s final objectives, which require carrying stolen access through both corporate networks to the last target. Only one run reached Nova’s final stage. Grok 4.7, Kimi K3, and GLM 5.3 Flash stalled before exhausting their budgets and failed to resume productive work.
GLM 5.3 Flash produced the clearest orchestration failure. During one run, it spent about 36 hours and roughly 1,000 turns sleeping and polling for a background job that never returned. It ended with five completed objectives.
Every evaluated run stopped before the token or time limit forced termination. The primary constraints were therefore planning, state tracking, tool use, and recovery from errors. Those weaknesses become visible during multi-hour tasks that require an agent to remember earlier discoveries and revise failed strategies.
Open weights change control and cost
Open-weight deployment lets operators run a model on infrastructure outside an API provider’s monitoring, rate limits, and access controls. Once weights are downloadable, the provider cannot observe individual deployments or revoke them centrally. Running an effective cyber agent still requires suitable hardware, an orchestration harness, tools, and technical configuration.
- Capability: DeepSeek exceeded GPT-6 Sol Daybreak on Nova and reached 21 of 31 Lighthouse objectives.
- Economics: The reported model-usage cost for a long DeepSeek run remained below three dollars.
- Evaluation: Equal token budgets revealed progress that shorter tests would miss.
- Operational limits: No system completed the deepest attack chain, and several agents became stuck without recovering.
- Provider control: Downloadable weights allow deployment beyond centralized monitoring and throttling.
Defenders can run the same loop
Security teams can apply the same approach to authorized staging environments, internal cyber ranges, and red-team exercises. An open model can run overnight, search for multi-step paths through a network, and return a command trace showing which weaknesses it exploited and how it moved between systems.
The evaluation design also applies beyond cybersecurity. Hidden external markers prevent self-reported success from affecting scores, equal usage budgets make efficiency differences visible, and multi-hour runs expose failures in memory, planning, and recovery.
The results describe performance on two controlled ranges using a specific harness, toolset, prompt, and budget. Broader claims about arbitrary enterprise networks require additional environments and repeated runs, but the benchmark demonstrates that inexpensive open-weight agents can already execute substantial portions of realistic intrusion chains without human intervention.
Comments
