DeepSeek V4.1 Flash Beats GPT-6 Sol in a Simulated Cyber Attack

Vals AI has reported that DeepSeek V4.1 Flash 1017 outperformed GPT-6 Sol on one simulated corporate-network scenario. Each model completed three attempts on fresh copies of the range, and Vals compared their average results. GPT-6 Sol ran through OpenAI’s Daybreak trusted-access program, which reduces refusals on approved cybersecurity tasks.

DeepSeek’s downloadable parameters and lower token prices place the result outside the usual provider-controlled access model. The full benchmark remains under construction, however, and Vals has yet to publish the scenario, scoring rubric, complete run logs, or broader results. The available evidence supports a narrow claim about one multi-step task.

A simulated network and a terminal agent

Vals is building the benchmark to estimate how much progress an AI agent can make inside a simulated corporate network. Each model receives infrastructure credentials and an objective, then operates until it determines that no useful actions remain. The setup is intended to approximate the technical stages of an intrusion conducted by a capable operator.

Vals ran the models through the Pi agent harness, Mario Zechner’s minimal terminal-based runtime for coding agents. Pi supports multiple model providers, while the operator remains responsible for sandboxing, permissions, credentials, and tool access.

• Repeated runs: DeepSeek V4.1 Flash 1017 and GPT-6 Sol each completed three runs on fresh copies of the range.
• Reported result: Vals averaged those runs rather than selecting the best attempt.
• Other models: Every other model in the preliminary comparison completed a single run.
• Access mode: GPT-6 Sol used Daybreak, which can materially reduce refusals on tasks that resemble offensive security work.

Daybreak lowers the refusal rate

The Vals run used a version of GPT-6 Sol whose behavior differs from the default public API configuration. OpenAI’s Daybreak overview describes Blue and Red access tiers for verified defensive and offensive security work. Its onboarding guide says GPT-6 Sol and GPT-6 Luna support reduced refusals after approval and activation through the Responses API.

An AlphaSignal report on a separate tracker found Daybreak Blue scoring 32 points above publicly available GPT-6 Sol. Artificial Analysis attributed nearly all of that gap to the private configuration’s zero refusal rate on tasks resembling offensive work. Because that tracker used a different evaluation, its 32-point difference cannot be transferred to the Vals scenario. It does show why access mode must be recorded alongside model name and version.

A 19-fold gap on public tests

Vals says DeepSeek consumed significantly more tokens during the new scenario, but its lower input and output rates reduced the total bill. The published figures below come from CyberBench v1.1 and illustrate the pricing gap. Vals has yet to disclose the exact cost of the new multi-step run.

On the public CyberBench v1.1 leaderboard, which measures security-bug discovery and patching, GPT-6 Sol scores 77.98% at $1.34 per test. DeepSeek V4.1 Flash scores 73.69% at $0.07. GPT-6 Sol therefore delivers 4.29 percentage points more accuracy while costing about 19 times as much per test. The new simulated-range scenario evaluates a different, multi-step offensive workflow, so its results require a separate comparison.

Strong cyber scores, uneven reliability

Across the broader Vals Index, the 1017 revision ranks 14th with 56.58% accuracy at $0.64 per test. GPT-6 Sol records 57.54% at $7.58 per test, a gap of less than one percentage point at almost 12 times the cost.

DeepSeek’s technical report lists an 88.1% pass@1 score on CyberGym, compared with 84.5% for GPT-5.6 Sol, plus 62.8% on SEC-Bench Pro. Pass@1 measures whether the model succeeds on its first attempt, without relying on multiple samples.

Weak spots and run-to-run drift

• Legal work: DeepSeek scores 6.67% on Harvey’s Legal Agent Benchmark.
• Long-horizon programming: It records 40.28% on an International Olympiad in Informatics benchmark, compared with 96.89% for GPT-6.1 Sol.
• Consistency: Aikido results for a model labeled DeepSeek V4 Flash show 73.8% precision and a 74.4% F1 score, the lowest figures in the tested frontier group. Precision measures how many reported findings were correct, while F1 balances precision with recall. Only 12 of 24 pooled known vulnerabilities appeared in all three runs.
• Version clarity: Aikido’s “DeepSeek V4 Flash” label differs from “DeepSeek V4.1 Flash 1017,” so direct version equivalence requires confirmation.

Downloadable weights narrow the gate

Daybreak Red requires separate approval and stronger verification for authorized penetration testing, red teaming, and exploit development. An open-weight model can be deployed locally, subject to its license and infrastructure requirements, without OpenAI account approval. Here, “open-weight” means the trained parameters are available for download; access to training data, source code, and unrestricted licensing remains a separate issue.

A model reaching comparable outcomes in one offensive scenario reduces the coverage of provider-side access controls. Those controls still govern hosted systems, but they cannot constrain equivalent capabilities distributed as downloadable weights.

Evidence boundaries

1. Scope: The claim rests on one scenario from an unpublished benchmark. Vals says a fuller evaluation will follow.
2. Variance: Three runs provide an initial average but cannot tightly characterize reliability, especially given Aikido’s consistency findings.
3. External validity: A simulated range omits parts of real attack chains, including social engineering, unfamiliar infrastructure, persistence, and operational security.
4. Reproducibility: Independent verification requires exact model identifiers, Daybreak tier, system prompts, range images, scoring rules, tool permissions, token budgets, time limits, and per-run traces.

What teams should measure next

• Model evaluators: Compare outcome per dollar, run-to-run variance, refusal behavior, and task completion across several network topologies.
• Benchmark authors: Publish prompts, environment images, agent settings, tool traces, stopping conditions, and individual run scores.
• Security teams: Run cyber agents inside isolated sandboxes with least-privilege credentials, complete logging, and human approval for exploit execution.
• Policy teams: Include locally deployable models in threat models instead of relying solely on provider approval gates.