Claude Opus 5.5 leads visual coding benchmark under stricter network controls

Vals AI evaluated 15 frontier models on SWE-bench Multimodal, with internet access disabled to prevent agents from retrieving published fixes. Claude Opus 5.5 resolved 70.21% of tasks, nearly nine percentage points ahead of the runner-up. The results also show that API cost poorly predicts performance on screenshot-based software issues.

A bug report the model has to see

SWE-bench Multimodal evaluates coding agents on real issues from user-facing JavaScript repositories where screenshots contain information needed to diagnose the bug. The model must interpret the image, navigate an unfamiliar codebase, produce a patch, and satisfy the repository’s tests.

A typical task might involve a chart with incorrect axis labels, a diagram editor that drops connections, or a syntax highlighter that assigns the wrong color. These tasks test visual interpretation alongside code search, debugging, implementation, and test execution.

The v2 test split contains 480 tasks selected for reproducible evaluation after known flaky or ungradeable cases were removed. John Yang, Carlos Jimenez, Ofir Press, and collaborators at Princeton and Stanford built the benchmark.

Claude opens a nine-point lead

Claude Opus 5.5 led the field with a 70.21% resolution rate, while scores across all 15 models ranged from 37.5% to 70.21%. The table shows the six highest-scoring entries reported by Vals.

The final column divides reported benchmark spend by resolved tasks. It excludes retries, validation, and production integration costs.

Anthropic held three of the top four positions, with Gemini 4 Argon as the highest-ranked model from another provider. OpenAI’s GPT-6 variants scored between 37% and 44%.

Price is a weak signal

Reported cost across the full 15-model field ranged from $0.03 to $11.40 per task, while higher spending produced inconsistent accuracy gains. Several comparisons illustrate the gap between price and performance:

• Gemini 4 Argon cost $11.40 per task and scored 14.17 percentage points below Claude Opus 5.5, which cost $4.92.
• DeepSeek V4.1 Flash cost $0.23 per task and beat Gemini 3.8 Flash by 5.6 points at roughly one-sixteenth of the price.
• Claude Opus 5.5 cost slightly more per task than Claude Sonnet 5.5, yet its higher resolution rate produced a lower benchmark spend per resolved task.

These figures make DeepSeek V4.1 Flash the strongest budget option among the six leaders, while Claude Opus 5.5 combines the highest accuracy with better cost efficiency than the other models in the top four.

Open internet leaks the patches

Network access exposes benchmark answers because every task ID contains the number of the upstream pull request that originally fixed the issue. In internet-enabled trials, agents could locate that pull request and copy its patch instead of deriving a solution from the screenshot and repository.

Vals disabled network access for every model in the published run. Across three models tested under both configurations, removing internet access reduced scores by 21 to 35 percentage points. Muse Spark 1.3 Max fell from 77.92% with network access to 43.33% without it.

Published SWE-bench Multimodal results therefore need an explicit network policy. Scores produced with unrestricted access can include direct retrieval of the reference patch, while the network-disabled configuration measures work performed from the supplied issue, screenshot, and checked-out repository.

A lean evaluation harness

Vals runs the benchmark with mini-swe-agent, the reference agent from the SWE-agent team. The evaluation follows a compact workflow:

1. Each task starts in an isolated sandbox containing the target repository.
2. The agent receives the issue text, while screenshots downloaded in advance are attached to the first message.
3. The model uses a shell to inspect files, edit code, and run commands over multiple turns.
4. The agent stops when it declares the task complete.
5. The upstream SWE-bench harness grades the final diff against the repository’s tests.

The reported error bars use binomial standard error from one run per model. That calculation estimates uncertainty across the 480 tasks, while run-to-run variation remains unmeasured because Vals did not repeat each model evaluation.

Where the numbers apply

Many coding benchmarks present issues primarily as text, while production bug reports often rely on screenshots of broken charts, misplaced controls, incorrect colors, or malformed diagrams. SWE-bench Multimodal measures whether an agent can extract evidence from those images and turn it into a tested patch.

For teams selecting a model for visual front-end triage, Claude Opus 5.5 offers the highest measured resolution rate, and DeepSeek V4.1 Flash offers the lowest spend per resolved task among the six leaders. The strongest model still left 29.79% of the benchmark unresolved.

These results cover a curated set of JavaScript repositories under a single agent harness and one run per model. Performance on backend systems, native applications, other languages, different agent designs, and repeated trials remains unmeasured.

Teams running the benchmark themselves should disable internet access, record the harness and model configuration, and report both task-level cost and resolution rate.