Gemini 4 Carbon Tops Scientific Vision Benchmark With 90x Cost Gap

Researchers use gel blots to estimate protein size, flow cytometry plots to isolate cell populations, and phylogenetic trees to infer relationships. Artificial Analysis has launched the VIALS-AA leaderboard, an independently run evaluation built on Handshake AI’s VIALS benchmark. It tests whether multimodal models can interpret scientific figures accurately and repeat the result across three runs.

The benchmark covers 161 image-based tasks across eight life sciences domains, including gel blots, flow cytometry, plasmid maps, and phylogenetics. Each task presents an image and asks for a specific conclusion. Models run through Stirrup, an open-source reference harness that provides code execution and image-inspection tools. GPT-6 Luna then compares each response with a reference answer. Artificial Analysis describes the full setup in its evaluation methodology.

Three Clean Runs Count as a Pass

A task contributes to the headline pass³ score only when the model answers correctly on all three independent attempts. One or two correct responses receive no credit for that task. The metric therefore captures repeatability alongside accuracy, an important distinction when a wrong interpretation can trigger unnecessary experiments or corrupt downstream analysis.

Gemini Leads a Crowded Frontier

Gemini 4 Carbon with the high reasoning setting leads at 64%, passing all three runs on 103 of 161 tasks. GPT-6 Astra at its maximum setting follows with 101 stable passes, or about 63%. Claude Opus 5.5 scores 61%, while GPT-6.1 Terra and GPT-6.1 Sol reach 60% and 59%, respectively.

The narrow spread makes cost, domain performance, refusal rates, and tool behavior relevant deployment criteria. A two-task lead also deserves more scrutiny than a broad percentage ranking conveys, particularly because an automated model grades the responses.

Costs Vary Nearly 90-Fold

Reported cost per task ranges from about $0.02 for GPT-6 Luna to $1.77 for Grok 4.7. Based on those rounded figures, the most expensive run costs roughly 89 times as much as the cheapest. GPT-6.1 Sol reaches 59% for about $0.06 per task, while Claude Opus 5.5 reaches 61% for about $1.64, a 27-fold cost increase for two percentage points.

The cost difference tracks the number of model-tool turns and the tools invoked. Image-measurement actions account for 5% to 6% of tool calls from GPT-6 Astra and GPT-6.1 Sol, compared with 25% to 27% for Claude Opus 5.5 and Claude Sonnet 5.5. The Claude models frequently measure pixels, draw reference lines, and interpolate molecular weights. OpenAI’s models more often inspect an enlargement and answer directly.

One Image, Three Strategies

A protein-spot identification task shows how those approaches diverge. GPT-6 Astra opened the image, enlarged it fourfold, inspected the result, and answered 40 to 50 kDa in six turns without taking measurements. Claude Sonnet 5.5 cropped the relevant region, measured ladder bands, enhanced contrast, drew guides, and interpolated a value across 18 turns. Both produced the correct range.

Grok 4.7 used 27 turns to label pixels and draw grids, then selected the wrong range after anchoring on the position of a printed label instead of the faint protein spot. The trajectory shows how additional measurements can reinforce an incorrect visual assumption.

Artificial Analysis also tested GLM-5.3, which lacked native vision access but received the image files and code tools. The model loaded images into arrays, applied optical character recognition to enlarged and thresholded regions, and compared pixel intensities. This image-processing route consumed 54 turns and 57 code calls on the protein-spot task, yet still produced the wrong answer.

Open Models Cover More Than They Repeat

Open-weight models, whose parameters can be downloaded and hosted by users, showed broad task coverage across three attempts. Kimi K3.1 Pro, MiMo-V2.6-Pro, and MiniMax-M3.1-Flash each solved 114 to 119 tasks at least once. The seven highest-scoring proprietary models solved 111 to 118 tasks at least once.

Repeatability separated the groups more sharply. The open-weight models passed all three attempts on 48 to 73 tasks, compared with 95 to 103 for the leading proprietary systems. Kimi K3.1 Pro leads the open-weight group at 45%, followed by MiMo-V2.6-Pro at 43%, leaving a 19-point gap between Kimi and the overall leader.

Inconsistent interpretation creates operational risk in laboratory workflows. A plasmid-digest answer that changes between runs can lead to unnecessary validation, repeated experiments, or incorrect follow-up decisions. Teams using the open-weight models may need multiple samples, reconciliation logic, or human review for consequential outputs.

Safety Filters Reject Published Figures

At least one model issued a safety refusal on 14 of the 161 tasks, and Claude Sonnet 5.5 declined six. The affected examples included protein structures and gel images from published papers. These refusals indicate that some safety filters classify benign wet-lab figures as restricted content, reducing coverage even when the underlying model may be capable of answering.

A Practical Deployment Shortlist

• Highest repeatable accuracy: Gemini 4 Carbon leads overall and posts the strongest results in plasmid maps, phylogenetics, and flow cytometry.
• Lower-cost frontier: GPT-6.1 Sol and Terra finish within five points of the leader while costing substantially less than Claude Opus 5.5.
• Self-hosted access: Kimi K3.1 Pro leads the open-weight group, though its lower repeatability increases the need for validation.
• Agent design: Tool-call volume alone provides little guidance. Effective systems must identify the relevant visual evidence, measure selectively, and stop before an incorrect assumption compounds.

Limits Behind the Scores

• Scope: The 161 tasks sample figure-reading problems across eight domains. Full experimental workflows include additional planning, data-quality, and validation steps.
• Grading: GPT-6 Luna evaluates responses against reference answers. Close margins inherit the limitations of automated judging and merit inspection at the task level.
• Cost: The reported figures reflect this harness and its tool-use patterns. Provider pricing, retries, context length, and routing choices will change production costs.

Reproducing the Evaluation

Teams building scientific copilots can inspect the open-source Stirrup harness, download the VIALS dataset, and rerun the setup with their own models, prompts, graders, and tool policies.