Google DeepMind Folds Computer Use Directly Into Gemini 3.5 Flash

Gemini 3.5 Flash now has computer use built in as a native tool, letting agents see and control browsers, mobile, and desktop interfaces alongside Search and Maps in a single model call.

·
·
·
Read6 min
TypeNews
TopicLlms · Agents
  • Native integration: Computer use is now a built-in tool in Gemini 3.5 Flash, replacing the standalone Gemini 2.5 computer use model.
  • Single-model agents: Agents can now combine screen control with Search, Maps, and function calling in one model context.
  • Benchmark performance: Scores 78.4% on OSWorld-Verified, within 0.3 points of GPT-5.5 (78.7%), at roughly one-third the per-token cost.
  • Three environments: Supports browser, mobile (Android), and desktop environments with a full action set including click, type, scroll, drag-and-drop, and hotkeys.
  • Safety architecture: Adversarial training against prompt injection plus opt-in enterprise safeguards for sensitive/irreversible actions; prompt injection remains an unsolved industry problem.
  • Available now: Via the Gemini API and Gemini Enterprise Agent Platform at standard Gemini 3.5 Flash pricing ($1.50/$9.00 per 1M tokens).

Google DeepMind just folded computer use directly into Gemini 3.5 Flash as a native built-in tool. That means the same model you already use for function calling, Search grounding, and Maps integration can now also see a screen, click, type, scroll, and navigate software without any human at the keyboard. It is available right now through the Gemini API and the Gemini Enterprise Agent Platform.

From two models to one

This is a consolidation story as much as a capability story. Previously, developers had to call a separate Gemini 2.5 Computer Use model for browser-based agent tasks, feeding it screenshots and receiving structured commands in a loop. Now, computer use is a native tool inside the same model developers already use for function calling, Search grounding, and Maps integration , meaning a single Gemini 3.5 Flash agent can see a screen, look something up on Search, and interact with a map without routing requests between multiple models.

The practical implication: agent pipelines that previously required orchestrating two separate models can now be expressed as a single tool call. Computer use now sits alongside code execution, search, and function calling as a native tool inside Flash, the model Google launched at I/O 2026 as its fastest agentic AI model.

How it actually works

The model operates through a continuous observe-think-act loop. Your application captures a screenshot of the target environment and sends it to the API along with the user's goal. The model analyzes the screen and returns a structured action , a click, a scroll, a keystroke , along with an intent field explaining its reasoning. Your code executes that action, captures a new screenshot, and sends it back. Repeat until done.

Here is the minimal setup using the new Interactions API:

python
from google import genai
client = genai.Client()
interaction = client.interactions.create(
    model="gemini-3.5-flash",
    input="Search for 'Gemini API' on Google.",
    tools=[{
        "type": "computer_use",
        "environment": "browser",
        "enable_prompt_injection_detection": True
    }]
)
print(interaction)

Gemini 3.5 Flash supports browser, mobile, and desktop environments, with text, image, video, audio, and PDF inputs. Each action response now includes an intent field that explains the model's reasoning behind each step , a significant improvement over the legacy model, which returned raw coordinates with no explanation.

The benchmark picture

On OSWorld-Verified , the standard benchmark for computer interface control , Gemini 3.5 Flash scores 78.4 compared to GPT-5.5's 78.7, a gap of 0.3 points. The more significant difference is pricing: Gemini 3.5 Flash costs $1.50 per million input tokens and $9 per million output tokens, compared to GPT-5.5's $5 and $30 respectively. For high-volume agentic workloads, that differential compounds substantially at scale.

Gemini 3 Flash, the predecessor model, scored 65.1 on OSWorld, making the generation-over-generation gain 13.3 points. That is a meaningful jump. All 16 entries on the OSWorld-Verified leaderboard are self-reported scores; as of now, none have been independently verified by a third party. The benchmark is valuable for tracking directional progress, but absolute performance comparisons should account for the possibility that self-reported scores reflect vendor-optimized evaluation conditions.

Where Gemini 3.5 Flash already led before this update is in tool orchestration. It scores 83.6% on MCP Atlas, beating GPT-5.5's 75.3% by a meaningful margin. MCP Atlas tests multi-step tool calling and schema adherence across complex agent workflows. Computer use now plugs into that strength.

What it is good at , and where it still struggles

The practical use cases Google is targeting:

  • Continuous software testing, where agents navigate applications and verify functionality without human testers stepping through each screen
  • Knowledge work automation: filling forms, extracting data from dashboards, navigating internal tools
  • Research tasks across multiple websites, gathering and comparing information
  • Automated data entry workflows that span multiple applications

Computer use in AI is still early. The models can handle familiar interfaces but struggle with unexpected pop-ups, CAPTCHAs, dynamically loaded content, and layouts they have not seen before. OSWorld measures success on predefined tasks with consistent environments; production agents operate in applications that change continuously, require authentication, and present UI states the model was not trained to handle.

Safety is the real engineering story

A model that can click, type, and submit forms on your behalf introduces a specific class of risk that a chatbot does not: it can be hijacked. When AI agents browse websites, read emails, or interact with applications, they become vulnerable to indirect prompt injection attacks , where malicious content embedded in a webpage or document attempts to hijack the agent's instructions, causing it to perform unintended or harmful actions.

Google shipped three layers of protection:

  • Adversarial training: targeted adversarial training specifically for prompt injection risks in live environments.
  • Opt-in enterprise safeguards: two optional systems that require explicit user confirmation for sensitive or irreversible actions, and automatically stop tasks if an indirect prompt injection is identified.
  • Configurable safety policies: built-in policy categories covering financial transactions, sensitive data modification, communication tools, account creation, and legal agreements , each individually overridable.

Despite these measures, prompt injection remains an unsolved problem across the industry. The UK National Cyber Security Centre has stated there are currently no guaranteed mitigations. Google's own documentation recommends against using computer use for critical decisions, sensitive data, or situations where errors cannot be corrected. The opt-in nature of the safeguards is a signal worth reading carefully.

The real shift: computer use as infrastructure

Google's decision to make computer use a built-in tool rather than a standalone model signals confidence that the capability is mature enough for general availability , but the opt-in safety guardrails signal equal awareness that it is not yet mature enough to run unsupervised.

What changed architecturally is that computer use is now composable with every other Gemini tool. An agent can look up a flight on Search, navigate to a booking site via computer use, fill in the form, and pause for user confirmation before submitting , all within a single model context. The previous Gemini 2.5 computer use model was a standalone system: it could control interfaces, but could not simultaneously use Google Search, Maps grounding, or custom function calls within the same agent context.

The pricing math also changes the calculation for production deployments. Flash is one of the cheaper models in Google's lineup, which could make computer use more accessible for large-scale automation than running it through a heavier model. At $1.50 per million input tokens, running thousands of agentic sessions is a different cost conversation than it was with a dedicated Pro-tier computer use model.

How to get started

The official documentation covers the full Interactions API. Google also released a reference implementation on GitHub with a Docker-based sandbox, and there is a live demo environment hosted by Browserbase if you want to test capabilities before building. The model identifier is gemini-3.5-flash and the tool type is computer_use. You will need the google-genai Python SDK version 2.7.0 or higher.

The key thing to build correctly from the start is your execution environment. Google strongly recommends running agents in a sandboxed VM or Docker container, not directly on a host system. The reference implementation includes a ready-to-use sandbox as a starting point , use it.

Trending
  • No trending articles

Comments

avatar

Next Reads