Google Ships Gemini Computer Use to Let AI Click, Type, and Scroll

Google has made the Computer Use tool generally available through the Gemini API’s Interactions and GenerateContent APIs. The tool lets agents navigate browser, mobile, and desktop interfaces by interpreting screenshots and returning structured actions such as clicks, keystrokes, and scrolls.

Existing preview integrations require no migration because Google reports no breaking changes. The release focuses on stability while preserving the execution model: Gemini proposes each action, and the developer’s runtime performs it in a controlled environment.

A screenshot-to-action loop

Computer Use applies function calling to graphical interfaces. Given a prompt and screenshot, the model returns a UI action with normalized coordinates on a 0-to-1000 scale. A client-side runtime, commonly Playwright, converts those coordinates to the current viewport, executes the action, and captures the resulting screen. This design leaves authentication, session state, network access, and action approval under application control.

1. Send the task, current screenshot, and relevant interaction history to the model.
2. Receive a function_call describing an action such as click, type, or scroll. The response also includes an intent field summarizing the action’s purpose.
3. Scale the coordinates to the active viewport and execute the action inside a sandboxed browser, device, or desktop session.
4. Capture a new screenshot and return it as a function_result. Repeat until the task completes, fails, or reaches an application-defined limit.

One API, three control surfaces

The GA release preserves the preview’s multi-environment design. Browser and desktop agents receive a broad mouse-and-keyboard action set, while mobile agents add Android-specific controls.

Model choices

Token use drives the bill

Computer Use carries no separate tool fee. Google bills its requests at each model’s standard token rates on the paid tier, with no free-tier access. Costs grow with screenshot size, prompt history, response length, and the number of interaction turns. Teams can control spend by using the smallest practical viewport, limiting retained context, setting iteration and timeout caps, and terminating loops when progress stalls.

Guardrails travel with each action

Responses can include a safety_decision that classifies an action as allowed, blocked, or require_confirmation. Applications must enforce that decision before executing the proposed action, especially when a workflow can create an account, modify records, send a message, or commit a purchase.

Google’s built-in policy categories cover:

• Financial transactions and purchases of regulated goods.
• Changes to sensitive health, financial, or government records.
• Autonomous emails, messages, and social posts.
• Account creation and consent-banner interactions.
• Acceptance of terms of service and other legal agreements.

Prompt injection requires a separate defense. The optional enable_prompt_injection_detection setting scans screenshots for adversarial page content that attempts to redirect the agent or expose data. Google continues to recommend supervision for consequential tasks because detection can miss attacks and models can select incorrect actions.

Production controls remain yours

A production deployment should isolate the execution environment, restrict credentials and network access, require confirmation for irreversible actions, and log screenshots, function calls, results, and safety decisions with appropriate redaction. Developers also need limits for retries, time, tokens, and total actions, plus recovery logic for stale screenshots, navigation failures, unexpected dialogs, and partially completed tasks.

• Run sessions in disposable browser, device, or desktop sandboxes.
• Grant each agent only the permissions required for its task.
• Use domain and action allowlists where the workflow permits them.
• Make transactional steps idempotent to prevent duplicate submissions.
• Route blocked, ambiguous, or stalled tasks to a person or deterministic fallback.
• Measure completion rate, action count, latency, cost, and unsafe-action frequency before setting service targets.

GA removes the preview caveat

Anthropic, OpenAI, and Google have all released systems that operate graphical interfaces. Production adoption depends on task reliability, security controls, and a supported API surface. Google’s GA release addresses the API-stability concern and gives teams a supported path for web QA, data entry in systems without APIs, mobile testing, and research across permitted sites.

Google provides a reference Docker environment in its GitHub sandbox. A hosted Browserbase demo offers a quicker way to inspect the interaction loop before building a runtime.