Gemini Live adds spoken visual guidance on Android

Google has added Guided Vision to Gemini Live, giving people who are blind or have low vision spoken descriptions from their phone’s camera. The feature maintains context during a conversation and provides instructions such as tilting the phone, panning slowly, or stepping back when it needs a clearer view.

Guidance before the answer

Guided Vision processes camera input inside an ongoing Gemini Live conversation. A user can ask about an object, adjust the camera in response to Gemini’s directions, and continue with follow-up questions without starting a new image request each time.

When the camera points too high, sits too close, or misses part of the subject, Gemini can request a specific adjustment before answering. That feedback creates a loop between the model and the person holding the phone, which helps address a common problem in camera-based assistance: the user may be unable to verify whether the relevant object is visible or in focus.

Tasks built for short visual checks

Google positions Guided Vision for everyday tasks performed from a stationary or controlled position. Its examples cover four broad categories:

Professional interpreters shaped the training data

Google partnered with Aira, a professional visual interpreting service, to develop the feature. Aira visually interpreted tens of thousands of hours of data used to train Gemini Live for conversational, real-world visual assistance.

More than 1,000 members of Aira’s Trusted Tester network evaluated the system across daily routines. Aira specialists also worked with Google as subject-matter experts, helping the product team refine its behavior and assess safety guardrails.

Professional visual interpretation can capture the sequence of actions needed to complete a task, including when to request a wider view, a slower pan, or a different angle. Google has not published the tuning method, dataset composition, benchmark results, or an analysis measuring how much the Aira data improved individual capabilities.

Three routes to the camera

Guided Vision is integrated with Gemini Live and Android’s accessibility controls, giving users several ways to start it:

1. Gemini Live: Enable Guided Vision in Gemini’s profile settings, start a Live session, and share the camera.
2. Accessibility shortcut: Configure Guided Vision to open through the floating accessibility button, a two-finger swipe gesture, or both volume keys.
3. TalkBack: Open the TalkBack menu with a three-finger tap and select Guided Vision.

Google lists Android 9 or later as the operating-system requirement. Access also depends on Gemini Live being supported in the user’s region and language. The company says it tested the feature with blind and low-vision communities in countries including India, Brazil, Singapore, Indonesia, and Japan, though it has not released language-specific quality measurements.

Safety limits define the product

Google describes Guided Vision as a fallible assistive utility and sets explicit boundaries around its use:

• Outputs may contain incorrect descriptions or missed details.
• The feature is not intended for navigation, safe-travel guidance, or obstacle detection.
• Its output cannot replace a white cane, mobility aid, professional visual interpreter, or medical device.

Those restrictions reflect the consequences of errors in live visual systems. Misreading a label can disrupt a task, while missing a curb, vehicle, or overhead obstacle can create an immediate physical hazard. Guided Vision’s announced use cases therefore focus on inspecting nearby objects and spaces from a stable position.

The reusable input loop

Guided Vision demonstrates an input-quality loop that can transfer to other camera-based AI systems:

1. Inspect the available visual evidence.
2. Determine whether the framing supports a reliable response.
3. Request a specific physical adjustment when more evidence is needed.
4. Carry conversational context into the revised view.
5. Answer once the camera provides sufficient detail.

This pattern can support remote inspection, equipment troubleshooting, teleoperation, and other workflows in which a model depends on a person to position a camera. Its value comes from making evidence collection part of the interaction instead of treating every camera frame as complete.

Google currently presents Guided Vision as a Gemini app feature. The announcement includes no API, SDK, or programmatic access to its reframing behavior or Aira-derived training resources, so developers cannot directly add the complete Guided Vision workflow to their own applications.

A crowded field of visual assistants

Google enters a market that already includes Be My AI, developed by Be My Eyes with OpenAI, visual assistance through Meta’s Ray-Ban smart glasses, and Apple features such as image descriptions and Recognition Mode. Google’s approach combines Gemini Live’s conversational context with Android accessibility shortcuts and training input from professional interpreters.

Several technical questions remain open for researchers and product teams evaluating the approach. Google has not reported task-level accuracy, latency across supported devices, performance under poor lighting, error rates by language, or feature-specific details about camera-data retention and model improvement. Those measurements will determine how reliably the interaction design performs outside controlled demonstrations.