A $7 ESP32 Lets Vision Models Fully Control an iPhone Without Jailbreak
A $7 ESP32-S3 board acts as USB mouse hands while AirPlay feeds eyes to a vision model, driving a stock iPhone with no Mac or Xcode required.
- Open source project lets a vision model drive a stock iPhone via a $7 ESP32-S3 acting as USB mouse.
- AirPlay mirroring through UxPlay provides vision; no Mac, Xcode, developer account, or jailbreak required.
- Works with any OpenAI-compatible vision API, or local models via vLLM, SGLang, Ollama.
- Reference local setup hits ~1.3s per action using Qwen3.8-Flash-Next on dual DGX Spark.
- Closed-loop pointer correction: wiggles pointer, finds it on screen, verifies screen change after each tap.
- Limits: cannot see passcode, Apple Pay, or DRM screens; ASCII keyboard input only.
A $7 ESP32 lets vision models control an iPhone
esp32-iphone-agent is an MIT-licensed project that lets an OpenAI-compatible vision model operate a stock iPhone. A Seeed Studio XIAO ESP32-S3 acts as a USB mouse and keyboard, while AirPlay sends the phone’s screen to a host computer for visual analysis.
The design removes the need for Xcode, a signed helper app, an Apple developer account, or a jailbreak. It still requires a Linux or macOS host to receive the AirPlay stream, run the agent server, and communicate with the ESP32 over a local network. The author’s demo runs locally with a vision model served by two NVIDIA DGX Spark systems.
The Apple SDK stays out of the loop
Common iPhone automation stacks rely on XCTest or WebDriverAgent. Those tools typically require a Mac, Xcode, code signing, and an app installed on the phone. Free provisioning signatures also expire after seven days.
This project works through interfaces already supported by iOS: AirPlay screen mirroring, USB Human Interface Device input, and the AssistiveTouch pointer. The phone remains unmodified, and the agent can attempt to control apps and games that expose little or no useful accessibility metadata.
Pixel-based control provides no semantic element IDs or guaranteed target coordinates. Every action depends on what the model can see, how accurately it locates controls, and whether the interface changes between capture and input.
Eyes, hands, and a planner
The host computer performs the screen capture, model inference, action selection, and error recovery. The phone supplies the interface, while the microcontroller converts compact network commands into USB input.
- Screen capture: The iPhone mirrors its display to UxPlay through AirPlay. A GStreamer pipeline converts the stream into JPEG frames for the model and web dashboard.
- Physical input: The ESP32-S3 appears to iOS as a standard USB mouse and keyboard. With AssistiveTouch enabled, a click becomes a tap, pointer movement becomes a drag, and scrolling produces a swipe.
This story is for Pro members
You've reached the end of the free preview. Upgrade to AlphaSignal Pro to read the full article - and everything else behind the paywall.
Comments
