ESP32 Agent for iPhone

An AI that uses your iPhone. With its own hands.

Plug a $7 ESP32 board into your phone. It watches the screen through AirPlay and taps through USB, while the app decides each step. No computer, no jailbreak.

Build it on GitHub Watch the demoApp Store · soon

Task Set an alarm for 7:00 AM that repeats every day

Reading the screen…1/12

Hi, I'm the agent. Poke me.

Tap · double-tap · hold · drag

How it works

One phone, one board. That's the whole setup.

The board is the agent's eyes and hands. The app on the same iPhone is its brain.

01 · Eyes

It sees your screen

The board opens its own Wi-Fi and shows up in Screen Mirroring. It passes the AirPlay stream to the app over Bluetooth, untouched.

02 · Hands

It taps like you do

Plugged into USB-C, the board is a plain mouse and keyboard. With AssistiveTouch on, a click is a tap and a drag is a swipe.

03 · Brain

It thinks in the app

The app sends each screenshot to the vision model you choose, plans the next step and checks that the screen really changed.

iPhone ──Screen Mirroring──▶ ESP32-S3 ──Bluetooth──▶ ESP32 Agent app
  ▲                       own hotspot, H.264                 screenshot → vision model → step
  └──── tap · swipe · type ◀── USB HID ◀────────── Bluetooth ◀──────────┘

An iPhone with a XIAO ESP32S3 plugged in, setting an alarm — "Set an alarm for 8:00 AM that repeats every Monday and Tuesday, then save it." The agent leaves its own app, opens Clock, types the time into the wheel, ticks the days and saves.

The real phone next to what the agent sees, and pointer calibration — Left: the real phone. Right: what the agent sees through the board's mirror, and the pointer calibration. Every run is kept on the phone to replay step by step.

Safety

It works for you. It checks with you.

Asks before risky stepsBefore it pays, sends a message, posts, deletes or touches account settings, it stops and asks. Answer from the notification.
Stop at any timeStop a run from the app, the notification or the Dynamic Island. Take over yourself, then hand it back.
Your model, your keyUse OpenRouter, OpenAI or your own server. Nothing is sent until you agree, and you can withdraw that any time.
No servers of oursRuns, screenshots and recordings stay on your iPhone. No account, no ads, no tracking.

Read the privacy policy.

What you need

Less than a lunch.

Board: XIAO ESP32S3About $7. Any ESP32-S3 with native USB and PSRAM.
Phone: iPhone, iOS 18+USB-C. The agent runs on the same phone.
Model: Any vision modelOpenRouter free models work. A paid model is faster.

Give your iPhone a pair of hands.

Build it on GitHub Get help