Skip to content
AI Primer
breaking

Vincent Koc previews iPhone automation with accessibility XML and 2 ms latency

Vincent Koc previewed an iPhone automation package that reads accessibility XML alongside vision. He reports 2 ms latency in an 18-hour multilingual keyboard test, but the package requires a host computer.

2 min read
Vincent Koc previews iPhone automation with accessibility XML and 2 ms latency
Vincent Koc previews iPhone automation with accessibility XML and 2 ms latency

TL;DR

Screen sharing can block native camera and sharing options, Koc argues. People without developer accounts face a staggered rollout.

Typing latency

Koc built the package for his own agent to use. He says the agent spent 18 hours looping through English, Japanese and Chinese keyboards.

Typing-rate settings and 500-character action times

The more revealing number is the full-action speedup: about 2.6×, from 13.6 seconds to 5.2 seconds for 500 characters, against an 8× increase in the configured typing rate.

An Appium contributor has separately warned that higher typing frequencies can produce flaky input.

Accessibility XML and MCP

Koc describes three pieces of the interface:

  • Page state: Accessibility supplies an XML feed describing the page and its layers.
  • Visual input: Vision models remain in use alongside XML.
  • Agent connection: The package uses MCP. Koc describes over-the-air support as straightforward to add.

He also rejected an ESP32-based approach using AirPlay on latency grounds.

For comparison, Appium's XCUITest settings document separate costs for retrieving the UI hierarchy: deeper trees and expensive attributes such as visible can slow source generation.

Host computer and Apple scope

Koc calls the package Apple-centric and floated an Android version in response to interest. The preview does not specify which host operating systems are supported.

He also criticised go-ios's reverse-engineering burden and said it had gone more than three months without a release.

go-ios itself supports running XCTests and WebDriverAgent on Linux, Windows and macOS. Koc's preview leaves his package's device-communication backend unspecified.

MIT release and developer accounts

Koc promises free community access under an MIT license in the preview thread. He describes the release as forthcoming.

People without developer accounts face unspecified “pitfalls.” Koc says he has workarounds and will stagger the release.

Separately, Koc promised a release within 24 hours and called it “95.5% cheaper,” without naming a comparison in that reply.

Further reading

Discussion across the web

Where this story is being discussed, in original context.

On X· 5 threads
TL;DR5 posts
Typing latency2 posts
Accessibility XML and MCP2 posts
Host computer and Apple scope2 posts
MIT release and developer accounts3 posts
Share on X