This article reports the replay experiments behind agent-device v0.20: 20 repeated runs, 38 Maestro flows, and 36 repair trials.
Day-to-day, I work on tools that make agents more efficient at mobile development. One of them is agent-device, a CLI that gives agents eyes and hands on iOS and Android. They see the screen in an optimized format and act on it like a user would, on simulators and real devices, locally or remotely.
A few weeks ago, I gave Claude Sonnet a routine QA task on a connected Android emulator:
Open Android Settings, read the "Network & internet" and "Connected devices" row labels on the homepage, tap into Connected devices, go back, scroll to the bottom, and read the "Tips & support" row label. Then close the session.
Claude used agent-device to inspect and control the emulator. The task required nine device actions. The agent completed it in 3 minutes and 18 seconds. It felt slow, though this was the latency I had come to expect.
The event log showed that agent-device spent little time operating the emulator. Taps and snapshots occupied thin slices; most of the elapsed time fell outside the measured device execution.
I saved the completed flow as a replayable script, a feature available in agent-device, and ran it again with agent-device replay. It finished in 8.8 seconds, including the app launch, with zero model calls.
Reducing model work
We could not break down every second of the roughly 190-second gap. In the opening run, two of 29 tool calls were help lookups or rejected commands; together, they accounted for 7.9 seconds of measured tool-call time. In a separate audit of 108 spontaneous agent runs, 375 of 2,668 calls (14.1%) were help lookups or rejected commands that never reached the device. The vast majority of the opening run still happened outside device execution: model inference, API round trips, screen reading, and deciding what to do next.
In April, we cut agent-device's token usage in half by changing what the agent sees. One debugging session dropped from 150k to 69k tokens and from 9 to 6 minutes. It made each observation cheaper.
This time, we asked a different question: how often does the agent need the model to look at the screen and decide what to do next? The answer shaped what we shipped and when the model needs to re-enter the loop.
Before the experiments
- Environment: Simulators and emulators on a workstation. Android tests used a stock Pixel 9 Pro XL image on an API 37 emulator. iOS tests used the iOS Simulator. We did not test physical devices. We gated repair timing and its control replays on the host's five-minute load average. The control began only after two readings, taken at least five minutes apart, both came in below 10. The other experiments did not use the same gate, so their load-sensitive timings are approximate.
- Models: A small model and a mid-tier model from one provider, in the Haiku and Sonnet classes available at the time of testing. The data repository includes the exact model IDs, settings, and pricing formula. A "turn" is one model response in the agent loop. Continuous run metrics report medians, with ranges where variance is part of the finding.
- Tokens: Counts include cached input reads, which cost much less than output tokens. A 22% token reduction can therefore produce only a 9% cost reduction. These counts compare conditions within the experiments. They are not durable benchmarks across model generations or pricing changes.
Fewer observations with --settle
An agent driving a phone often repeats the same cycle: tap, take a screenshot, inspect the result, then decide what to do next. A screenshot can arrive during an animation, which gives the model a half-transitioned screen and may trigger another observation.
agent-device snapshot -i
agent-device press @e10
agent-device snapshot -i # observe: screen mid-transition, refs half-stale
agent-device snapshot -i # observe again: the screen has settled, so think and actagent-device v0.18 introduced --settle. The command waits for the UI to stop changing and returns the settled observation in the same response.
agent-device snapshot -i
agent-device press @e10 --settle
# one response: tap confirmed + the settled interactive snapshotWe measured it on a navigation-heavy flow in React Navigation's example app. For each model class, we ran the same task three times with --settle required and three times with it prohibited. The harness enforced the setting.
Across these runs, --settle reduced the number of model turns and the tokens processed. For the Haiku-class model, wall time changed by only 4% despite 25% fewer turns and 22% fewer tokens. With three runs for each model and mode, we cannot draw a stronger conclusion about wall time.
Variance also fell. Sonnet-class runs cost between $0.86 and $1.53 for the same task without --settle, a 1.8× spread. With --settle, they cost between $0.51 and $0.55, a 1.1× spread.
The benchmark used three runs for each model and mode on one task, with two model classes from one provider. We report medians and ranges without significance claims.
We now recommend --settle by default, and agents can freely skip it when it does not fit.
Replay steps the agent already knows
We see agents rediscover the same app every day. In our test apps, each run worked through the same login and navigation from the beginning, spending model turns to rebuild knowledge available from the previous run. Every repeated decision added cost, latency, and another chance to choose the wrong action.
Replayable scripts keep the model for discovery and changed screens. Once the agent knows the route, it starts recording with open <app> --save-script=flow.ad, performs the route, and publishes the script on close. To keep the session open, it verifies the destination with a selector and calls session save-script. The resulting script replays without the model:
agent-device replay flow.adThis produced the 8.8-second replay from the opening. We repeated it 20 times. Nineteen runs completed cleanly, with a median of 8.0 seconds and a range of 7.2 to 8.5 seconds. The remaining run stopped before replay sent another action because scroll verification incorrectly reported a divergence. We traced the false alarm to an Android scroll-hint timing bug and fixed it before v0.20 in PR #1314. The replay executes the known path without model calls.
A replay script resembles an end-to-end test written for Maestro or Appium, but it handles failure differently. In a conventional suite, a human reads the failure, edits the test, and runs it again.
When the replay diverges, the report names the failed step and reason, then includes either a fresh, bounded screen digest or an explicit unavailable state. In this run, the fresh refs were enough to diagnose the failure immediately. If resume.allowed is true, from gives the step where the plan can continue, either the failed step or the next one, and planDigest binds the continuation to the same plan. Session state is separate: when the replay started with --save-script, resume.repairSessionHeld: true confirms that the live session is still available.
Reuse supported Maestro flows
Agent-device accepts Maestro YAML through agent-device replay flow.yml --maestro. Supported flows run without conversion; unsupported syntax produces an explicit error.
agent-device test --maestro e2e-tests runs a directory in bulk.
A retained session note reports that all 38 flows passed in 539 seconds with zero model calls and zero retries. We did not keep the raw per-flow log, so we treat this as a compatibility summary, not a benchmark that readers can re-derive from the published data.
What a recording contains
This trimmed recording contains one CLI command per line and machine-readable comments describing the elements used during the original run:
open "com.android.settings" --relaunch
# agent-device:target-v1 {"role":"textview","label":"Connected devices","ancestry":[...],"sibling":0,"viewportOrder":0,...}
press "role=\"textview\" label=\"Connected devices\" || label=\"Connected devices\" || value=\"Connected devices\""
scroll "bottom"
wait "label=\"Tips & support\""The press command contains a fallback chain, so replay has several ways to find the element.
The comment records the target's identity and structural evidence. It uses a unique ID when available; otherwise, it records role and label, plus ancestry and relative sibling and viewport-order cues. Before sending an annotated action, replay checks the resolved target against that evidence and stops if it cannot establish the same binding.
The wait command records synchronization as part of the flow. That becomes relevant during repair: deleting a failing wait may make a warm run pass while leaving the script unreliable on a cold launch.
When the app changes
A recording becomes stale when the app changes underneath it. Recovery cost depends on whether the tool can explain the mismatch and stop before performing the wrong action.
A divergence report an agent can use
A failed replay step produces a structured report. This trimmed example comes from the false scroll-verification run:
{
"code": "REPLAY_DIVERGENCE",
"message": "Replay failed at step 6 (scroll \"bottom\"): Failed to verify scroll bottom state",
"hint": "scroll bottom needs a snapshot showing hidden content below before it will move.",
"details": { "divergence": {
"step": { "index": 6, "source": { "path": ".../flowA-settings.ad", "line": 10 } },
"screen": {
"state": "available",
"refsGeneration": 42,
"refs": [ /* fresh, bounded screen refs */ ]
},
"resume": { "allowed": true, "from": 6, "planDigest": "0168fcf2..." }
} }
}The report names the failed step and source line, explains why the replay stopped, and includes the current screen in the reference format used by a live agent session. In this run, that screen data was enough to diagnose the failure. resume.allowed: true confirms that the plan can continue from step 6; it says nothing about whether the session remains alive. This report came from a plain replay, so repairSessionHeld is correctly absent. If the required session is still open and the app is in the expected state, the agent resumes with:
agent-device replay flow.ad --from 6 --plan-digest 0168fcf2... \
--session "$REPAIR_SESSION" --state-dir "$REPAIR_STATE_DIR"For a one-shot CLI repair started with agent-device replay flow.ad --save-script, the agent reuses that replay's session name and passes the --state-dir from the divergence hint to every corrective command and the resumed replay.
To keep the same session for repair, the agent continues only when the report contains resume.repairSessionHeld: true. That is the one field that confirms the session is still available. The heal-loop workflow shown here follows this path. The plan digest binds the continuation to the same expanded replay plan. Earlier steps are skipped only when resume is allowed, and the agent must restore the app state expected by the resumed step. An earlier implementation could return an empty current screen when a full-screen overlay was present. The report was structurally valid but gave the agent nothing to diagnose. We fixed that in v0.20.
Refusing a plausible but wrong action
I would rather pay for a loud failure than let the replay act on the wrong element. After a UI change, a different element may match the recorded selector. The script can appear to pass while performing the wrong action.
Agent-device compares the element resolved during replay with the identity saved during recording. If they disagree, the replay stops before sending the action and returns both identities for review.
A real Android Settings run produced the mismatch below, trimmed to the v0.20 contract fields that matter:
{
"kind": "identity-mismatch",
"cause": {
"code": "IDENTITY_MISMATCH",
"message": "A disambiguation signal (sibling or viewport position) isolated a different element than the resolved target."
},
"targetBinding": {
"classification": "identity-mismatch",
"matchCount": 5,
"recorded": { "role": "textview", "label": "Battery percentage" },
"observed": { "role": "textview", "label": "Battery Saver" }
},
"repairHint": "caution",
"resume": { "allowed": true, "from": 6, "planDigest": "a446..." }
}Five elements matched. Selector resolution chose Battery Saver; the target-binding check compared it with the recorded Battery percentage evidence and withheld the action.
These checks can still report false mismatches. Under high host load, the check fired in roughly one replay out of five. We suspect the accessibility tree and visual state were sampled at different moments during a stretched animation. When it fired, the replay reported a false mismatch and withheld the action.
In 10 quiet control runs, the check did not fire. Benchmark replay under representative load.
Repair study: passing is not enough
"Self-healing tests" sounds like the natural next step. We measured how often an agent could repair a recording while preserving the evidence captured in it.
The repair matrix included 36 trials on our iOS test app:
- two model classes: Haiku and Sonnet
- two prompts: the documented repair flow and a bare "make it pass" instruction
- three types of UI drift: a renamed label, a cascading structural change, and a renamed read target
- three trials for each combination
For comparison, we also tried re-recording once for each drift type. Here, re-recording meant starting at the beginning and having the Sonnet-class agent perform the changed journey while recording a replacement script.
Before each timed run, the harness waited for the host CPU load to fall below its threshold. A same-day control then replayed the unchanged baseline 10 times without failure. The 36 repair trials cost $35.90; the three re-recording attempts cost another $3.80, bringing the total to $39.70 across 39 runs.
A repair counted only when it met all of these conditions without manual file edits:
- replayed successfully in a fresh session
- preserved identity annotations on every interaction
- contained no references tied to the repair session
- introduced no unintended steps
- replayed successfully five consecutive times
That bar keeps the evidence available for later target-binding checks.
In 21 of 36 trials, the agent opened the recording and edited it as source code. Many of those scripts replayed successfully, but their identity annotations were stale or invented. A benchmark checking only the exit code would count 22 repairs. Only 10 kept their evidence intact, a 2.2× gap. Our own harness misclassified this twice before we caught it.
The documented workflow reduced direct file editing for the Sonnet-class model from five of nine trials to two of nine. The Haiku-class model produced no evidence-intact repairs in 18 attempts; all six of its automated passes were direct edits. More models and test environments are needed before generalizing.
Repair versus recording again
Apps don't sit still. Labels and layouts change, and a replay that worked yesterday can fail today. End-to-end tests have always carried that maintenance cost. With agent-device, the agent can take the first repair pass: replay the known route, stop at the divergence, record the correction, and resume.
We measured how often that process produced a passing script with its identity evidence intact. Failed attempts still cost money. The table divides the total cost of six Sonnet-class attempts for each drift type by the number of valid repairs:
A full re-record of the flow by the agent started at the beginning of the changed journey and recorded a replacement script. The repair used our novel functionality to replay a script up until the failure and then hand over to the agent. In our testing the “Renamed label” run was the only fair comparison: a valid re-record cost $1.21, while the six repair attempts cost $4.57 per valid result. The other two re-records were hand-edited, so they cannot establish a break-even point.
I expected the repair to be cheaper than starting over. Replay already knows most of the route: it can run to the divergence, let the agent correct the changed part, then resume. The model should have less work to do.
At first, I disliked this result. I've come around to it. If recording the journey again costs less than saving it, the agent should start again. We need more valid re-records to learn where that boundary lies.
Failure modes we found
- Deleted wait: The agent removes a synchronization step because it is failing. The revised script passes against a warm, settled app and fails on a cold launch. A deleted synchronization step is a warning that the behavior changed.
- Silent no-op: Agents tried to save a script without an actively resumed session. The command produced no file and did not explain why, causing the model to repeat a billed action. Several apparent model failures came from this tool's behavior. v0.20 now reports the missing session requirement.
- Recording format drift: An older fixture reported a false divergence on a newer build because the snapshot format had changed. Recording format stability belongs in the reliability budget alongside app stability.
Choosing between live execution, replay, and repair
Replay the known path, then hand control to the agent
Many tasks start with a familiar route and end with new work. Agent-device v0.20 supports this with open-to-destination scripts. The agent records the route once and publishes it while the authoring session stays open. Later, a persistent daemon or typed client can replay the route and continue working in the live session created by the replay. Structured clients read the session ID from the response; the CLI names it up front:
agent-device open com.example.app --relaunch --save-script=screen-x.ad
# navigate to the destination
agent-device wait 'role="heading" label="Screen X"'
agent-device session save-script
# Publishing leaves the authoring session live. Close it before replaying
# later on the same device.
agent-device close
# Start a reusable local daemon, then replay into a named live session.
agent-device session list --state-dir .agent-device-handoff >/dev/null
agent-device replay screen-x.ad \
--state-dir .agent-device-handoff --session screen-x
agent-device snapshot -i \
--state-dir .agent-device-handoff --session screen-x
# Continue live with the same state directory and session. When finished:
agent-device close --state-dir .agent-device-handoff --session screen-x
agent-device daemon stop --state-dir .agent-device-handoffIn this example, the recorded wait is a required destination guard. It confirms that a matching landmark exists before control passes to the agent. In v0.20, the guard is selector-based, so it does not yet prove that the landmark has the same identity observed during recording. Issue #1349 tracks extending identity checks to guards.
Recorded fill and type values appear verbatim in the .ad file. The agent must not record passwords, tokens, or other secrets. Give it pre-authenticated test state or test-only credentials until parameterized inputs are available.
We tested the hybrid approach on a composite task: reach checkout, then complete form work that the script did not cover. In five runs, the agent handled the whole journey interactively. In five more, the replay delivered it to checkout before the agent took over.
Starting with the replay lowered the median from 357 to 209 seconds, a 42% reduction. Median reported cost fell from $1.15 to $0.62, a 46% reduction, and median turns fell from 57 to 35. Both approaches passed four of five runs. Each failure exhausted the 60-turn cap amid the same off-screen and expired-reference friction, so we could not tie either failure to one approach.
Composable fragments would let a reusable login recording feed multiple flows; replacing one file would refresh every dependent journey. We are tracking that work in issue #1336.
My rule after these runs is to record the route once, replay it while it holds, and spend model time only when the route changes or new work begins.
What we learned
The experiments left us with six working rules:
- The device was not the slow part in this run. The vast majority of our 3:18 run sat outside device execution: model inference, API round trips, screen reading, and decision-making. Across 108 spontaneous runs, one tool call in seven never reached the device. We now look for repeated decisions before investigating device latency.
- Exploration is what a model is for, and repetition is what a recording is for. The timing difference is not close: 3:18 versus 8.8 seconds and no model calls, on the same flow.
- A green test can still be an invalid repair. 22 of 36 repair trials went green; only 10 kept their evidence intact. Check how the fix was made, not just whether it passes.
- Repair and re-recording need the same bar. The only evidence-intact re-record cost $1.21, compared with $4.57 of repair spend per valid result for the same label drift. The other two re-records were hand-edited. We need more valid re-records before deciding when starting over is cheaper.
- A useful failure report saves model work. The report becomes the agent's next input. Missing screen evidence can turn one corrective command into a whole session.
- Test under the load you expect. Our identity check produced a false refusal on roughly one replay in five under heavy host load. It did not fire in 10 quiet runs.
Limitations
- The repair matrix used three runs for every model, prompt, and drift combination on one app in iOS Simulator. We tried re-recording once per drift type; two of the three scripts still required direct edits. The individual combinations are too small to generalize from. Across the full matrix, the model classes produced 0 of 18 versus 10 of 18 evidence-intact repairs; automated checks passed 22, but only 10 were valid repairs.
- The published settle benchmark used three runs with each mode on one task, two model classes, and one provider.
- The bootstrap benchmark used five runs for each approach, one composite task, and one mid-tier model. Each approach had one failure that exhausted the 60-turn cap amid the same scrolling problems with off-screen elements and expired references.
- The clean replay result comes from one 20-run Android experiment. For the 38-flow Maestro sweep, we retained only a session summary, not the per-flow logs. The suite also favors short, stable flows.
- Tests ran on simulators and emulators. We consistently checked host load only for repair timing and its control replays, so the other load-sensitive timings are approximate. Cost figures use the July 2026 model pricing.
- The data repository includes per-run JSON and TSV results, failures and exclusions, prompts, model IDs, token breakdowns, environment notes, the repair checker, and a script that regenerates five headline result sets. For the Maestro sweep, we retained a session note rather than a per-flow log. The repository excludes agent transcripts and videos.
Try agent-device with your agent
Agent-device is open source. Star it on GitHub, then install it on the machine where your agent runs and check the setup. It requires Node.js 22.12 or newer:
npm install -g agent-device@latest
agent-device doctorFrom there, describe the outcome in a prompt:
Use agent-device to [describe the task]. Start by running agent-device help workflow. Record the known route so you can replay it on future runs, then take over where new work begins.You provide the outcome and constraints; the agent handles the command sequence. It can save the full journey on close, or call agent-device session save-script at a guarded destination to publish the known route while keeping the session open.
If you repeat the experiments on your hardware and find a result that breaks this pattern, send it to us. We want the benchmark to expose the limits of the approach as clearly as its gains.
