Pull requests now arrive in minutes after ideation. By the end of the day, there may be dozens. No human can read them all properly, not under pressure. So we ask another agent to review what the first agent wrote. The agent approves the diff because the code looks correct. CI checks are running and return green with unit tests, linters, and compilers passing. The pull request looks ready. But has anyone asked the app whether the thing really works?
Why mobile apps still break after good code review
Code review catches one class of issues. Tests catch another. Mobile regressions keep slipping through because the failure often lives in the space between code and runtime.
Navigation can loop unexpectedly. A permission prompt can block the first run. A keyboard can cover the primary action. A deep link can land in the wrong place. A release build can crash where debug worked. A native dependency can behave differently under load than it did in local development. The implementation can look fine in review and still fail on the first real interaction.
That is the constraint mobile teams are working with now: the diff can look clean while the user flow is broken. In an agent-written codebase, this gets harder, as the volume of plausible changes goes up, while the human review capacity does not.
Why code review agents and E2E suites do not close the gap
The obvious response is to add more automation to review, let one agent write the code, and allow another to inspect the patch. That helps with surface-level issues, but it still leaves the product itself mostly unverified.
Code review agents are good at checking plausibility. They can flag missing tests, suspicious type usage, and obvious problems in a patch. They are much weaker at verifying runtime behavior across state transitions, native APIs, permission flows, and layout changes.
The next response is usually end-to-end testing. That is part of the answer, but it solves a different problem. Scripted E2E suites are valuable for stable coverage. However, they are also slow, flaky, and broad enough that they do not always tell you whether the exact flow affected by a PR is still sound.
What is missing is a bounded behavior review: a fast verification pass tied to the change under review.
For mobile apps, that means giving an agent a narrow task. Verify onboarding and login. Check checkout. Validate profile editing. Inspect the flow the PR is most likely to affect. Collect evidence. Report what happened in a way a human reviewer can scan quickly.
What a mobile QA agent should actually do
A review agent should not roam the whole app looking for surprises. That makes the output noisy and the runtime expensive.
Give the agent a bounded mission, enough product context to understand the app, and a deterministic environment that removes guesswork. The loop is as follows:
- Observe the current app state.
- Pick the next action based on the task and current screen.
- Act on the app.
- Capture evidence where it changes the review decision.
- Report what was verified, what failed, and what remains uncertain.
That is what turns a QA agent into a review tool instead of a demo. The output should answer a reviewer’s real question: would I ship based on this evidence?
If you want the broader framing behind this workflow, watch the recording of our Behavior Review for AI-Native Delivery webinar, where together with Piotr, weexplain why runtime behavior still escapes code review and how bounded QA agents fit into the delivery loop.
The stack behind reviewable mobile QA
Most teams already have most of the pieces. At a minimum, you need:
- a PR workflow that decides when QA should run
- an agent runtime that can reason about the task and call tools
- a device-control layer that can inspect and operate the app
- artifact storage for screenshots, logs, and reports
- a PR reporting step that posts the result back to GitHub
Here, the cleanest setup is to keep the agent, the device layer, and the workflow separate.
In practice, that separation maps cleanly to the stack. In our QA agent setup, Eve handles the agent runtime, agent-device handles app inspection and device actions, and the workflow handles everything around the run: booting the simulator or emulator, launching the app, passing PR context into the session, collecting artifacts, and posting the result back to GitHub.
const agent = new ToolLoopAgent({
model: 'openai/gpt-5.4-mini',
instructions: `
You are a mobile QA agent.
Treat the app as a black box.
Infer acceptance criteria from the PR or diff.
Use agent-device to inspect the UI, navigate,
capture evidence, and write a report.
`,
tools: {
get_pr_context,
agent_device,
write_report,
},
});That division keeps the QA logic focused on the product behavior under review. The agent decides what to verify, the device layer interacts with the app and captures evidence, and the workflow packages the result so a reviewer can use it.
In practice, the agent itself stays intentionally small. agent.ts only selects the model. The behavior contract lives in instructions.md: infer acceptance criteria from PR metadata, verify only the highest-signal flow for the requested platform, confirm the app under test is actually foregrounded, prefer snapshot for semantic checks, save a small set of screenshots as evidence, and return a structured result.
Device control sits behind one typed tool, agent_device.ts, which wraps the local agent-device CLI. That keeps platform logic out of the prompt while still letting the agent call commands such as help workflow, appstate, open, snapshot -i, and screenshot.
agent-device help workflow
agent-device appstate
agent-device snapshot -i
agent-device press @e2
agent-device screenshot ./artifacts/login-step.pngAgent-device fits into the device-control layer. It gives agents structured access to running apps through accessibility snapshots, interactive refs, screenshots, logs, traces, profiling, and replayable flows. We introduced the tool in more detail in the Agent Device: iOS & Android automation for AI agents article.
We covered the Eve side of this setup in more detail in Building mobile QA agents with Vercel Eve.
Why cloud and remote execution matter
A mobile QA run rarely lives on one machine.
CI can run the agent and post the result back to the pull request, but the app still has to run where the platform allows it. iOS simulators need macOS. Android emulators can run on Linux or macOS. Real-device checks depend on whichever host has the phone plugged in.
A straightforward setup covers a lot of pull requests: start the QA run in CI, boot a simulator or emulator, verify the flow, collect screenshots and logs, and post the result to GitHub.
Teams that need real-device coverage usually split the workflow. CI still starts the run, but device access stays on a separate machine. agent-device proxy fits that setup. It exposes the simulator, emulator, or connected device without moving the whole QA workflow onto the same host.
The infrastructure can change underneath, but the review job does not. Run the flow, capture evidence, and attach something a reviewer can judge.
For token efficiency and screen-reading trade-offs inside that loop, How we optimized Agent Device for mobile app automation covers why structured snapshots usually beat screenshot-only reasoning.
What makes the result trustworthy
Reviewers trust the setup when the output is repeatable, scoped, and easy to inspect.
Start with a deterministic setup.
In practice, that means scripting the same bootstrap path on every run: install the app consistently, launch it the same way, reset what should be reset, and reuse artifacts when that improves stability.
Then add product context. The agent needs to know the important screens, the expected behavior, and the platform constraints that shape the flow. Without that context, it drifts toward shallow exploration.
Model choice comes next. Some review tasks need a stronger model, others do not. If the scope is tight and the tool surface is strong, a smaller model can often do the job for cents instead of dollars.
The report has to carry its weight too. Screenshots, visited screens, notes, logs, and recordings should support the claim the agent is making. A vague pass or fail label is not enough when the reviewer is deciding whether the change should ship.
Finally, make the result repeatable. Replayable flows, stable setup, and structured reports give reviewers a way to rerun the case, compare platforms, or confirm a failure on another branch.
Where real devices still beat simulators
Simulators and emulators are the right default for a large share of PR validation. They are faster to provision, easier to reset, and good enough for many navigation and UI checks.
Real devices still earn their place when behavior depends on device-specific prompts, connected hardware, physical performance characteristics, OEM quirks, or runtime conditions that do not show up reliably in a simulator. Permission flows are a common example. So are some native integrations, push-related flows, and bugs that only appear on connected hardware.
That is another reason shared device hosts matter. If real-device access is part of the review workflow, teams need a way to expose those devices to the agent system without turning every validation run into a one-off local setup.
The next review surface is the working app
Agent-written code changes the economics of software delivery. Teams can produce more code, faster, with less direct human typing. Trust still has to be earned before the merge.
The review stack needs a third layer alongside code review and automated tests: an app-behavior review tied to the change and backed by evidence from the running product.
If your team wants to build reviewable QA agents, shared device-host infrastructure, or CI workflows that verify mobile behavior before merge, talk to us about how we build agent-driven delivery systems. This is the kind of workflow we design and ship with teams that need faster delivery without weaker release confidence.
