Late one evening, I started two agent runs and went to bed expecting to wake up to merged pull requests. They were working on Simlock, an open-source tool I maintain at Callstack that hands out iOS simulators and Android emulators to agents. Both features had approved specs, and I had spent days building a workflow that could take them through implementation, review and testing. When an agent asked whether the device tests could run, I said yes and added: "I won't be here to answer your questions." I thought I had given it what it needed to carry on.
By morning, both runs had spent hours waiting for me, for different reasons. One had reached a gap in its tests and used up the review rounds I had allowed. The other had merged a pull request, started the next task, and found a product decision missing from the spec. It left me a question and stopped. At my desk, I would have read those messages, answered them, and watched the work continue. Overnight, each one had become the end of the run.
I thought the hard part was teaching agents to write, review and test code. The part I had missed was teaching the workflow what to do when they needed me. I started changing how the workflow handled permissions, review limits, and unanswered product questions. In this article, I'll walk through those changes and how they helped me build a workflow that can run without constant human intervention.
Where the workflow still needed me
The workflow around Simlock is what I call a software factory: agents, scripts and rules that turn approved work into merged code. A feature splits into tasks, each tracked in a GitHub issue and delivered as a pull request. An agent called the orchestrator coordinates the run, handing implementation, review and testing to separate agents and deciding what comes next from their reports. That arrangement lets different agents concentrate on different jobs. It also means the orchestrator needs an answer whenever a report leaves the next step open.
During the day, agents would ask me to approve the next step. I’d check what they had done and tell them to go ahead. Each exchange felt reasonable on its own, especially when the agent was about to use a shared device or touch the main branch. Across a whole feature, though, I kept returning to the conversation to keep the work moving. I hadn’t realised my presence had become part of the workflow.
An earlier daytime run made the pattern visible: almost all its questions were about permission or process; only one needed a product decision. I’d started turning recurring answers into instructions, but a rule about running tests couldn’t answer what a feature should do. I needed to separate the decisions I could settle once from the choices that would emerge as the agents built something.
Give routine decisions an answer
Device tests boot simulators and emulators on a shared machine, so two runs trying to use them at once can interfere with each other. The agent kept asking whether it could run them, but I was usually making the same decision: proceed when the machine is available. I added a script that holds a lock while the device tests run and updated the instructions to let unattended agents run those tests when the lock is free. When I'm present, it asks once, and my answer covers the whole delivery run, from implementation through review, testing, and merge
Review needed a more explicit ending too. The run that hit the test gap had found that its tests didn't prove startup happened in the required order. It had reached the review limit, so I had to authorize another fix and review. The missing test was added, and the feature finished the next day after further fixes and checks. That work needed verification before it could merge, but every extra round also needed me to let it continue.
I revised the review rules so that now they tell the orchestrator when to attempt another fix and when to hand the work back. Later checks focus on the fixes; corrections to comments and documents get recorded without another review cycle, while an unresolved defect in production code can still stop delivery.
Merging follows the same pattern. Agents go through a script that checks whether the pull request is ready, the automated checks have passed and the branch can merge. Direct merges are denied in the agent's settings, along with commands such as force pushes and releases. I can give the agent permission to finish approved work because that permission comes with conditions it must satisfy. If the merge script refuses, the run has a specific issue to fix or hand back to me. It doesn't need to ask whether I feel ready to merge.
Those changes moved repeated decisions into the workflow. The product question from the night was harder, because I hadn't made the decision at all.
The product decision the spec left open
Simlock was getting a warm pool: devices booted ahead of time so an agent could get one sooner. Requests for devices wait in a queue, and the spec hadn't settled how far ahead the pool should prepare. It could boot a device only for the first request, or prepare devices for the requests waiting behind it as well. That choice reached into an existing part of Simlock that shuts down idle devices. A device prepared for someone further back in the queue could sit unused, get shut down, and then get booted again by the pool.
The agent left a handoff explaining the loop and the alternatives. Preparing only for the first request limits how far ahead the pool works. Preparing for everyone waiting means the shutdown rule has to recognize those devices and keep them alive until they are needed. In the morning, I chose to prepare devices only for the first request. Keeping the existing shutdown rule made the change smaller. I wanted to ship a working pool sooner and improve how it prepared devices later. The agent folded that choice back into the spec and added it to the test list, then resumed implementation and review. The handoff had made the question quick to answer, but the run had still needed me there to answer it.
At the time, I required agents to stop for unresolved product decisions. I had already allowed them to make conservative assumptions about smaller details, provided they recorded those assumptions in the pull request description.The wording of an error message, for example, can usually follow the existing code. An Assumption: line records the question, the choice and why it fits. I can read it afterward and ask for a correction without keeping the run waiting over every detail.
For a while, I drew a firm line: agents could settle implementation details, but product decisions came back to me. I later changed that boundary. The orchestrator could make a provisional product choice, as long as it recorded the reasoning for me to review.
Decide, then tell
I updated the delivery instructions to let the orchestrator decide how a feature should behave when the spec leaves that behavior open. I told it to choose the option closest to the code’s existing behavior and easiest to reverse, then record the choice for my review and continue.
Under the current instructions, the orchestrator could handle the same warm-pool question during an unattended run. It could provisionally choose the queue-head option, record that choice in the spec as pending my review, and add a test for the behavior. The next agent could build against that decision. A Decision taken comment would capture the reasoning and what changing the choice would require, so the run could continue while I was away.
When I came back, I could review both the implementation and the decision behind it. If I wanted the other option, I’d review the consequences described in the decision comment and ask for the corresponding change. While the pull request was open, that answer would start a fix run. After a merge, it would become a follow-up task. Letting the agent continue means accepting that I may ask it to revise some of that work afterward.
I still approve the feature's requirements and architecture before delivery. Those decisions give the agent a starting point for choosing conservatively when a gap appears. A choice that contradicts an accepted architecture decision or a project rule requires the run to stop, as does a required device check that cannot run or a blocking defect left after the final review. The delegated choice still goes through the rest of the checks; the agent has permission to choose within those boundaries.
When a run does stop, it leaves a handoff clear enough for me to pick up the work without reconstructing the conversation. Other tasks that don’t depend on it can continue. A decision made on my behalf needs the same clarity: enough context to understand the choice and change it if needed.
Find the gaps before delivery starts
The warm-pool question belonged in an earlier conversation. I had given the agent permission to build the feature before we had finished deciding how it should interact with code already running around it. A provisional choice could keep delivery moving, but I wanted to discover more of those gaps while we were still planning. I needed the spec session and its second reader to uncover those interactions before I told the delivery agents to begin.
You can start with the points in your own sessions where work waited for you. Look at what your answer contributed: permission to run an operation, a decision about the next review step, or a choice about the product. Put recurring answers into the workflow, and decide which remaining choices you are willing to delegate and review later.
For each question that stops a run, decide whether the workflow should answer it, the agent can make a reversible choice, or delivery needs to wait for you. To apply this beyond individual agent sessions, explore how we build agentic delivery workflows, with verification and human review built into the process.
Next: how an agent interview and a second reader turn a feature request into a spec you can leave an agent to build from.
