How to Maximize AI SDLC? Measure the Outer Business Loop, Not the Inner Developer Loop

In the last year, people measuring the ROI of AI have been rediscovering insights I've run into several times in my multi-decade career. They're the same end-to-end values and economics behind Lean Software Development, Software Profit Teams, DORA, and Value Stream Mapping.

These principles predate AI, but they also apply to AI-assisted development:

• Measure the whole system, from development work to business outcomes.
• Give every change a hypothesis about the engagement or monetization metric it should improve.
• Confirm that enough users actually encountered the change before interpreting the results.
• Wait for sustained, statistically significant business uplift before declaring success.
• Weigh neutral changes against the code and maintenance burden they leave behind.

AI adds another layer: tracking tokens, time, and retries against a defensible contribution to the business.

The clearest example of these principles recurring comes from 2021, at a gaming platform with roughly 100 million monthly active users. No AI was involved at all.

We had about 980,000 players on one device: the 1GB Kindle Fire 7. The application was crashing due to out of memory about 500,000 times a month. Other devices with 1GB RAM had out-of-memory crashes too, but that particular device more sensitive than the rest according to our telemetry data.

Another Principal Engineer and I disagreed about whether fixing it was worth the effort. Their view was that those players would never monetize anyway, and they didn't care that players with lower-end devices were locked out of the platform. My disagreement was partly about values and partly dollars-and-sense. We were supposed to be bringing the world together with play, but it was also about scale: With close to a million players per month on that model of device, I expected enough monetization that the work would pay for itself. In this article, I detail the systems thinking and data-driven decision making that I encourage any business working with humans and agents should adopt to go from AI to ROI.

Waiting for the sustained business uplift before self-congratulating

The problem was memory footprint of our application, and it’s not unique to that company or its technology choices. Everyone who remembers supporting React Native on iPhone 4S or Amazon Fire TV Stick can probably relate.

We strung together a number of by-the-book optimization tricks in the scripting language, and further closed the gap with an optimization from the native garbage collector team. Together we reduced monthly crashes from around 500,000 to under 100,000, which put that device back in line with the normal spread of crashes per user across device classes.

The changes each went out as a phased rollout, a few percent at a time over a few hours, with uptake saturation (how many users got the rollout) being the rate limit, globally. Then we enabled it with focused feature flags that isolated our change as cleanly as possible so that competing experiments didn’t confuse us about attribution of uplift. Players on that device had close to zero engagement before the fix, so we could see the direct impact of each feature flag’s enablement on that device model. Nobody could credit another team's experiment for the uptick, either. The crashes happened so early that those players never got far enough to encounter anyone else's flags.

Before the fix, 0.01% of players on that device model somehow managed to join a multiplayer lobby. We could never get that far ourselves when testing on real 1GB devices. After the fix, 70% of them successfully joined a multiplayer game per session, according to both client telemetry and server telemetry. I wasn't measuring engagement minutes. I was measuring if the user value proposition was being met: playing together with friends.

I didn't go to leadership reporting success until 14-day retention went up globally with statistical significance. For new players on that device, it went up by over 400% in the 90 days after we enabled the flag for 100% of players (and removed it once its proof was incontrovertible), monetization on those devices went up 1,000x. That was enough to pay for a fully loaded Principal Engineer's total compensation for an entire year.

Exec leadership loved it. Middle management was less happy, because the next question from above was: "why was this broken for so long?"

None of the numbers that mattered in that story came from the Inner Developer Loop or purely from DORA metrics. This is why any activity, including that driven by AI, needs to be evaluated by its effect on the whole system and not just a local speedup.

Two loops

The Inner Developer Loop is the edit, build, and test/debug cycle a developer runs many times a day. A lot of AI productivity measurement lives there right now: time to a: working change, PRs opened, lines generated, reviews gotten, review feedback incorporate, approval, and merge. Those numbers are easy to collect. They tell you very little about whether the business got anything for its money (or the users got anything for an app update).

The Outer Business Loop is a variant of the DevOps outer loop that I first presented in a meetup talk about twelve years ago in San Francisco. The usual outer loop runs from merge through deploy and observation. The Outer Business Loop keeps going until you can say what the deploy did to the metrics your business runs on.

Every change carries a hypothesis

Any change should carry a hypothesis about which north-star monetization or engagement metric it should improve. The Kindle Fire fix had one. If the experience doesn't crash, players spend more time and are more likely to come back. The more of those players have longer sessions, the more likely they are to monetize (and have fun!).

It's fine if the hypothesis gets disproven. It obviously shouldn't regress your north-star metrics, though. At worst, a deploy should be net neutral.

Sometimes the results are mixed: metrics up for some users, some down, some flat. Then you and your agent need to dive in and understand why it wasn't a slam dunk. I've seen all kinds of non-technical, non-process reasons:

• a national holiday in one country overlapping your measurement window
• a confusing layout for the new feature in right-to-left languages
• a war starting in a neighboring country, so users are refreshing the news instead of following their normal engagement patterns

It really, really just depends. Investigating these scenarios were some of the most educational moments of my multi-decade career.

Absence of signal is not presence of neutrality

A net-neutral deploy doesn't mean much if the new functionality is behind a feature flag or A/B experiment that hasn't been activated and encountered by a statistically significant number of users. Here, encountered means the user actually navigated or scrolled to the place in the application where the flag or experiment lives and its code gets run. Until that happens, you don't know yet whether the change is neutral.

When Technical “Success” Rings Hollow For The Business

Similarly, "crashes didn't go up" isn't good enough, unless the deployment is part of a patch stack being deployed in one-at-a-time due to an abundance of caution. You only expect a business metric uptick when the last change in the stack lands. That can be okay, and isn’t uncommon. That said, if you have a single change deployed whose best achievement is “didn’t make things worse”, you're potentially adding cruft to the product and the code that you'll need to scrape out or adapt for the next hypothesis. We’ve seen different AI model providers appear to do a great job at first, but after a few months without an expert weaving in incremental refactoring to most changes, a Blob Class (classic from the 1998 book ‘Antipatterns’) emerged with tentacles so expansive that it would challenge even a 500k token context window.

So measuring lines of code merged, deployed, or reviewed, or their time to complete and accurate (C&A), doesn't go far enough. The better question is this: how few lines of idiomatic code can be written, well-tested, and deployed for each fractional percentage point that your north-star metric improves?

The Metrognome post here on the Callstack blog shows the same discipline inside the Inner Developer Loop. Its gate reverts a performance change unless the gain clears measurement noise, even when the new average looks a little better.

The Outer Business Loop applies that discipline one level up, to the metrics your business runs on. Like with performance metrics, a sub-metric might show a gain that demands a deeper measurement: L1 CPU cache misses might go way down but without app launch times showing improvement, does that mean you should throw out the change? Probably not, since it did eliminate a variable of the performance challenge.

Similarly, if an A/B experiment does get more users deeper into an engagement or monetization funnel, but not the actual monetization, should we throw out that change? Once again, it really depends on nuance and context — did users bail out of the funnel because of a fundamental design flaw in the experiment? Was there a bug or crash on the fourth screen/prompt? You can see how much of this decision-making depends on your own context and set of variables.

Dollars per defensible contribution

Then the next level: how many tokens did it take, what was the cost per token, were results Complete and Accurate (C&A), and how much time did it take? For bonus points, how many incorrect commands, bad builds, or retries were there in the session?

Now we're really getting down to the brass tacks: How many dollars did you spend in tokens for a defensible contribution to your business's bottom line? By defensible, I mean the kind of result I waited for in 2021. That's a north-star improvement with statistical significance, on a population that actually encountered the change, with no competing explanation for the uptick.

This is the real Outer Business Loop. Get that nailed with a local model that can be fine-tuned on validated successful iterations of the loop, and you’ll be unstoppable with happier users to show for it.

A checklist for AI-assisted product deployments

Use this on the next AI-assisted change you ship, or score one you've already shipped:

• Hypothesis: Which north-star metric should this change improve, and why?
• Exposure: Was the flag or experiment activated and encountered by a population of users that is defensibly diverse and large enough to draw insights from?
• Result: Did the north-star metric go up with statistical significance? If the results are mixed, can you do some detective work to determine why?
• Floor: Did anything regress? At worst, is it net neutral?
• Cruft: If it's neutral and not part of a phased patch stack, what will you need to scrape out later?
• Code: How few lines of idiomatic, well-tested code did it take for each fractional percent of improvement?
• Cost: How many tokens, at what cost per token, over how much time, with how many incorrect commands, bad builds, or retries?
• Bottom line: How many dollars in tokens per defensible contribution?

And the question the whole checklist adds up to: if you did all of the checklist items 100 times, would users, operations, product leaders, and developers all be happier and better off? These are nuanced questions and getting the answers can often need external experts who know how to jump in and build sustainable software factories focused.