DX's Justin Reock on the AI ROI that isn't showing up in the data

retained text

DX's Justin Reock on the AI ROI that isn't showing up in the data
DX Deputy CTO Justin Reock on why PR throughput keeps climbing, AI spend is up 50x, and the data still shows no clear return on agentic coding tools yet.
A year after the Opus 4.5 moment, when a lot of engineering orgs decided they were done writing code by hand, the question worth asking is what the telemetry actually says. At ELC 2026 I put that to Justin Reock, Deputy CTO at DX, whose company measures exactly this across a large population of engineering organizations.
His answer: velocity is up, spend is up by a lot more, and nothing in the data yet shows a clear, measurable return.
Velocity is up, and it is not 10x
DX tracks a metric it calls true throughput: PR throughput weighted by the relative complexity of each PR. Reock calls PR throughput an imperfect metric but a decent proxy for the rate of change in an organization.
I think pretty definitively we can say that velocity, at least as measured by PR throughput, is going up steadily, but it hasn't even hit a 2x inflection from a year ago, let alone the 5x, 10x, 20x that I think people are looking for.
The output side looks flatter still. Innovation ratio, a metric DX borrows from manufacturing, measures the share of engineering time spent building new features rather than keeping the lights on. It has stayed roughly flat for over a year even as velocity climbed. Reock won't draw a conclusion from that yet. One candidate explanation he offers is that managers are spending the new capacity on backlog clearing and platform improvements instead of new value-creating work.
Then there is the denominator.
We have to juxtapose that against cost, which is up about 50x in terms of spend over the last year for the larger organizations. Right now, while I believe we are still at an inflection point, I can't point to anything in the data that's showing a clear, measurable ROI.
That is not a market call. Reock's read is that the industry is still mid-inflection, not that the bubble pops tomorrow. The gap between a 50x cost line and a sub-2x throughput line is the thing to explain, and the explanation turns out to be about where work actually gets stuck.
Somebody still has to read all that code
DX sees a linear correlation between the share of AI-generated code reaching production and average review time. The shape of it is stark:
Organizations that are shipping 80% AI-generated code — their time to first review is actually twice as long as organizations that are only shipping 40%.
Reock reaches for the Theory of Constraints to explain why that is exactly what you'd expect. The industry elevated and fully exploited the bottleneck of code creation, which was never the bottleneck to begin with. Relieve a constraint that wasn't binding, and the queue forms somewhere else.
It formed in the most expensive place available. Review falls on senior engineers, the people who got senior by liking to build, and Reock counts the morale hit alongside the throughput hit:
Review is necessary. It needs to be done to de-risk what we're doing in the organization, but it's not how most seniors want to spend the majority of their time.
Downstream of review, the picture gets noisier. DX is seeing real volatility in change failure rate: for plenty of organizations defects are up, which creates more rework on the deployment side. Rework is the second tax on generation speed.
Relieving the code-generation segment doesn't speed up the pipeline. The queue re-forms at ideation and again at review.
The constraint moved upstream too
The other place work piles up is before anyone writes anything. Backlogs are getting cleared, or they're full of tickets that sat for two years precisely because they were never that strategic. Which raises an unglamorous question: who fills the backlog now, and with what?
Reock's observation is that over the last six to nine months the industry collectively discovered the term PDLC, product development life cycle. The reason, in his telling, is arithmetic.
Well, gosh, the work phase in our organization was taking 60 days, but now that we've started looking at idea-to-commit, that's like a 200-day period. This is a much longer area of the value stream than just doing the work.
Squeeze a 60-day phase inside a 200-day cycle and the headline number barely moves. His prediction is that the fixes will look archaeological: semantic storytelling, scientific ideation, formal verification, techniques from 30 or 40 years ago repurposed for ideation friction, the way DevOps went after faster CI, faster builds, and fewer flaky tests.
What the leaders handling this well have in common
Reock's answer here was conspicuously boring, which is the point. He frames it as the first principles of good leadership rather than anything new.
The first pattern is time. Safe sandboxes, safe-to-fail pilots, hours engineers can actually spend learning the tooling, with AI skill-building tied to their own success. The failure mode is the mirror image: mandate the tools, fund no time to learn them, and the adoption number goes up while the capability doesn't.
I've been saying for two years now that AI is not going to take your job, but somebody really good at AI will probably take your job.
He compares the shift to cloud, a whole discipline rather than a catchphrase, with the same hype-cycle shape underneath. The skills inside it are concrete and already hiring-relevant: model fit, model routing, token economics, intelligent model orchestration. His argument for experimenting now is about cost:
It's good to do that now, while we can take advantage of these heavily subsidized tokens, because none of us really understand yet how much this technology actually costs.
The second pattern is messaging. Psychological safety remains the number one driver of productivity in an engineering organization, something Google's Project Aristotle established back in the mid-2010s. Managers who get out ahead with transparent, proactive conversations, framing this as augmenting capability rather than replacing it, run teams that feel safer and perform better. Staying quiet and letting people guess is a productivity decision too.
Zapier, as a worked example
The company Reock pointed to applied agents across the whole PDLC rather than to code generation alone. Zapier built a fleet of internal agents aimed at administrative overhead: commit log distillation, meeting note summarization, enough of it to cut from five standups a week to two.
The numbers they report are the kind that move an innovation ratio.
They're estimating that a new engineer at Zapier gets onboarded within about a two-week period — the industry benchmark there is about 30 days.
Zapier also estimates each engineer now contributes 15% to 20% more value than before, and is hiring more than at any point in the company's history. An organization that treats agents as a force multiplier on engineers buys more engineers. Worth saying what the interview didn't surface: any corresponding downside at Zapier. This is the one stretch of the conversation that reads as pure upside, and I'd want their change failure rate next to those onboarding numbers before copying the playbook wholesale.
Measure the part you never measured
The practical instruction here is situational awareness: watch where the needle is moving, and when it isn't moving where you expected, go find the new constraint instead of buying more of the old solution. If spend went up 50x and the innovation ratio didn't move, the answer is upstream in ideation or downstream in review, and most teams have no instrumentation on either.
What I'm still waiting for is an Agile moment for agentic coding: shared norms teams can borrow from each other instead of every org reinventing the wheel privately. We're not there. Until we are, the thing separating the organizations with an ROI story from the ones guessing is the thing Reock keeps coming back to.
First principles have never been more important. Physics has not changed. Neuroscience has not changed. The tools have changed.
The tools changed the cost of producing code. They didn't change where the queue forms, and most of us still aren't measuring that part.

document fetched 2026-10-24 11:26