Insights | Thinslices Blog

AI coding agents: measuring ROI before you scale them

Written by Tudor Iordache | Sep 16, 2026, 2:33:52 PM

Gartner's 2026 market guide for enterprise AI coding agents reports that 90% of engineering leaders see improvements from the tools, against a net average productivity gain of 19.3%. A separate Gartner analysis of technology adoption puts the share of software engineering leaders reporting significant ROI from AI across the development lifecycle at 35%. The distance between those figures is not a tooling failure. It is what happens when a team adopts agents without a pre-adoption baseline, budgets for them as seats while vendors move to usage-based pricing and expects acceleration in the parts of delivery that agents do not touch.

Almost every engineering leader reports that AI coding agents help. Very few can say by how much.

That is the shape of the finding in Gartner's 2026 market guide for enterprise AI coding agents. Ninety percent of engineering leaders report improvements. The net average productivity gain is 19.3%. Both are true at the same time, and the distance between them is where most adoption decisions quietly go wrong.

A caveat worth stating up front, since the rest of this piece leans on the figure: Gartner publishes the 19.3% without a named survey, sample size or methodology on its public page. Treat it as directional, not as a benchmark any single team should expect to hit.

What the AI coding agent productivity numbers actually say

Gartner is not alone on the first number. Google Cloud's 2025 DORA report, drawn from close to 5,000 technology professionals, found 90% of respondents using AI at work and more than 80% believing it had increased their productivity. Two independent surveys landing on the same figure is about as settled as this field gets. What nobody has settled is what the improvement is worth.

Nineteen percent is a real result. Across a thirty-person engineering org, it is meaningful, compounding, and worth paying for.

It is also nothing like the number most teams budgeted against. Agents get bought on the promise of parallel execution and multi-agent orchestration, which reads as a multiplier. Nineteen percent is not a multiplier. It is an improvement.

The second figure makes the point sharper. In Gartner's analysis of 6 trends shaping technology adoption ROI, only 35% of software engineering leaders report significant ROI from AI across the software development lifecycle. So most leaders see improvement, and a minority see returns. Improvement and return are not the same claim, and teams that conflate them end up defending a tool they cannot measure.

The underlying problem is usually simpler than the technology. Very few teams captured a baseline before adoption. Without cycle time, review turnaround, defect escape rate and rework volume from the quarter before the agents arrived, there is nothing to compare against, and the assessment defaults to how the work feels. It feels faster. That is the 90%.

The sharpest test of that gap came from METR, a research nonprofit, in July 2025. In a randomised controlled trial, sixteen experienced maintainers of large open-source repositories worked through 246 real tasks, with AI tools allowed on some and withheld on others. They finished 19% slower with the tools. Asked afterwards, they estimated the tools had made them 20% faster.

Two caveats travel with that study and both matter. Sixteen developers working in codebases they had maintained for years is a small sample in close to the hardest case for an AI tool. More importantly, METR published an update in February 2026 saying it now believes developers are likely more sped up than its early-2025 estimate, while conceding it can no longer measure the effect cleanly, because between 30% and 50% of participants were declining tasks rather than do them without AI. The slowdown number has aged. The 39-point distance between measured and perceived has not.

Why AI coding agent ROI disappears at the review boundary

Agents compress the writing of code. They do not compress most of what happens after.

Review still runs at human speed, and it runs slower per line when the reviewer did not write the code and cannot assume the intent behind it. Integration testing, environment provisioning, stakeholder sign-off and release coordination all sit outside the loop the agent operates in. If writing code was 30% of the elapsed time from ticket to production, a large gain on that 30% is a modest gain overall.

The 2025 Stack Overflow developer survey describes this from the inside. The top frustration, reported by 66% of developers, was AI output that is almost right but not quite. The second, at 45%, was that debugging AI-generated code takes longer. Only 3.1% said they highly trust the accuracy of what the tools produce, and more developers actively distrust it than trust it.

DORA reaches the same place from a different angle. Its 2025 report found AI adoption carrying a positive relationship with delivery throughput and a negative one with delivery stability. More change, moving faster, into systems whose control surfaces did not change with it. The report's own framing is blunt: AI does not fix a team, it amplifies what is already there.

This compresses further in regulated work. Across the fintech, regtech, healthtech and publishing platforms we build in, the binding constraint is rarely how fast a change can be written. It is traceability, auditability and the evidence trail that a change was reviewed by someone accountable for it. Generating more code faster puts more load on exactly the step that cannot be automated away, which is why several of our engineers have argued that code quality discipline matters more in the era of AI, not less, and why regulated companies end up designing their AI architecture around data boundaries first.

None of this is an argument against agents. It is an argument for knowing which 30% you are accelerating.

AI coding agent pricing moved from seats to usage

The commercial side changed underneath the adoption decision. Gartner puts the enterprise AI coding agent market at roughly $9.8 billion to $11.0 billion annualised as of April 2026, and notes vendors shifting from seat-based subscriptions to usage-based pricing as agentic workflows consume more compute. Parallel agent execution is the selling point and it is also the cost driver.

That inverts the budgeting model. A seat is a fixed line item that a finance team can forecast. Usage scales with exactly the behaviour the tool was bought to encourage, which means the spend curve bends upward precisely when adoption is going well. Teams that modelled this as a per-developer licence are the ones now reporting that returns did not materialise. The gains were real. The cost line moved.

The same pattern shows up wherever AI moves from pilot to production, which is why it is worth building the business case before the first line of code rather than after the second invoice.

How to measure AI coding agents before you scale them

Three things, in order, before expanding a pilot.

1. Capture an AI coding agent baseline you can defend

Cycle time from ticket to production, review turnaround, defect escape rate and rework volume, measured for a full sprint cycle before agents are introduced. Without this, every later ROI conversation is an opinion.

2. Measure elapsed time to production, not lines written or tickets closed

Output volume is the metric agents flatter most and the one that correlates least with delivered value. If review queues lengthened while throughput rose, the gain moved rather than appeared.

3. Model usage cost against your own workflow, not the vendor's example

Run a bounded pilot, record actual token and compute spend per merged change, then extrapolate. Usage-based pricing punishes teams that budget from a list price.

Teams further along this path tend to arrive at a deliberate workflow design rather than a tool rollout, which our team has written about in lessons on designing an AI software development workflow and in a practical account of using agents on live product builds.

The honest read on AI coding agent ROI

At board level, the pattern is already visible. McKinsey's 2026 State of AI survey fielded across 1,719 respondents in 97 countries in May and June 2026, found 80% reporting that AI had improved their individual productivity, and 37% attributing any EBIT impact at all to AI. Six percent attributed more than five percent of EBIT. Individual productivity is not in dispute anywhere in the data. Earnings impact is.

The 19.3% is not a disappointing number. It is a realistic one that almost nobody planned around.

Teams reporting no return are rarely using the tools badly. They set an expectation the technology was never going to meet, skipped the baseline that would have shown the actual gain and budgeted a variable cost as a fixed one. Fix those three things and 19% is a result worth defending to a board.

The teams getting the most out of agents right now are not the ones with the best tooling. They are the ones who can produce a number.

AI coding agent ROI: common questions