How to measure productivity in AI projects

AI coding tools have become routine on many teams. The question that comes right after is easy to ask and hard to answer: did the team become more productive?

The honest answer is usually “it depends on how you measure”. A team can write more code, close more tasks and, at the same time, break more things in production. Another can feel faster without being faster. This article shows how to set up a measurement that holds up against these traps.

If you want to know how much of programming work AI can already do, see also AI in programming. The focus here is different: how to measure the effect on your own project.

Why the feeling of a gain is not enough

The most cited data point on the subject comes from a study by METR, a research organization that evaluates AI systems. In July 2025, it published a randomized controlled trial (a study in which chance decides who receives the treatment) with 16 experienced developers from open-source projects. They completed 246 real tasks in repositories they knew well, and each task was randomly assigned to allow or forbid early-2025 AI tools.

The result went against expectations. Before the study, participants predicted AI would cut task time by 24%. Afterwards, they estimated it had cut it by 20%. Measured by the clock, time with AI was 19% longer.

METR itself, in its write-up of the study, lists what the result does not show. It does not prove that AI slows down most developers, the sample is small and the setting (veterans working on mature codebases) does not represent every kind of work. The lesson that matters here is about method: the perception of those using the tool can differ substantially from the measured effect.

The picture is also changing fast. In February 2026, METR reported that the follow-up study, with late-2025 tools, had stopped producing reliable data. Many developers refused to work without AI, which skews the sample. The new estimates pointed to a speedup (an 18% reduction in time for returning participants and 4% for new ones), but with confidence intervals that include zero, meaning no firm conclusion. The organization believes the gain has probably increased since early 2025, but says its data is very weak evidence of how large that change is.

The metrics that mislead

Lines of code

More code does not mean more value. The paper that introduced the SPACE framework, published in ACM Queue in 2021 by researchers from GitHub, Microsoft and the University of Victoria, points out that someone who produces a lot of code may not be producing quality code. With AI, the problem grows: generating text has become cheap, and a line counter rewards exactly what has become cheap.

Suggestion acceptance rate

The acceptance rate is the share of the AI’s code suggestions that the developer accepts. It looks like a good indicator, and there is a reason for that. A 2022 study of GitHub Copilot users by Ziegler and co-authors concluded that this rate was the metric that best tracked users’ perceived productivity.

Note the word “perceived”. The rate tells you how much the person liked the suggestions, not whether the software got better, reached production sooner or had to be redone later. As the METR study showed, perception and outcome can diverge.

Closed tasks and story points

Counting completed tasks or estimation points (story points) is also measuring activity. The SPACE paper warns that activity metrics should never be used on their own to reward or penalize developers, and that a team judged only on points tends to optimize the points at the expense of invisible work, such as helping colleagues.

What to measure: delivery and stability (DORA)

DORA (DevOps Research and Assessment) is a research program associated with Google Cloud that studies software delivery performance and published its tenth annual report in 2024. Its metrics measure the delivery system, not the individual.

For years the program spoke of “four keys”. According to the DORA metrics guide, updated in January 2026, the current set has five, split into two groups:

• Throughput: change lead time (from commit to production), deployment frequency and time to recover from a failed deployment.
• Instability: change failure rate (deployments that require immediate intervention, such as rolling back) and deployment rework rate (unplanned deployments triggered by production incidents).

This split is useful for evaluating AI because it captures the typical side effect: moving faster and breaking more. The guide itself makes warnings that apply to any measurement: numeric targets invite people to “game” the indicator (the so-called Goodhart’s law), a single metric misleads, and comparing different applications or teams with each other distorts the reading. The idea is to compare your team with itself over time.

What the DORA reports found

DORA’s data shows why looking only at speed is risky. In the 2024 report, the team estimated that a 25% increase in AI adoption was associated with a 7.5% improvement in documentation quality and a 3.4% improvement in code quality, but also with a 1.5% drop in delivery throughput and a 7.2% drop in stability.

In the 2025 report, based on nearly 5,000 responses, 90% of participants said they used AI at work and more than 80% believed it had increased their productivity. This time, the relationship between AI adoption and delivery throughput came out positive. The relationship with stability remained negative. The report’s explanation is that a higher volume of changes tends to create instability when strong automated testing, mature version control and fast feedback are missing. DORA’s summary line is that AI amplifies what already exists on the team, for better and for worse.

One caveat: these are opinion surveys with statistical associations, not experiments. They show trends, not causation.

What to measure: the people (SPACE)

The SPACE framework complements DORA by looking at people and workflow. The name comes from its five dimensions, according to the original paper:

• Satisfaction and well-being: how satisfied people are with their work, tools and team, and signs of burnout.
• Performance: the outcome of the work, such as quality, reliability and customer impact.
• Activity: counts of actions, such as commits, code reviews and deployments.
• Communication and collaboration: how people share information and integrate each other’s work.
• Efficiency and flow: the ability to make progress without interruptions, waits and handoffs.

Two of the paper’s recommendations apply directly. First, use metrics from at least three different dimensions so that they balance each other. Second, include at least one perception measure, such as an internal survey. The authors also ask for results to be reported in aggregate and anonymously, by team, not by person.

Note the apparent contradiction: after seeing that perception misleads, why measure perception? Because it measures something else. Satisfaction and the sense of flow matter in their own right and can signal problems such as burnout early. The mistake is to use perception as a substitute for measured time or quality.

How to set up an experiment on your team

In the editorial team’s assessment, a simple design, inspired by the practices of the studies cited, fits most teams:

1. Define the question. “Does AI help us?” is too vague. Prefer something like “does the coding assistant reduce change lead time without worsening the failure rate?”.
2. Record a baseline. Before changing anything, collect a few weeks of the chosen metrics. Without a “before”, there is no “after”.
3. Create a fair comparison. Ideally, randomize tasks or periods with and without the tool, as METR did. Comparing a team that adopted AI with one that did not mixes the effect of the tool with differences between the teams.
4. Pick metrics in tension. For example: change lead time (throughput), change failure rate (instability) and a short satisfaction survey (perception).
5. Measure actual time, not just estimated time. Ask people to estimate the gain and compare it with what was measured. The difference between the two is already a result.
6. Allow time for learning. Results may change as the team learns to use the tool; METR cites learning effects as one of the limitations of its study.
7. Revisit the experiment design. METR’s own 2026 case shows that once everyone uses AI, it becomes hard to keep a group without it. Plan for that.

Be careful with lab experiments too. In a 2023 controlled study, Peng and co-authors showed that developers with GitHub Copilot implemented an HTTP server in JavaScript 55.8% faster than the control group. It is a legitimate result, but for an isolated, well-defined task. Real projects have legacy code, review, integration and maintenance, which the lab does not capture.

Quality and rework count too

Speed without quality is just deferred rework. A METR analysis from August 2025 illustrates the point. An agent based on Claude 3.7 Sonnet passed the maintainers’ automated tests in 38% of runs, across 18 tasks from two repositories. But of the solutions reviewed manually, none was ready to be merged without changes: tests, documentation or code style were missing. The estimated fix time was 26 minutes on average, even for those that passed the tests. The sample is small and the repositories have demanding standards, but the message is clear: “passed the tests” is not the same as “ready”.

To capture this on your team, track:

• Change failure rate and deployment rework, from DORA.
• Review rounds per pull request (a request to merge code) and time spent on review. If AI generates more code, someone has to review it.
• Defects found after delivery, per period.
• Code rewritten shortly after it was merged, if your tools let you measure it.

For criteria to check what AI produces, see how to evaluate AI responses. And if your team uses agents that carry out entire tasks on their own, read autonomous agents: where they work and where they fail.

Key takeaways

Software productivity does not fit into a single number, and AI makes that more evident. Lines of code and acceptance rate are easy to collect, but they measure activity and preference, not outcomes.

A useful measurement combines throughput and stability (DORA) with several human dimensions (SPACE), starts from a baseline, compares fairly and treats quality and rework as part of the cost. The studies cited range from 2021 to 2026 and the tools change every few months, so today’s result on your team may not hold a year from now. Measure again.