AI in programming: how much work it automates

“AI already writes half the code.” “AI makes programmers slower.” Both statements circulate, and both rest on some real data. The problem is that they answer different questions.

This article brings together the main public evidence on how much of programming work artificial intelligence actually performs. We separate three types of evidence: experiments with people, standardized model tests (benchmarks) and usage data. Each one measures something different, and none answers the question on its own.

Why the question is hard to answer

“Automating work” can mean quite different things:

• Speed: how long it takes a person to finish a task with and without AI.
• Autonomy: what fraction of a task AI solves on its own, without human help.
• Delegation: how much of what people ask AI for is “do it for me” rather than “help me do it”.

One study can show a large speed gain on a simple, isolated task. Another can show a loss on a large project full of unwritten rules. Both can be right. So when reading any number, ask: which task, which people, which tool and on what date.

What controlled experiments found

The most reliable design for measuring an effect is the randomized controlled trial: people (or tasks) are randomly assigned to use or not use the tool, and the results are compared. Random assignment prevents the difference from coming from who chose to use AI.

GitHub Copilot, 2022. In an experiment conducted between May and June 2022, 95 programmers hired through the Upwork platform were randomly split into two groups and had to write an HTTP server in JavaScript. Those with access to Copilot finished the task 55.8% faster, with a 95% confidence interval between 21% and 89% (Peng et al., 2023). The authors note that developers with less experience and those who programmed more hours per day benefited more. The task was single, standardized and started from scratch.

Google, 2024. A trial with 96 of the company’s own engineers, using internal AI tools on a complex task, estimated that the AI group was about 21% faster (Paradis et al., 2024). The authors warn that the confidence interval is wide, that the result comes from a lab study and that it reflects Google’s tools in the (northern hemisphere) summer of 2024.

Three companies, 4,867 developers. A set of field experiments at Microsoft, Accenture and an anonymous Fortune 100 company gave a randomly selected group of developers an assistant that suggests code snippets. In the pooled analysis, the authors estimate a 26.08% increase in the number of completed tasks, with a standard error of 10.3% (Cui et al.). Here too, less experienced developers adopted the tool more and gained more. The authors themselves say each individual experiment is noisy.

Note that these studies mainly measure autocomplete assistants, which suggest the next line or block, not agents that carry out entire tasks.

The study that found the opposite

In July 2025, the research organization METR published a result pointing the other way. Sixteen experienced developers from mature open-source projects, with about five years of average experience on those projects, worked on 246 real tasks. Each task was randomly assigned to allow or prohibit AI. The main tools were the Cursor Pro editor with the Claude 3.5 and 3.7 Sonnet models (Becker et al., 2025).

The result: with AI, tasks took 19% longer. The most curious finding is about perception. Beforehand, developers predicted a 24% gain. Afterward, even though they were slower, they estimated they had gained 20%. Experts in economics and machine learning had predicted gains of 39% and 38%.

This study does not “refute” the earlier ones. It measures a different scenario: people who know the code deeply, on large projects, with high quality standards. In this context, the time spent reviewing and fixing AI output can outweigh the time saved.

The attempt to repeat the study

METR began a new round in August 2025, with 57 developers and more than 800 tasks. In February 2026, it published estimated gains of 18% for developers returning from the original study and 4% for new ones, but with confidence intervals that include zero (METR, February 2026).

METR itself classified this data as an “unreliable signal”. The main reason is revealing: many developers refused to participate because they did not want to work without AI, and between 30% and 50% of them stopped submitting some tasks for the same reason. This tends to exclude precisely the cases with the largest gains. The organization says it believes developers are probably getting more out of AI in early 2026 than in early 2025, but that its data is “very weak evidence” of the size of that increase.

In the editorial team’s assessment, the episode shows a practical limit: as AI becomes part of the job, it gets harder to set up a comparison group without it.

What benchmarks measure

A benchmark is a fixed set of problems used to compare models. In programming, the most cited is SWE-bench, created in 2023 with 2,294 tasks taken from real GitHub issues (problem reports) in 12 popular Python projects. The model receives the code and the problem description and has to edit the project until automated tests pass. In the original paper, the best model solved 1.96% of the tasks (Jimenez et al., 2023).

Then came SWE-bench Verified, a subset of 500 tasks reviewed by people to ensure the description is clear and the tests are correct (SWE-bench Verified). According to Stanford’s AI Index 2026, performance on this test rose from 60% to close to 100% in a single year (AI Index 2026).

This progress is real, but there are important caveats.

Contamination. Since the tasks come from public projects, models may have seen the solutions during training. A 2025 study asked models to identify which file contained the bug using only the problem description, without seeing the code. Accuracy reached 76% on SWE-bench Verified tasks and at most 53% on projects outside the benchmark, a gap the authors associate with memorization (Liang et al., 2025).

Harder tests tell a different story. SWE-bench Pro, from Scale AI, brings together 1,865 problems from 41 projects, some of them private, with tasks that can take a professional hours or days. In the November 2025 version of the paper, the authors report that models such as GPT-5 and Claude Opus 4.1 solved about 23% of the Pro tasks, compared with more than 70% on tests such as SWE-bench Verified (Deng et al., 2025).

How much human work time AI can cover

METR proposed another measure: the time horizon, defined as the duration (in human time) of the tasks a model completes with 50% success. In the March 2025 paper, models such as Claude 3.7 Sonnet had a horizon of about 50 minutes, and that horizon had been doubling roughly every seven months since 2019 (Kwa et al., 2025).

METR itself lists the limits of this metric: the tasks are mainly in software engineering, machine learning and cybersecurity, they are “cleaner” than real work, and a 50% horizon does not mean tasks of that duration can be safely delegated (METR, time horizons page).

Passing the test is not being ready

In August 2025, METR ran a small and illuminating test. It took 18 tasks from two open-source projects and let an agent based on Claude 3.7 Sonnet solve them. By the automated criterion, 38% of the solutions passed the tests. In a human review of 15 of them, none could be merged into the project as it was (METR, August 2025).

The most common problems were insufficient tests, missing documentation, non-standard formatting and other quality issues. Fixing each solution would take, on average, 42 minutes of human work. The original task had taken the maintainers 1.3 hours on average.

This result helps reconcile the numbers. The benchmark measures “the test passed”. The experienced developer needs “I can accept this into the project”. The gap between the two is human work that does not show up on the scoreboard. To understand how to evaluate AI output more fully, see how to evaluate AI responses.

What usage data shows

Another way to measure is to look at what people ask AI for. Anthropic analyzed 500,000 programming-related interactions between April 6 and 13, 2025. In Claude Code, a tool that acts as an agent, 79% of conversations were classified as automation (the AI performs the task), compared with 49% in the Claude.ai chat (Anthropic, April 2025).

The company cautions that the data probably reflects early adopters and that the analysis measures what is delegated, not the quality of the resulting code or whether there was a real productivity gain. In a January 2026 report, Anthropic noted that the model itself estimated software development requests would take a professional about 3.3 hours without AI, and rated 61% of those requests as successful, compared with 78% for personal tasks (Anthropic, January 2026). These are estimates made by the model, not time measurements.

We explain how this category of tool works in Claude Code: how it works, and where agents work and fail in the limits of autonomous agents.

How to read these numbers in your case

Putting the evidence together, some patterns emerge:

1. Well-defined, new tasks favor AI. Writing something from scratch with a clear specification is where measured gains are largest.
2. Deep context reduces the gain. On large projects, with implicit rules and high review standards, the gain can disappear or turn into a loss.
3. Less experience, greater relative gain. Two independent studies found this. It does not mean AI replaces knowledge: those with less background are also less able to spot errors.
4. Perception is not measurement. In the METR study, those who became slower believed they had become faster.
5. The numbers age quickly. Tools from 2022 to 2025 do not describe today’s. This applies to gains and to losses.

If you want to know the effect on your team, the way forward is to measure in your own context, with metrics defined in advance. We detail how to do this in productivity with AI. For the impact on the software process as a whole (review, testing, security and roles), see how AI is changing software development.

Key takeaways

• There is no single percentage of “automated work”. Speed, autonomy and delegation are different measures.
• Controlled trials with autocomplete assistants found gains of roughly 21% to 56% on specific tasks, with large uncertainty.
• On mature projects, with experienced developers and early-2025 tools, METR measured a 19% slowdown. The attempt to repeat the study ran into developers’ refusal to work without AI.
• Benchmarks have advanced a great deal, but suffer from contamination and from the gap between “passed the test” and “ready to merge into the project”.
• Use the numbers as an order of magnitude and measure in your own context before planning staffing, deadlines or budgets based on them.