Autonomous AI agents: where they work and where they fail

An autonomous AI agent is a system in which a language model receives a goal, decides on the steps, uses tools and checks the result, with little or no human intervention along the way. If you want to review the mechanism, see what AI agents are.

The question in this article is a different one: what does the evidence say about where these agents work and where they fail? To answer it, we use benchmarks (standardized sets of tasks used to compare systems), independent studies and technical reports from the companies themselves.

A warning before any number: agent benchmarks age quickly. A result from six months ago may already be outdated, and a test considered hard in 2024 may be practically solved today. That is why every figure below is dated. Treat them as snapshots, not permanent truths.

How an agent is measured

Agent benchmarks try to reproduce real tasks in controlled environments. Four of them help explain how things have evolved:

• SWE-bench (programming). Launched in October 2023, it brings together 2,294 real problems taken from GitHub issues (bug reports) and pull requests (code change requests) in 12 popular Python repositories. The agent has to change the code so that the tests pass. At the time, the best model evaluated, Claude 2, solved 1.96% of the problems.
• WebArena (web navigation). Published in July 2023, it simulates realistic websites such as stores and forums. The best GPT-4-based agent completed 14.41% of the tasks; humans completed 78.24%.
• OSWorld (computer use). Launched in April 2024, it has 369 tasks in real desktop and web applications, files and workflows across programs. Humans solved more than 72.36%; the best model, 12.24%. The authors identified locating and operating graphical interface elements, and practical knowledge of how to use the programs, as the main difficulties.
• τ-bench (tool-assisted customer service). Published in June 2024, it has the agent talk to a simulated user, following a company’s policies and using internal systems. GPT-4o solved fewer than 50% of the tasks.

τ-bench introduced an important idea: the pass^k metric, which checks whether the agent gets the same task right in all k attempts, not just in one. In the simulated retail setting, the success rate across eight consecutive attempts fell below 25%. For a business, this matters more than the average success rate: a service agent that sometimes resolves the same request and sometimes does not is hard to put into production.

The leap from 2024 to 2026

The starting numbers were low. The most recent ones are not. According to the AI Index 2026, the annual report from Stanford University’s Institute for Human-Centered AI (HAI), performance on SWE-bench Verified, a revised version of SWE-bench, went from 60% to close to 100% in one year. On OSWorld, agents’ success rate rose from 12% to about 66%. The same report notes that agents still fail roughly one in three attempts on these structured tests.

Measuring by task duration

METR, an independent organization that evaluates AI systems, proposed another way to track progress in March 2025: the 50% time horizon. It is the length of tasks, measured by how long a human expert takes to do them, that a model completes with a 50% chance of success. In the original paper, Claude 3.7 Sonnet had a horizon of about 50 minutes, and the indicator had been doubling roughly every seven months since 2019.

With an expanded version of the task set, published in January 2026, METR estimated that since 2023 the horizon has doubled about every 131 days, a little over four months. The organization itself warns that the trend is sensitive to the composition of the tasks.

The data published by METR, consulted in October 2026, show a revealing detail. For Claude Opus 4.6, released in February 2026, the 50% horizon was estimated at about 12 hours, with a very wide confidence interval of roughly 5 to 60 hours. The 80% horizon, which demands more reliability, was about 70 minutes. In other words: the same system that sometimes completes full-day tasks is only reliable, four times out of five, on tasks of just over an hour.

METR’s tracking page makes two caveats. Measurements above 16 hours are not reliable with the current task set. And the tasks are mostly in software engineering, machine learning and cybersecurity, with little prior context, which may not transfer to messier real-world work.

Why to distrust the leaderboards

Contamination and flawed tests

In February 2026, OpenAI, which had created SWE-bench Verified in 2024, announced it would stop using it. Two reasons were given. First, in an audit of 138 hard problems, 59.4% had material flaws in the tests or the description, to the point of rejecting functionally correct solutions. Second, every frontier model tested could, in some cases, reproduce from memory the original human-written fix, a sign that they had seen the problems and solutions during training. This is called contamination: the model “studied the exam” before taking it.

The company’s conclusion was that gains on this test had come to reflect exposure to the dataset more than real capability. It is a textbook case of why a number close to 100% does not mean a solved problem.

Cost left out of the equation

The paper AI Agents That Matter, by Kapoor and co-authors, published in July 2024, criticized agent evaluation for focusing only on accuracy. According to the authors, this favors needlessly complex and expensive agents, makes it easier to take shortcuts that exploit flaws in the test (for lack of separate validation sets) and makes results hard to reproduce. Their proposal is to evaluate cost and accuracy together.

Passing the test is not being ready

In August 2025, METR compared automatic grading with human review. An agent based on Claude 3.7 Sonnet passed the maintainers’ tests in 38% of runs, on 18 tasks from open-source projects. Of the manually reviewed solutions, none could be merged without changes: tests, documentation or code quality were missing. The sample is small, but it suggests that automatic grading tends to overestimate practical usefulness.

The same group published an experiment in July 2025 in which experienced developers, using early-2025 AI tools, took 19% longer to complete real tasks, despite believing they had gained speed. We discuss that study, and how to measure gains on your team, in how to measure productivity in AI projects.

Where agents work today

Putting the evidence together, the editorial team’s assessment is that agents perform best when four conditions appear together:

1. There is objective verification. Programming is the clearest case: automated tests say whether the change worked. Anthropic, in its December 2024 piece Building effective agents, highlights programming and customer service as promising areas and recommends that the agent obtain “ground truth” from the environment at each step.
2. The task is bounded. Time horizons show that reliability drops with duration. Short, well-specified tasks have a much better chance of success than long, ambiguous projects.
3. Errors are reversible. An agent that proposes a code change, reviewed before it enters the system, makes mistakes at low cost. One that executes payments does not.
4. The environment is controlled. Anthropic recommends extensive testing in isolated environments (sandboxes) and appropriate guardrails.

In the editorial team’s assessment, this profile matches coding agents that work in a repository, run tests and deliver a proposal for human review.

Where they still fail

Compounding errors

Each step depends on the previous one. Anthropic sums it up: the autonomous nature of agents brings higher costs and the potential for compounding errors. A wrong assumption at the start becomes the basis for everything that follows, and a hallucination stops being merely wrong text and becomes a wrong action.

Consistency

Getting it right once is not enough. τ-bench’s pass^k and the gap between METR’s 50% and 80% horizons tell the same story: the average hides the variation. For processes that need to work the same way every time, that variation is the central problem.

Real-world interfaces and context

In the original OSWorld, the greatest difficulty lay in operating the graphical interface. And METR notes that its tasks require little prior context. Real work usually depends on a company’s tacit knowledge, legacy systems and unwritten rules, something benchmarks capture poorly.

Prompt injection

An agent that reads web pages, emails and documents may encounter malicious instructions hidden in that content. The AgentDojo test environment, from June 2024, with 97 realistic tasks and 629 security test cases, showed that advanced models failed many tasks even without an attack and that existing attacks broke some of the security properties.

In November 2025, Anthropic reported a 1% rate of successful attacks against its browser agent, measured with an adaptive attacker that had 100 attempts per environment, and stated that this level, although an improvement, still represents meaningful risk. In the company’s words, “no browser agent is immune” to this kind of attack.

Cost and oversight

Each turn of an agent’s loop consumes compute and tokens. More elaborate agents can cost much more without being more accurate, as Kapoor and co-authors showed. And human review, necessary for the reasons above, also has a cost: if reviewing the agent’s work takes almost as long as doing it, the gain disappears.

How to read the next announcement

When a new record appears, a few questions help separate signal from noise:

• What is the date and version of the test? Compare only results from the same benchmark, in the same version.
• Could the test be contaminated? Public and older sets are at greater risk.
• How many attempts? Success on one attempt is different from consistent success.
• How much did it cost? Without a cost per task, there is no way to compare solutions.
• Who graded it? An automated test, human review or the vendor itself?
• Does the task resemble yours? Duration, context and reversibility make a difference.

For general checking criteria, see also how to evaluate AI responses.

Takeaways

The evidence up to October 2026 shows real and rapid progress: agents that solved almost nothing on 2023 tests now do well on software tasks with automatic verification, and the size of the tasks they can complete grows every few months.

At the same time, the best-known leaderboards have become contaminated or saturated, reliability drops considerably when consistency is required, and security against prompt injection remains incomplete. Use agents where there is objective verification, bounded tasks and reversible errors. In other cases, keep a person in control and treat any benchmark number as a dated snapshot.