*This article is reprinted fromQuantum Bit WeChat Official Account
Agent capabilities double every 7 months!
According to the latest report released by the non-profit research organization METR, this pattern has been validated across 9 benchmark tests.
These tasks involve programming, mathematics, computer usage, autonomous driving, and more, indicating that large models are continuously advancing towards high levels of automation.
The report states: In tasks such as software development, mathematics competitions, and scientific Q&A, agents can now complete tasks that would take humans 50–200 minutes to accomplish, and this capability is rapidly improving—approximately every 2–6 months it can double.
In computer operation tasks, although the task duration is shorter, the growth rate is consistent with that of software development tasks.
The performance growth rate of agents in autonomous driving tasks is slower, doubling approximately every 20 months.
In video understanding tasks, models can achieve a 50% success rate on videos lasting 1 hour.
As a research team dedicated to studying the capabilities and risks of cutting-edge AI systems, METR’s report further narrows the timeline for AI autonomy. Let’s take a look at the contents of the report.
Moore’s Law for Agents
In previous tests, METR focused on evaluating software development and research tasks, discovering that the capabilities of AI agents exhibit a “Moore’s Law” growth trend— on average, every seven months, the time horizon for tasks they can complete doubles.
In the latest report, METR has expanded this evaluation method to a broader range of fields and continues to ask a key question: Can AI’s capabilities continue to leap in a doubling manner across a wider range of tasks?
However, we must first ask, what is the time horizon?
For example, if humans take an average of 30 minutes to complete a task, and AI has a success probability of half on such tasks, we say its time horizon is 30 minutes. If its success rate is much higher, say 80%, it indicates that it can handle longer and more complex tasks.
In summary, the time horizon is the stable time span within which an agent can complete tasks.
Since a longer time horizon ≈ more difficult tasks ≈ requires more strategic reasoning and planning capabilities ≈ higher intelligence level of the agent, the doubling of the time horizon is also referred to as Moore’s Law for agents.
Given the significant differences in AI capabilities across different tasks, the current question is: Will this exponential growth pattern hold true in other fields as well?
How to Measure Time Horizon Across Domains?
To address the above question, the report selected 9 benchmarks, including software development (METR-HRS, SWE-bench), computer usage (OSWorld, WebArena), mathematics competitions (Mock AIME, MATH), programming competitions (LiveCode-Bench), scientific Q&A (GPQADiamond), video understanding (Video-MME), autonomous driving (Tesla FSD), and robotic simulation (RLBench).
For each benchmark, METR constructed probabilistic models to estimate the agent’s time horizon. The report used maximum likelihood estimation (MLE) or simplified estimation methods to handle the label granularity of different benchmarks to estimate the growth curve of AI’s time horizon over time in each field.
It is noteworthy that the time horizon boundaries of different benchmark tests differ by more than 100 times. Many reasoning and coding benchmark clusters have time spans of 1 hour or more, while the time for computer usage (OSWorld, WebArena) is only about 2 minutes, which may stem from misclicks by the agent while using the mouse.
Research Findings: Agent Capabilities Double Monthly
In addition to the changes in agent capabilities mentioned at the beginning, the report also tested the capabilities of several mainstream large models. For instance, cutting-edge models like o3 have consistently performed above trend levels on METR tasks, with a doubling time faster than 7 months, and a median doubling time of about 4 months across 9 benchmark tests (ranging from 2.5 to 17 months).
Finally, the time horizon is not equally important across all foundational tests. In some benchmarks, the difficulty of challenging problems is far greater than that of simple problems, while in others, the difficulty of challenging problems is not much different from that of simple problems. Therefore, for agents, the time horizon in these benchmark tests may not fully reflect their performance.
For example, the difficulty of LeetCode (LiveCodeBench) and mathematics problems (AIME) is much higher than that of simple problems, but the Video-MME problems on long videos are not significantly more difficult than those on short videos.
It is evident that the performance of agents is not solely about “knowing more techniques” but rather about whether they can handle longer and more complex tasks.
From seconds and minutes to tens of minutes and hours, the range of tasks that agents can handle is increasing; if the doubling trend continues, we may see AI completing tasks that take “days to weeks” in the coming years.
In summary, this research reveals a clear pattern: from code reasoning to mathematics competitions, from GUI control to autonomous driving, no task domain shows signs of diminishing intelligence growth. In most scenarios, AI is rapidly evolving towards greater spans, deeper memory, and more complex planning.
Reference Links:[1]https://arxiv.org/abs/2503.14499[2]https://metr.org/blog/2025-07-14-how-does-time-horizon-vary-across-domains/
Related Reading
2025 Global AI Survey: Trust, Attitudes, and Current Applications
What Are the Opportunities and Potential of AI in the Future? A Joint Report by the World Economic Forum, Accenture, and KPMG
<< Swipe to view the next card >>