How Do You Actually Measure an AI Agent's Ability?
A METR research note by Tom Cunningham lays out a taxonomy of ways to score AI agents against human performance and against their own spending.
What Cunningham is trying to fix
Most leaderboards report a single score for an AI agent at one fixed amount of spending, whether that spending is money, tokens, or time. In a new research note, METR researcher Tom Cunningham argues that this single number can be misleading, because an agent's score often keeps rising the more it is allowed to spend, while a weaker agent may simply plateau earlier. The note sets out to organize the many different ways people already try to work around that problem.
The core idea: score as a function of spending
Cunningham frames every agent and every human as a curve: a score that changes as spending on the task increases. Some tasks show scores that barely move once a modest amount is spent, in which case a single reported score is fine. Other tasks show scores that keep climbing the more an agent spends, which Cunningham says is where a plain leaderboard number becomes a poor sufficient statistic for ability, since two agents that look similar at one spending level can behave very differently at another.
Ways to score an agent on its own
The note walks through metrics that never touch human data at all. "Score at fixed expenditure" is the ordinary leaderboard approach, useful only when a score reliably levels off at a low spending level. "Score at practical plateau" instead reports whatever score a model reaches once its improvement slows or stops, following a line Cunningham quotes from Kwa and West et al. (2025), who gave models "sufficiently high token limits to reach a plateau." A third option, "expenditure at fixed score," flips the question and asks how much spending is needed to reach a target score at all, which the note ties to tracking how model costs have fallen over time.
Calibrating against human effort
A separate family of metrics grades an agent against a human doing the same task. Cunningham points to METR's own "time horizon" metric, credited to Kwa and West et al. (2025), which turns many pass/fail results into a human-equivalent time by finding the spending level at which an agent matches a human's score about half the time. He also cites the Opus 5 system card, which graded a series of AI R&D tasks against how long a human needed to reach the same result, using thresholds such as 8 hours or 40 hours. Cunningham and coauthors go further with an "expenditure horizon," introduced in Cunningham et al. (2026) and built on the apple-picking model from Cunningham and Shetty (2026): the point at which an agent stops matching or beating a human at the same spending level.
Key facts
- The note is a research post by METR researcher Tom Cunningham, published 2026-07-24.
- It compares agent ability against spending measured as money, tokens, or time.
- METR's time horizon metric finds the spending level at which an agent matches a human's score about half the time.
- An example cited in the note graded human baselines at thresholds like 8 hours or 40 hours on the Opus 5 system card.
- Cunningham describes the piece as a conceptual taxonomy of metrics, not a finished evaluation standard.
Check your understanding
- Why does reporting an agent's score at just one fixed expenditure level hide real differences between agents?
- What does METR's time horizon metric compare between an agent and a human doing the same task?
- What thresholds does the Opus 5 system card example use to grade human performance on AI R&D tasks?