Code 2025
Why Agent Hype can fall short of reality – Joel Becker, METR
Overview
This talk addresses the discrepancy between AI capabilities suggested by benchmarks and real-world performance, particularly in developer productivity. It introduces two distinct methods of evaluation: benchmark-style assessments measuring AI performance on diverse tasks against human baselines, and field experiments examining AI's impact on experienced developers in complex, real-world coding environments. The core thesis is that while benchmarks show rapid AI advancement, practical application, especially in messy, high-context scenarios, reveals a more nuanced and sometimes even negative impact on productivity.
Who should watch
- AI Engineers
- Product Managers
- Builders evaluating AI tools
- Researchers interested in AI capability measurement
- Developers curious about AI's impact on coding
Key takeaways
- Traditional AI benchmarks, like SWE Bench or GPQA, can be saturated quickly and may not accurately reflect real-world performance due to low-context human baselines and simplified task environments.
- A novel approach involves measuring AI performance by comparing its completion time on tasks to human expert completion times, establishing a "time horizon" for AI capabilities.
- This time horizon metric shows a remarkably steady, near-exponential trend of AI progress over calendar time, even for complex research engineering challenges.
- Field experiments with highly experienced developers on large, mature open-source projects revealed that AI tools, on average, *slowed down* developers by 19% compared to not using AI.
- Potential reasons for this slowdown include overoptimism about AI usefulness, the time cost of instructing AI versus direct typing for experts, low AI reliability requiring extensive verification, and the inherent messiness and interdependencies of real-world development tasks.
- The study suggests that AI reliability needs to be exceptionally high (e.g., 95-99%) to yield significant time savings in complex development scenarios.
- The findings highlight a gap between benchmark-indicated AI prowess and its practical translation into productivity gains, especially in high-context, complex software engineering.
- Future research aims to explore these discrepancies further with larger studies and more diverse evidence sources.
Notable quotes
*Benchmarks seem to have less and less time between coming online and being fully saturated.*
*We find that developers are slowed down by 19%. They take 19% more time when AI is allowed relative to when AI is not allowed.*
Unofficial community note. Prefer the recording for nuance.