There is a particularly good AI experiment from METR that I keep thinking about because it measured something almost nobody measures: whether AI coding tools actually make experienced developers faster.
METR: Early-2025 AI Experienced Open-Source Developer Study
METR recruited 16 experienced developers working on large open-source projects they knew extremely well. These weren’t juniors learning React from YouTube. The developers had, on average, about five years of experience with their respective repositories, and those repositories averaged more than 22,000 GitHub stars and over a million lines of code.
They were given 246 real tasks: bugs, features and refactoring. Some were completed with AI tools, primarily Cursor Pro with Claude 3.5/3.7 Sonnet. Others were completed without AI.
Before the experiment, the developers predicted that AI would make them about 24% faster.
Then METR measured what actually happened. They were 19% slower. Not 19% less productive according to some subjective code-quality score.
They literally took longer to finish the tasks.
The Best Part
After completing the study, the developers were asked again how much AI had helped them. They still believed it had made them about 20% faster. This may be the most interesting result of the entire experiment.
They had spent additional time prompting, waiting, reviewing generated code, correcting mistakes, reverting changes and telling Claude some variation of:
No, not like that.
And still walked away thinking: Wow, I’m moving fast.
There is something psychologically convincing about watching a machine generate 200 lines of code in seconds.
Typing those lines manually would obviously take longer. But typing isn’t necessarily the bottleneck.
Understanding an unfamiliar change, discovering that it subtly violates an architectural assumption, explaining the problem to the model, waiting for another attempt, reviewing that attempt and eventually fixing it yourself can easily cost more than simply writing the correct 30 lines in the first place.
The generation feels fast. The workflow isn’t necessarily fast.
This Doesn’t Mean AI Coding Is Useless
And this is where people usually turn an interesting result into a stupid binary argument.
“See! AI coding doesn’t work.”
No. That’s not what the experiment established.
It studied a particular population doing a particular kind of work in early 2025: highly experienced developers working inside mature repositories they already understood extremely well.
That’s almost the worst possible environment for AI to demonstrate an information advantage.
The human already knows the codebase. They know where things live. They know the conventions. They know which innocent-looking abstraction will cause three horrible problems six months later.
AI has to reconstruct that context. The expert already has it in their head.
Give the same tools to someone entering an unfamiliar repository, doing boilerplate work, prototyping something new, or operating with a much more autonomous 2026 agent stack, and I would expect a different result.
METR itself has emphasized that the finding should not simply be generalized to every developer or every use of AI coding tools. But that’s exactly why I like the study.
It attacks one of the most dangerous assumptions in the current AI boom: feeling faster is not the same thing as being faster.
Measure the Whole Loop
This applies far beyond programming. AI writes an email in ten seconds. Great.
How long did you spend correcting it? AI creates a presentation in three minutes. Great.
How long until the presentation is actually good? An agent implements a feature while you drink coffee. Excellent.
How much time did you spend reviewing the PR, discovering what it broke and explaining the task again?
We tend to measure the spectacular part because that’s the part AI makes spectacularly fast. We don’t measure the boring remainder. And increasingly, the boring remainder is where the work moves.
Generation gets cheaper.
Judgment, context, verification and integration become the bottleneck.
Which brings us to a wonderfully uncomfortable possibility.
Maybe we’re not accelerating nearly as much as we think. Maybe we’re just watching the progress bar move much faster. And apparently we’re surprisingly bad at telling the difference.