Vibe Coding: What the Productivity Studies Actually Found

Overview

The claim that AI has collapsed the cost and time of building software is widespread, rarely measured, and has been tested at least once under controlled conditions with a result that runs directly against it.

In a randomised controlled trial published in July 2025, METR found that experienced open-source developers took 19% longer to complete real coding tasks when allowed to use AI tools. The same developers estimated afterwards that AI had made them 20% faster.

That 39-point gap between measurement and perception is the most important finding in this area, because almost every confident claim about AI productivity rests on self-reported impressions of exactly the kind the study found to be wrong.

Survey data points the same way. Adoption is near-universal at 84% of developers, while trust in AI accuracy has fallen to 29%, more developers actively distrust it than trust it, and two thirds report spending more time fixing AI-generated code that is almost right.

A Randomised Trial Found the Opposite

METR's study is notable for its design. Rather than surveying developers about how AI affected them, it randomised whether they were permitted to use AI tools on particular tasks and measured completion times.

The setup involved 16 experienced open-source developers working on 246 real tasks in mature repositories they already knew well, using tools available between February and June 2025 — principally Cursor Pro with Claude 3.5 or Claude 3.7 Sonnet, including chat, agent mode and autocomplete.

The measured result was a 19% increase in completion time when AI tools were allowed. Not a smaller improvement than expected — a slowdown, on real work, among developers using the tools they had chosen.

The 39-Point Perception Gap

The finding that should change how this subject is discussed is not the slowdown itself but the disagreement between what happened and what participants believed had happened.

Developers who were measurably 19% slower estimated they had been 20% faster. The gap between measurement and perception is roughly 39 percentage points, and it appeared among experienced engineers reflecting on work they had just completed.

This matters because virtually all public evidence for AI coding productivity is of the type the study invalidated. Surveys asking developers whether AI makes them faster, vendor case studies quoting perceived time savings, and personal accounts of dramatic acceleration all measure impressions — and impressions were wrong by nearly forty points under controlled conditions.

Why the Study Is Narrow

The finding deserves its caveats stated plainly, because overstating it would repeat the error it exposes.

Sixteen developers is a small sample. All were experienced contributors working in mature codebases they knew intimately — precisely the situation where AI assistance should help least, because the developer already holds the context an AI must infer. The result may not transfer to unfamiliar code, junior developers or greenfield projects.

METR itself now labels the result as historical, noting it does not necessarily reflect current AI tools or workflows, and has revised its experimental design since. Tools available in early 2025 are not those available now.

What survives all of that is the perception gap. Even if current tools genuinely produce speedups, the finding that developers cannot accurately assess their own productivity effect stands — and that undermines the evidence base, not just one result.

84% Use It, 29% Trust It

Survey data describes a workforce that has adopted these tools comprehensively while becoming steadily more sceptical of them.

Around 84% of developers now use AI tools. Over the same period, trust in the accuracy of AI output fell to 29%, down from 40%.

Adoption rising while trust falls is an unusual combination and a coherent one. It describes tools that are useful enough to be indispensable and unreliable enough that experienced users have learned not to believe them. Developers are not rejecting AI; they are using it while checking its work.

More Distrust Than Trust

The distribution of opinion is sharper than the headline trust figure suggests. More developers actively distrust the accuracy of AI tools, at 46%, than trust it, at 33%.

Only 3% report that they highly trust AI output. Among the people who use these tools most intensively and understand best what they produce, near-complete confidence is almost nonexistent.

That is worth holding against marketing describing AI as an autonomous developer. The population with the most direct experience of the output has converged on treating it as a capable assistant requiring verification — which is a considerably narrower claim than the one being sold.

Almost Right Is the Worst Outcome

The top frustration reported by developers, cited by 45%, is dealing with AI solutions that are almost right but not quite. Two thirds say they spend more time fixing this kind of output.

Code that is obviously wrong is cheap. It fails immediately, the error is visible, and it gets discarded in seconds. Code that is almost right is expensive: it compiles, it runs, it passes casual inspection, and the defect surfaces later in a context where the cause is no longer obvious.

This is the mechanism most likely to explain the METR slowdown. Generating a plausible solution is fast; determining whether a plausible solution is actually correct takes as long as understanding the problem properly would have. The time saved in production is spent again in verification.

Why 'Almost Right' Costs More Than Wrong

The economics of reviewing generated code are worth setting out, because they explain why raw generation speed does not translate into delivery speed.

Writing code and verifying code are different activities with different costs, and only the first has been accelerated. To confirm that a generated function is correct, a developer must understand the problem, read the implementation, and reason about edge cases — most of the intellectual work of having written it.

Trust would resolve this, and trust is precisely what the survey data shows is absent. At 3% high trust and 46% active distrust, essentially every developer is reviewing output carefully. The verification cost is being paid on nearly every generation, which is why measured throughput can fall even as the volume of code produced rises.

Where It Genuinely Works

None of this means AI coding tools lack value, and the conditions under which they help are reasonably clear from the same evidence.

The METR result came from experienced developers in mature codebases they knew well. The inverse case is where the benefit concentrates: unfamiliar languages and frameworks, greenfield projects with no existing conventions, boilerplate with well-established patterns, and exploratory prototypes where correctness matters less than seeing something run.

Prototyping is the strongest case. Building something quickly to test whether an idea is worth pursuing is genuinely transformed when a working demonstration takes an afternoon rather than a week — and in that context, code that is almost right is entirely sufficient because it is going to be thrown away.

The Prototype-to-Production Gap

The claim that anyone can now build software is true in a narrow sense and misleading in the way that matters.

Producing something that works once, for its author, on their machine, is genuinely accessible now in a way it was not before. That is a real democratisation and it should not be dismissed.

Production software is a different artefact. It handles inputs nobody anticipated, fails safely, protects user data, survives dependency updates, and remains comprehensible to whoever maintains it in three years. Those properties come from understanding the system, and generating code without understanding it produces something that works until it does not — at which point nobody involved can diagnose why.

The honest position is that the barrier to a working prototype has genuinely collapsed, and the barrier to maintainable production software has moved rather than disappeared. It now sits at the point where someone has to understand what was generated.

Conclusion

The one controlled measurement of this question found the opposite of the popular claim. Experienced developers using AI tools on real tasks took 19% longer, while estimating they had been 20% faster — a 39-point gap between what happened and what they believed happened.

The study is narrow, its authors now describe the result as historical, and the tools have moved on. The perception gap is the durable finding, because it means the self-reported evidence underpinning most productivity claims is exactly the kind that proved unreliable.

Developers themselves have arrived somewhere sensible. Adoption is near-universal at 84%, trust in accuracy has fallen to 29%, only 3% highly trust the output, and two thirds spend more time fixing code that is almost right. That is a workforce using a useful tool and verifying its work.

The gate genuinely is open for prototypes. Building something that runs is now within reach of far more people than before, and that is a real change. Building something that keeps running, safely, for other people, still requires understanding what the code does — and that is the part no amount of generation speed addresses.

Data Source and Attribution

METRStack Overflow Survey 2025METR (arXiv)

Productivity findings come from METR's randomised controlled trial 'Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity', covering 16 developers and 246 tasks using tools available between February and June 2025. METR has since described this result as historical and revised its experimental design, and that caveat is reflected in the text. Adoption, trust and frustration figures come from the 2025 Stack Overflow Developer Survey and associated Stack Overflow reporting.

FactsFigs reviews, cleans, and cross-checks every source dataset before shaping it into a data story. Each visualization is created and designed in FactsFigs Design Studio — an internal tool developed and owned by FactsFigs — and is the original work of a FactsFigs author, not an AI-generated copy of any existing graphic. Individual assets within a visual may or may not be produced with AI tools, but the design of the visual itself is solely FactsFigs' own.

Figures are estimates at the time of publication, provided for information only — nothing here is financial advice or a guarantee of accuracy.

2026-07-20