Hi, this is Gergely with a bonus, free issue of the Pragmatic Engineer Newsletter. In each issue, I cover Big Tech and startups from the perspective of senior engineers and engineering leaders. Today, we explore one out of four topics from last week’s The Pulse issue. Full subscribers received the article below seven days ago. To receive articles like this in your inbox every week, subscribe here.
Many subscribers expense this newsletter to their learning and development budget. If you have such a budget, here’s an email you could send to your manager.
An intriguing study was released by the nonprofit Model Evaluation and Threat Research (METR). They hired 16 seasoned developers working on large open-source repositories to fix 136 real issues, paying $150/hour. Some developers used AI tools, while others did not. The study recorded the developers’ screens, analyzing 146 hours of footage. The findings:
“Surprisingly, we find that when developers use AI tools, they take 19% longer than without. AI makes them slower. (…) This gap between perception and reality is striking: developers expected AI to speed them up by 24%, and even after experiencing the slowdown, they still believed AI had sped them up by 20%.”
This result is very surprising! But what is happening? Analyzing the research paper:
The research investigates Cursor’s influence on developer productivity. Cursor, using Sonnet 3.5 or 3.7, was the main AI tool used by the participants. A total of 44% of developers had never used Cursor before, and most others had used it for up to 50 hours.
Developers using AI spent less time coding but took longer overall. They spent less time on research and testing but took longer promoting, waiting on the AI, reviewing its output, and on “IDE overhead” compared to those not using AI. Eventually, the extra time with the AI offset the time saved on coding, research, and testing, the study concluded.
It’s worth noting that this finding applies to all AI tools, not just Cursor, which was the chosen tool for this study.
Developers are overly optimistic about AI’s productivity impact – initially, at least. From the survey:
“Both experts and developers drastically overestimate the usefulness of AI on developer productivity, even after they have spent