Ask sixteen experienced open-source developers how much faster AI tools would make them, and they'll say about 24%. Ask them again right after they finish the task, and they'll still swear they were roughly 20% faster. The stopwatch says otherwise: they were actually 19% slower. That gap is the headline result from a 2025 randomized controlled trial run by METR , a group that studies how AI systems actually perform in the wild rather than on benchmarks. Instead of testing beginners on toy pr...