Language and display
BE THE BOSS OF AI.ADRIAN DUNKLEY ↗
← This newspaper edition

AI Wrote the Code. How Long Until It Works?

A quick software draft is only part of the job. Count the time spent checking, fixing and approving it before claiming that AI saved your business time.

A silver stopwatch with a white numbered dial rests at an angle against a black background.
William Warby / Unsplash · Illustrative photograph · Licence

If you pay someone to fix your website, the first piece of code is not the finished job. The change still has to pass tests and review. Measure how long that takes, including the time people spend correcting it, before deciding whether an AI coding tool is saving money.

In a randomised trial by Model Evaluation & Threat Research, 16 experienced developers completed 246 tasks on open-source projects they already knew. The researchers assigned each task to allow or prohibit generative AI. When access was allowed, developers mainly used Cursor Pro with Claude 3.5 or 3.7 Sonnet. The researchers estimated that allowing AI increased the time spent completing the work by 19%. Participants later estimated that access had cut their time by 20%.

The clock kept going while developers checked and corrected the code

Developers logged the time they spent implementing changes and responding to review. For a small number of tasks without a review record, the researchers estimated the time that responding to review would have added. The 95% confidence interval was 2% to 39% longer with AI allowed. In 74 tasks with usable, labelled screen recordings, developers spent a smaller share of time writing code or looking up information when AI was allowed. Prompting the tool, waiting for replies and reading its output occupied some of their time.

The trial covered experienced contributors using particular tools from February to June 2025. In a February 2026 follow-up, METR said it could not reliably measure the effect of newer tools in its next trial. Some developers or tasks were left out because people were reluctant to work without AI. Developers also reported difficulty timing a task when they worked on something else while an agent ran.

Measure elapsed time and staff effort

For a local comparison, ask your software team to select similar low-risk fixes and randomly assign which may use the tool. Apply the same tests and review standard to both groups. For each fix, record when work starts and when the software change passes review and tests. Ask developers and reviewers to record the minutes they spend on the fix. If a developer switches tasks while an agent runs, that wait can affect the delivery time without adding staff minutes to the fix.

Suppose a fix starts at 10 am and clears review and tests at noon. The developer logs 50 minutes and the reviewer logs 15. That is two hours to clearance and 65 staff minutes. Keep unfinished fixes in the record. Use elapsed time to plan when a fix can reach users and staff minutes to see whether the tool changes the effort needed to clear a fix.

Sources and further reading

Sources checked 9 November 2026. Edited and reviewed by AI Boss.