Products · Deployment & application
Developer productivity uplift (METR RCTs)
Measured speed change for experienced developers using AI tools versus not, from METR's randomised trials. Negative means slower. The band applies to the newest estimate (Feb 2026, newly recruited developers).
Reason through thisThe reading
Two trials a year apart, and the best estimate of the speed-up is still close to zero. What holds people back is fitting the tool into their work and checking its output, not what the model can do.
2 readings drawn from metr_blog.rct_*.dev_speed_change.pt; 2 observations in the evidence.
Source: METR research posts. METR, Time Horizon 1.1 (29 Jan 2026).
Why it matters
The most carefully controlled test of whether better models make people faster at real work: randomised trials in which experienced software developers do their own tasks with and without AI. If benchmark gains turned quickly into real gains, it would show here first.
What cuts against it
Small samples; the 2026 redesign has selection effects METR itself calls 'very weak evidence'; experienced open-source maintainers on their own repos are the hardest case for uplift.
How it is measured
- series
metr_blog.rct_*.dev_speed_change.pt - source METR research posts · default tier 6 · METR
- proxy product, deployment · cadence annual · valve invention to product
Normal = uplift of at most about 20%, setting-dependent (the range of controlled studies to date); fast = broad uplift above 40%, the scale the productivity-boom claims need.
Applied to metr_blog.rct_2026.dev_speed_change.pt.
Latest readings
- 2026-02-24 rct_2026 · -4.0%⚑disputed: METR itself calls this 'very weak evidence' because developers refusing the no-AI arm biases the sample.as of 2026-02-24
- 2025-06-30 rct_2025 · -19.0%⚑disputed: Sixteen experienced open-source maintainers working on their own large repositories with early-2025 tools; METR cautions against reading it as a general productivity effect.as of 2025-06-30
Why this status, and its history
METR's Feb 2026 redesign estimates a -4% speedup for newly recruited developers (CI -15% to +9%; METR calls it very weak evidence); the 2025 RCT found developers 19% slower. Both are far below the 20% ceiling of the normal band. Initial seed.
- 2026-09-10unmeasured to consistent with normalconf — → 70 · initial seed (model-drafted)
METR's Feb 2026 redesign estimates a -4% speedup for newly recruited developers (CI -15% to +9%; METR calls it very weak evidence); the 2025 RCT found developers 19% slower. Both are far below the 20% ceiling of the normal band. Initial seed.
source1
Why this confidence
Rests on 2 observations from METR research posts; best evidence tier 1, grade A; current.
Confidence is independent of status: 90–95 multiple strong independent sources; 70–89 good evidence, some ambiguity; 50–69 mixed or hard to operationalise; 0–49 limited or vague evidence.
Predictions that lean on this
- HappeningEach test AI passes shows the test's limits, not that AI has arrived. · Narayanan & Kapoor
- Slower than said
- Slower than said
- Too early to tell
- Too early to tellWilliam MacAskill and Fin Moorhouse think it fairly likely that AI matches human research labour within two decades, and possibly well within one. · William MacAskill and Fin Moorhouse
More predictions that lean on this
- Too early to tell
- Too early to tellAI costs too much for the problems it can solve today. · Jim Covello (Goldman Sachs)