Skip to content

Productivity studies — deep dive

The evidence pack summarizes four productivity findings in a single table. This page expands each into the population studied, the method used, the headline number, the things the study did not measure, the legitimate criticisms in the literature, and a short note on how to interpret the result for a small, mid-size, or enterprise team.

Why this page is structured this way

Productivity claims about AI-assisted development travel without their methodology. A 55.8% time reduction in a sandbox HTTP-server task is not a 55.8% throughput uplift in production — but the latter is what leadership decks tend to claim. Each study below is presented with the metric the authors actually measured, then the things the study cannot tell you.

1. The Copilot RCT (Peng et al., 2023)

Citation. Peng, Kalliamvakou, Cihon, Demirer, "The Impact of AI on Developer Productivity: Evidence from GitHub Copilot", arXiv:2302.06590, February 20231. Funded by GitHub. Methodology and dataset disclosed.

Population and method

  • n = 95 professional developers, recruited via a freelance marketplace.
  • Randomized into a treatment group (Copilot enabled) and a control group (no Copilot).
  • Single task: implement an HTTP server in JavaScript from a written specification, in a sandbox with no production constraints.
  • Outcome metric: task completion time, plus self-reported productivity, satisfaction, and enjoyment.

Headline finding

The treatment group completed the task in 1h 11m; the control group in 2h 41m. That is 55.8% faster, p < 0.0001. Self-report measures of productivity, satisfaction, and enjoyment all rose in the treatment group [primary, vendor-funded].

What this study did not measure

  • Production code quality. The output was a sandbox HTTP server, not reviewed code merged to a long-lived repository.
  • Defect rate. No follow-up measurement of bugs found later.
  • Maintenance cost. Code was discarded after the task.
  • Team-level delivery throughput. A single-developer task says nothing about how a team's lead-time changes.
  • Effect on senior developers in mature codebases. Recruitment was via a freelance marketplace; the task was greenfield.

Legitimate criticisms

  • Vendor funding. GitHub designed and funded the study. The authors disclose this; it is still a structural conflict.
  • Single-task generalization. One sandbox task cannot stand in for the variety of real engineering work — debugging, integration, review, release.
  • Self-report bias on satisfaction. Participants knew which arm they were in. METR's later study (below) makes the gap between perceived and actual productivity central.

How to interpret it for your size tier

  • Small (< 50 people): the study most resembles your reality if your engineers spend a lot of time on greenfield scripting, glue code, and scaffolding. Expect substantial speed-ups on those tasks. Do not extrapolate to integration work in legacy systems.
  • Mid (50–500): treat 55.8% as a ceiling for generation-heavy tasks in isolated sandboxes. Plan for review overhead to absorb a meaningful fraction of the gain.
  • Enterprise (500+): the population, task, and setting are unlike yours. Use the study to justify trying AI tools, not to set targets.

2. METR — early-2025 experienced open-source developer study

Citation. METR, "Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity", July 20252.

Population and method

  • 16 experienced open-source maintainers working on large, mature repositories they already knew well.
  • Randomized within-subject design. For each task, the developer was randomly assigned AI-allowed or AI-not-allowed.
  • 246 real-world tasks drawn from the maintainers' actual issue trackers — bug fixes, feature additions, refactoring.
  • Outcome metric: completion time per task, plus a self-report of expected and observed productivity change.

Headline finding

Developers were 19% slower on AI-allowed tasks than on AI-not-allowed ones, despite believing they were 24% faster [primary]. The gap between perceived and measured productivity is the study's most-cited result.

What this study did not measure

  • Effect on developers unfamiliar with the codebase. The whole population were senior maintainers in their own repositories.
  • Effect on greenfield work. All tasks were in mature repos.
  • Code-quality outcomes beyond task completion.
  • Effect of newer model generations released after data collection.

Legitimate criticisms

  • Small sample (n = 16 developers). Statistical power is limited; the authors are upfront about this.
  • Population is the worst case for AI assistants. Senior developers in code they wrote themselves benefit least from suggestions; the finding may not generalize to juniors or to unfamiliar code.
  • Tooling era. AI tooling moves quickly; a study of "early-2025" models will not predict the behavior of late-2025+ models.

How to interpret it for your size tier

  • Small (< 50 people): if your seniors maintain a codebase nobody else fully understands, expect AI suggestions to be a tax on them and a boost for juniors. Pair them deliberately.
  • Mid (50–500): the "perceived faster, actually slower" finding is the one to internalize. If your only metric is engineer self-report, you will mismeasure the program. Use DORA / SPACE metrics alongside.
  • Enterprise (500+): the senior-maintainer-of-mature-system population is not rare in your org. Carve out specific anti-patterns in the AUP — e.g., do not require AI tool usage on owners of crown-jewel systems.

3. DORA — 2024 Accelerate State of DevOps report

Citation. Google Cloud / DORA, "2024 Accelerate State of DevOps Report", October 20243. DORA is published by Google Cloud; the survey is run independently of any specific tool vendor.

Population and method

  • 39,000+ professional engineering responses worldwide, across industries and team sizes.
  • Survey design, not an experiment. Captures self-reported AI use and self-reported delivery metrics.
  • Headline AI-related analysis: for every 25% increase in AI tool use within a team, the survey found an associated drop in delivery throughput and stability.

Headline finding

  • Software delivery throughput dropped 1.5% per +25% AI tool use.
  • Software delivery stability dropped 7.2% per +25% AI tool use.
  • Small gains were observed in individual flow, code review speed, and documentation quality [primary].

What this study did not measure

  • Causation. This is correlation in survey data. AI-heavy teams may differ from low-AI teams on confounders (tenure, codebase age, sector).
  • Specific tools. The survey does not disaggregate by vendor.
  • Code-quality root cause for the stability drop.

Legitimate criticisms

  • Self-report end-to-end. Both the independent variable (AI use) and the dependent variables (throughput, stability) are self-reported. Anchoring effects are plausible.
  • One-year cross-section. No longitudinal measurement of the same teams.
  • Industry mix. Heavy responses from large enterprises may not reflect startup dynamics.

How to interpret it for your size tier

  • Small (< 50 people): team-level metrics in DORA are generally more meaningful at scale. Take the direction of effect seriously — not the precise number — and pair the AI rollout with explicit review budget.
  • Mid (50–500): this is the size where the throughput finding bites hardest. Plan for a pilot retrospective that compares throughput and change-failure-rate before and after.
  • Enterprise (500+): treat the stability number as a standing hypothesis about your environment. Instrument change-failure rate and MTTR with AI-attribution before the rollout, not after.

4. Stanford security study (Perry et al., 2023)

Citation. Perry, Srivastava, Kumar, Boneh, "Do Users Write More Insecure Code with AI Assistants?", arXiv:2211.03622, ACM CCS 2023 4. Independent academic research.

Population and method

  • 47 participants (a mix of professional developers and CS students).
  • Within-subject experiment: each participant attempted security- sensitive programming tasks both with and without an AI assistant.
  • Tasks included encryption, signing, and SQL-injection-prone database query construction.
  • Outcomes measured: correctness of the security-relevant code, and participants' self-reported confidence in that code.

Headline finding

Participants using the AI assistant wrote less secure code — more solutions had clear vulnerabilities — and were more confident their code was correct than the control group [primary].

What this study did not measure

  • Effect of in-prompt security guidance. Participants used the tool out of the box. Prompts that explicitly request safe defaults were not the focus.
  • Effect of post-generation review. No reviewer was in the loop; the study captures first-draft output.
  • Code quality dimensions other than security.

Legitimate criticisms

  • Sample size. n = 47 is small for an effect this consequential.
  • Tooling era. The study used 2022-era models; capability has changed. Replication on later models is needed and not all of it has been published.
  • Task selection. Cryptographic primitives are unusually error- prone; the effect may be smaller for routine CRUD.

How to interpret it for your size tier

  • Small (< 50 people): assume every AI-generated security- relevant change needs a security-aware human reviewer. You probably do not have AppSec; pair an experienced engineer.
  • Mid (50–500): invest in S.E.C.U.R.E. checklists in review. The confidence-vs-correctness gap is the key metric to design against.
  • Enterprise (500+): mandate AppSec pre-merge review on High and Critical tier systems. Do not let AI confidence substitute for a control.

Additional credible studies

A short note on other studies often cited in adoption discussions. These are included here only when a verifiable URL is available.

NYU "Asleep at the Keyboard"

Pearce, Ahmad, Tan, Dolan-Gavitt, Karri, "Asleep at the Keyboard? Assessing the Security of GitHub Copilot's Code Contributions", IEEE S&P 20225. Found that approximately 40% of Copilot suggestions in CWE-Top-25 scenarios contained vulnerabilities.

  • Population: synthetic security-prompt scenarios, not real users.
  • Caveat: an upper-bound estimate of the raw output risk, before any human review or in-prompt mitigation. Cite alongside Perry 2023, not in place of it.

GitClear code-churn analysis

GitClear, "Coding on Copilot: 2023 Data Suggests Downward Pressure on Code Quality", January 20246. Measured a near-doubling of code reverted within two weeks of authoring between 2020 and 2023, coinciding with AI-assistant adoption.

  • Population: GitClear's customer base — biased toward teams using GitClear's tooling.
  • Caveat: correlational and vendor-published. Use as a signal, not proof.

How to read these studies for your context

A reading order by company size

Small (< 50 people). Read Peng 2023 for the upside and Perry 2023 for the security caveat. Skip DORA — your team is too small for its team-level metrics to apply cleanly. Decide what you will measure before the rollout: lead-time for change on a single high-value workflow, and one quality signal (escaped defects or incidents).

Mid (50–500). Read all four. DORA is the most applicable to your setting; METR is the most useful corrective on optimistic self-reports. Plan a pilot that captures DORA-style team metrics from day one.

Enterprise (500+). Read all four, plus your own internal data. Public studies are noise compared to your own A/B-tested rollouts once you reach scale. Use the studies to set a prior; let your own measurement set the posterior.

The general lesson across the four: AI assistants accelerate constrained, well-scoped, individual tasks more than they accelerate ambiguous team work. Programs that measure only the first and ship on that basis tend to discover the second the hard way.

  1. Peng, Kalliamvakou, Cihon, Demirer, "The Impact of AI on Developer Productivity: Evidence from GitHub Copilot", arXiv:2302.06590, Feb 2023. Vendor-funded (GitHub) but methodology and dataset are disclosed. ↩

  2. METR, "Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity", Jul 2025. ↩

  3. Google Cloud / DORA, "2024 Accelerate State of DevOps Report", Oct 2024. DORA is published by Google Cloud; the survey is independent. ↩

  4. Perry, Srivastava, Kumar, Boneh, "Do Users Write More Insecure Code with AI Assistants?", arXiv:2211.03622, ACM CCS 2023. ↩

  5. Pearce, Ahmad, Tan, Dolan-Gavitt, Karri, "Asleep at the Keyboard? Assessing the Security of GitHub Copilot's Code Contributions", IEEE S&P 2022. ↩

  6. GitClear, "Coding on Copilot: 2023 Data Suggests Downward Pressure on Code Quality", Jan 2024. ↩