“AI writes code faster” is not in dispute. Whether it writes code worse is a separate, empirical question, and one we would rather answer from measurement than from either side’s talking points. Three independent studies, using three different methods, converge on the same answer.
The baseline: a documented defect rate
We covered the founding study in detail in A Clean Benchmark Is Not a Clean Codebase: Pearce et al. (IEEE S&P 2022) found roughly 40% of 1,689 Copilot-generated programs vulnerable against MITRE’s Top 25 weakness classes. That study is old enough now that its number should not be read as current — models have changed since 2022. What it established, and what later work builds on, is that the question is real and answerable with standard methods: targeted scenarios, a large sample, static analysis against a named weakness list.
The mechanism: not random bugs, a systematic absence
A number alone doesn’t tell you what to do about it. A controlled comparison of human-written and LLM-generated code across data structures, algorithms, cryptographic routines, and LeetCode-style problems — using unit testing, fuzzing, and static analysis (arXiv:2409.19182) — went further, and found LLM-generated code is less secure primarily because it lacks defensive programming constructs: the bounds checks and input validation a careful engineer adds by habit, not the feature that was actually asked for. That absence is what invites buffer and integer overflows. Under fuzzing, LLM-generated code was more prone to hangs and crashes than the human-written baseline. Correctness failures could be subtle rather than obvious: in one case the model produced an incorrect implementation of SHA-1 that nevertheless compiled cleanly — wrong, and silent about it.
The finding that should worry anyone treating “ask the model to fix it” as a safety net: the same study built a feedback loop, asking the model to re-generate code and eliminate issues it had just been shown. It did not reliably work. In some cases the fix succeeded. In others, the “fixed” code contained new issues, and in some cases the act of prompting introduced problems into files that had been clean before the prompt touched them. Asking the author to grade its own homework is not review.
The production evidence: the cost doesn’t fade with the novelty
The most recent evidence comes from outside the lab entirely. A study tracked 807 open-source GitHub repositories that adopted the Cursor AI coding assistant between January 2024 and March 2025, against a control group of 1,380 similar repositories that hadn’t, using SonarQube to track static-analysis warnings, duplication, and complexity through August 2025. The productivity story played out as expected: a short-lived spike in commits and lines added in the first one to two months, back to baseline by month three. The quality cost did not fade with it. Static-analysis warnings rose roughly 30% after adoption and stayed elevated. Code complexity rose more than 40% — more than codebase growth alone would explain. The velocity gain was temporary. The debt wasn’t.
One honest boundary
This is not a claim that AI-generated code is always worse, as a blanket law. Task, model, and workflow all matter, and a team that treats AI output as a first draft requiring the same review discipline as a junior engineer’s pull request sees different outcomes than a team that ships it unreviewed — the studies above don’t test every model or every workflow, and none of them isolate review discipline as a variable. What they do establish, consistently, across three independent methods (targeted CWE testing, controlled comparison, longitudinal production data), is that the effect is real and systematic rather than occasional, and that asking the model to check its own work is not a substitute for someone else doing it.
Why this is a compliance question, not just an engineering one
Under the Cyber Resilience Act, a manufacturer must ensure a product with digital elements is “designed, developed and produced in such a way that they ensure an appropriate level of cybersecurity based on the risks” and made available “without known exploitable vulnerabilities” (Regulation (EU) 2024/2847, Annex I Part I, points (1) and (2)(a)). Neither clause mentions how the code was written. Both become harder to satisfy, not easier, the more of a codebase’s surface area was generated by a process with a measured, systematic gap in exactly the constructs — bounds checks, input validation — that keep a known weakness class from becoming exploitable. Faster generation without a matching increase in review is not a shortcut around that obligation. It is a growing distance from it.
Why we care
At CounterProof we do adversarial review of machine-written code, and these three studies describe the exact gap our practice exists to close: treating AI output — generated or reviewing — as a claim to be checked, not a result to be trusted. See the companion piece, A Clean Benchmark Is Not a Clean Codebase, for what happens when the checking tool is also AI-generated and its own reliability goes unexamined. The longer argument, including the conditional form of every claim above, sits in our working paper on collation and provenance for machine-generated text, readable in full here. It is a draft preprint, not peer-reviewed, and says so.
Sources cited above were checked against their published abstracts. The Pearce et al. figure is from IEEE S&P 2022 and describes Copilot as it existed at the time of that study, not current models. The Cursor-adoption figures are from a longitudinal study of 807 repositories against a 1,380-repository control group, tracked January 2024 through August 2025. CRA provisions cited are to Regulation (EU) 2024/2847 and were checked against the Regulation’s text. If we have something wrong here, we will correct it in writing on this page.