A vulnerability scanner’s marketing page carries a number: some detection rate against some public dataset. We wrote before about why a benchmark score is testimony, not measurement — a fluent claim that has to be cross-examined rather than taken at face value. This note is the narrow, concrete case: what happens when you cross-examine a security-specific benchmark, and what the two studies that actually checked found.
The problem the tools exist to catch
The reason AI-assisted vulnerability detection matters is that AI-assisted code generation has a documented defect rate. The founding study here — Pearce et al., “Asleep at the Keyboard? Assessing the Security of GitHub Copilot’s Code Contributions” (IEEE S&P 2022) — built 89 scenarios targeting weaknesses on MITRE’s Top 25 Common Weakness Enumeration list, had Copilot complete them, and collected 1,689 resulting programs. Roughly 40% were vulnerable to the weakness class the scenario was designed to probe. Models have moved on since 2022; the study is old enough that its number should not be read as current. What has not moved on is the reason the finding mattered: it established that the output half of this loop needs checking, which is the whole justification for the detection tools this note is actually about.
Where the benchmark stops describing the repository
A.S.E, a repository-level security benchmark (arXiv:2508.18106), was built for a specific reason: existing benchmarks, in the authors’ words, “lack relevance to real-world AI-assisted programming scenarios” — most test isolated snippets pulled from a curated dataset, not code sitting inside a large, interconnected codebase with its actual dependencies and call graph. Moving from snippet-level to repository-level tasks is where the apparent competence breaks down: the paper reports that current models “demonstrate inadequate performance in generating secure code” once the task requires reasoning about a real codebase’s structure. The specific, counterintuitive finding worth naming: giving a model a larger reasoning budget does not reliably close the gap. More compute is not a substitute for the thing the benchmark wasn’t testing.
What happens when you watch a human use the tool
The second study did something rarer than building a harder benchmark: it put a real tool in front of real developers and watched. Microsoft researchers ran DeepVulGuard, an IDE-integrated vulnerability detection and repair tool, with 17 professional developers scanning their own real projects — 24 projects, 6,900 files, over 1.7 million lines of code, in a naturalistic setting with observation and interviews rather than a synthetic test harness (arXiv:2412.14306). The tool had already shown, in the paper’s own words, “promising performance on benchmarks of historic vulnerability data.” In production use it generated 170 alerts and 50 fix suggestions, and the paper’s verdict on the result is unambiguous: the tools “are not yet practical for real-world use due to a high rate of false positives and non-applicable fixes.” Developers cited incomplete context representation and a lack of codebase-specific customisation as the main obstacles. A strong benchmark number did not predict strong field performance — the same shape of gap A.S.E found by changing the test, this time measured by changing the tester.
One honest boundary
This is two studies, not a survey of the field, and neither claims detection tools are worthless — DeepVulGuard’s own benchmark performance was genuinely strong; it is the transfer to unscoped, real-world code that failed, not the underlying method. The claim here is narrower and, we think, harder to dispute: a benchmark score measures the benchmark. Whether it measures your codebase is a separate, empirical question that the vendor’s number cannot answer for you, and in the two cases where someone checked, the answer diverged from the benchmark in the same direction — worse in the field than on the leaderboard.
What to ask instead of the headline number
- Was it tested at repository scale, on code structured like yours — or on isolated functions pulled from a curated set? A.S.E’s finding says this is not a formality; it is where the result changes.
- What is the false-positive rate in naturalistic use, not just the detection rate against a labelled dataset? A tool that flags everything scores well on recall and produces nothing but noise for the developer who has to triage it — the same inflation dynamic we described for manually-generated finding volume, here arriving from the detection side instead of the reporting side.
- Would the vendor show you the field number, not just the leaderboard number? DeepVulGuard’s authors published both, including the one that made the tool look worse. That is the correct standard to hold every other vendor to.
Why we care
At CounterProof we do adversarial review of machine-written code, and “does this instrument’s benchmark score predict its performance on our actual target” is a question we ask about every tool in our own pipeline, not only about the ones we are reviewing. The general failure mode — an evaluation instrument that is not independent of, or not representative of, the thing it is measuring — is the same one we set out in The Illusion of the Score. This note is that argument applied to one concrete, checkable case. The longer version, including the conditional form of every claim above, sits in our working paper on collation and provenance for machine-generated text, readable in full here. It is a draft preprint, not peer-reviewed, and says so.
Sources cited above were checked against their published abstracts. The Pearce et al. figure (40% of 1,689 programs) is from IEEE S&P 2022 and describes Copilot as it existed at the time of that study, not current models. Where a claim is our inference — that the field-benchmark gap in these two cases plausibly generalises — rather than something the papers themselves state, we have said so. If we have something wrong here, we will correct it in writing on this page.