# A Clean Benchmark Is Not a Clean Codebase

> AI vulnerability scanners post strong numbers on published benchmarks. Two independent studies (one built specifically to test at repository scale, one that watched professional developers use a tool on their own code) found the strong number does not travel. Here is where it breaks.

Source: https://counterproof.io/notes/a-clean-benchmark-is-not-a-clean-codebase/ · Published 25 August 2026 · CounterProof is a practice of Clavestra Capital Limited (Malta, C 113987).


A vulnerability scanner's marketing page carries a number: some detection rate against some public
dataset. We [wrote before](https://counterproof.io/notes/the-illusion-of-the-score/) about why a benchmark score is
**testimony**, not measurement, a fluent claim that has to be cross-examined rather than taken at
face value. This note is the narrow, concrete case: what happens when you cross-examine a
security-specific benchmark, and what the two studies that actually checked found.

## The problem the tools exist to catch

The reason AI-assisted vulnerability detection matters is that AI-assisted code *generation* has a
documented defect rate. The founding study here: Pearce et al., "Asleep at the Keyboard? Assessing
the Security of GitHub Copilot's Code Contributions" (IEEE S&P 2022), built 89 scenarios targeting
weaknesses on MITRE's Top 25 Common Weakness Enumeration list, had Copilot complete them, and
collected 1,689 resulting programs. **Roughly 40% were vulnerable** to the weakness class the
scenario was designed to probe. Models have moved on since 2022; the study is old enough that its
number should not be read as current. What has not moved on is the reason the finding mattered: it
established that the output half of this loop needs checking, which is the whole justification for
the detection tools this note is actually about.

## Where the benchmark stops describing the repository

A.S.E, a repository-level security benchmark (arXiv:2508.18106), was built for a specific reason:
existing benchmarks, in the authors' words, "lack relevance to real-world AI-assisted programming
scenarios", most test isolated snippets pulled from a curated dataset, not code sitting inside a
large, interconnected codebase with its actual dependencies and call graph. Moving from
snippet-level to repository-level tasks is where the apparent competence breaks down: the paper
reports that current models "demonstrate inadequate performance in generating secure code" once the
task requires reasoning about a real codebase's structure. The specific, counterintuitive finding
worth naming: giving a model a larger reasoning budget does not reliably close the gap. More
compute is not a substitute for the thing the benchmark wasn't testing.

## What happens when you watch a human use the tool

The second study did something rarer than building a harder benchmark: it put a real tool in front
of real developers and watched. Microsoft researchers ran DeepVulGuard, an IDE-integrated
vulnerability detection and repair tool, with **17 professional developers scanning their own real
projects**, 24 projects, 6,900 files, over 1.7 million lines of code, in a naturalistic setting
with observation and interviews rather than a synthetic test harness (arXiv:2412.14306). The tool
had already shown, in the paper's own words, "promising performance on benchmarks of historic
vulnerability data." In production use it generated 170 alerts and 50 fix suggestions, and the
paper's verdict on the result is unambiguous: the tools "are not yet practical for real-world use
due to a high rate of false positives and non-applicable fixes." Developers cited incomplete
context representation and a lack of codebase-specific customisation as the main obstacles. A strong
benchmark number did not predict strong field performance, the same shape of gap A.S.E found by
changing the test, this time measured by changing the tester.

## One honest boundary

This is two studies, not a survey of the field, and neither claims detection tools are worthless, 
DeepVulGuard's own benchmark performance was genuinely strong; it is the *transfer* to unscoped,
real-world code that failed, not the underlying method. The claim here is narrower and, we think,
harder to dispute: a benchmark score measures the benchmark. Whether it measures your codebase is a
separate, empirical question that the vendor's number cannot answer for you, and in the two cases
where someone checked, the answer diverged from the benchmark in the same direction, worse in the
field than on the leaderboard.

## What to ask instead of the headline number

- **Was it tested at repository scale, on code structured like yours**, or on isolated functions
  pulled from a curated set? A.S.E's finding says this is not a formality; it is where the result
  changes.
- **What is the false-positive rate in naturalistic use**, not just the detection rate against a
  labelled dataset? A tool that flags everything scores well on recall and produces nothing but
  noise for the developer who has to triage it, the same inflation dynamic we described for
  [inflated finding counts in submitted reports](https://counterproof.io/notes/a-finding-is-not-a-verdict/), here arriving from the
  detection side instead of the reporting side.
- **Would the vendor show you the field number, not just the leaderboard number?** DeepVulGuard's
  authors published both, including the one that made the tool look worse. That is the correct
  standard to hold every other vendor to.

## Why we care

At CounterProof we do adversarial review of machine-written code, and "does this instrument's
benchmark score predict its performance on our actual target" is a question we ask about every tool
in our own pipeline, not only about the ones we are reviewing. The general failure mode: an
evaluation instrument that is not independent of, or not representative of, the thing it is
measuring, is the same one we set out in
[*The Illusion of the Score*](https://counterproof.io/notes/the-illusion-of-the-score/). This note is that argument applied to one concrete, checkable case. The same caution is one of the
claims we refuse to make on a client's behalf: we do not report "we found nothing" as "there is
nothing" ([What we will not claim](https://counterproof.io/refusals/)). The longer version, including the conditional form of every claim
above, sits in our working paper on [collation](https://counterproof.io/glossary/#collation) and provenance for machine-generated text,
[readable in full here](https://research.counterproof.io/). It is a draft preprint, not
peer-reviewed, and says so.

---

*Sources cited above were checked against their published abstracts. The Pearce et al. figure
(40% of 1,689 programs) is from IEEE S&P 2022 and describes Copilot as it existed at the time of
that study, not current models. Where a claim is our inference: that the field-benchmark gap in
these two cases plausibly generalises, rather than something the papers themselves state, we have
said so. If we have something wrong here, we will correct it in writing on this page.*