In roughly 108 hours of machine time in August 2026, one volunteer campaign pushed an open-weight model through 501 Bitcoin open-source projects and logged 7,958 potential security findings, 1,280 of them tagged high or critical. We wrote about that campaign, and the regulation it collides with, in The Machines Are Reading the Code. This note is about the number itself.
Because a security vendor has published a benchmark pitting three AI code-review systems against a private test application: one found 68 of 89 planted vulnerabilities, another 60, a third 58. The vendors now compete on that ratio — more findings, at lower cost. Finding has become a commodity in a price war.
So it is worth saying plainly what a finding actually is, because the count is measuring the wrong thing.
A finding is a witness, not a verdict
Textual scholars settled this five centuries ago, and their word for it is exact.
When an ancient text survives in many hand-copied manuscripts, no single copy is the text. Each is a witness: a derivative document carrying the original plus its scribe’s slips, habits, and inherited drift. Three manuscripts agreeing may mean the reading is genuine — or that all three descend from the same flawed exemplar. The discipline that sorts this out produces a critical edition: the text as best established, with an apparatus beneath it recording every witness weighed, every variant, every editorial judgement and its grounds.
An AI-generated vulnerability finding is a witness in precisely this sense. It is not the defect. It is one model’s report of a defect — the code’s real properties mixed with the model’s habits, its training lineage, and the prompt that produced it. Two scanners agreeing may mean the bug is real, or that both share the same blind spots and the same templates. Since they are trained on largely the same public corpus, they often do. Agreement between correlated witnesses is evidence about the witnesses, not about the code.
And the scholars’ hardest-won rule transfers intact. The fluent reading is the suspect one — scribes smooth a hard passage into an easy cliché, and the smooth version is more often the copyist’s improvement than the author’s words. A vulnerability report has a fixed genre: title, weakness class, severity, a citation, an “an attacker could…” paragraph. A language model produces that form convincingly whether or not the defect exists. The polish carries no information about the truth. In this genre, fluency is not a credential. It is the thing to check.
That is why the count is the wrong unit. It counts witnesses and reports them as verdicts.
Severity is where the reports overstate
The inflation is not uniform. It concentrates at one point, for a structural reason worth naming.
“This function compares a millisecond timestamp against a second-precision column” is a claim that cites something — a file, a line, a commit. You can open it and settle it in a minute. “Critical” cites nothing. There is no line of code that reads Critical. Severity is a judgement about reachability, exploitability, and blast radius, and because it has no anchor in the source, it is exactly where a fluent generator drifts upward unchecked. The dramatic reading is the easier one; every incentive in the reporting chain rewards a bigger number; and nothing in the artifact resists.
So when a campaign reports 1,280 findings “high or critical,” the honest reading is: 1,280 findings a generator labelled high or critical. The label is the least checkable field in the document, and it is doing the most work in the headline.
Are they exploitable? Nobody has said yet
Between a finding and a defect sits a step the count skips: adjudication — deciding which findings are real, which are reachable, which matter. That step is still human, and it does not scale with generation.
The clearest public measurement of what happens when it is skipped comes from curl. Its seven-person volunteer security team watched the share of confirmed vulnerability reports fall from above 15% to below 5% as AI-assisted submissions arrived — each bogus report still costing hours to disprove. One came complete with GDB sessions and register dumps for an HTTP/3 exploit in a function that does not exist in curl. Its maintainer described the triage load as “tantamount to a DDoS,” and in the end the project closed its bug-bounty programme, saying it had not seen a single valid security report produced with AI help.
Read that number the right way round. It does not prove AI findings are worthless — it proves that a finding’s fluency and its validity are independent, and that when you remove the adjudication step, the signal-to-noise ratio collapses. The reports got more numerous and more confident; the fraction that survived checking went down.
The flood is a security problem, not a nuisance
It is tempting to file “too many findings” under annoyance. It is not. When generation outruns adjudication, the real finding does not disappear — it is buried in the queue behind two hundred plausible ones, and a queue no one can clear is a queue that hides things. curl’s word for it — DDoS — is the correct register. A team drowning in confident reports is not more secure than a team with none; it is a team whose attention has been exhausted on witnesses that were never cross-examined.
Keep the two alarms apart
Two claims get conflated here, and separating them is the whole discipline.
In aggregate, the danger is real. There genuinely are enormous numbers of unfixed defects in old code, and a cheap, tireless finder will surface some of them. Dismissing the whole category as hype is wrong.
Per finding, the stated criticality is systematically overstated — upward, because the one field nobody can check is the one everybody inflates. Treating each alert as a verdict is also wrong, in the opposite direction.
The error is using either alarm to answer the other’s question. “Lots of real bugs exist somewhere in this ecosystem” does not make this finding on your line exploitable. And “most of these particular alerts are noise” does not mean your codebase is clean.
What sees through it
The medieval editor could not decontaminate the whole tradition. What they could do was attach conditions to every reading and make the conditions explicit. The same move works here. When you are handed a finding, ask:
- What evidence rung does it actually reach? Traced in source, compiled, tested, or reproduced running — these are not the same claim. A finding should never wear a certainty higher than the work behind it.
- Is it reachable in your build, or reachable in principle? Severity without a reachability argument is a mood, not a measurement.
- Was it adjudicated, or averaged? Three tools agreeing is not three confirmations if the tools share a blind spot. A minority finding is worth a cheap oracle check — run the test, read the spec, execute the path — not a vote.
- Is it already fixed, or already known? A finding the maintainers closed last month is a witness to history, not to your risk.
None of that is exotic. It is what a critical edition does to a pile of manuscripts: collate the witnesses, adjudicate against evidence stronger than another opinion, and keep the apparatus — the judgements, the uncertainty, the readings considered and rejected.
Not every dead finding died the same way
Most findings do not survive. That is normal, and it is the point — but how a finding died is information, and a review that tips them all into one bin marked “dismissed” has thrown that information away.
Textual scholars separate the cases, and the distinction is old enough to have a name. A manuscript shown to be copied from another surviving manuscript is struck from the reckoning entirely — eliminatio codicum descriptorum — not because it is wrong, but because it adds no independent testimony. Its agreement was never evidence in the first place.
Three deaths, and they do not mean the same thing:
- Redundant. Several tools produced the same finding. Where those tools are not independent — and sharing training data and templates, they often are not — their agreement is one reading in three copies rather than three confirmations, and counting it three times is how false confidence gets manufactured. Strike the duplicates, but record the independence assumption rather than assuming it away: agreement between genuinely independent reviewers is not nothing.
- False. The finding says the code does something the code does not do. Settled by opening the file. This is the only category most people have in mind when they say “false positive.”
- True, but about a text that no longer exists. The finding is accurate — about a revision that has since been fixed. The witness is honest; the code moved underneath it. A defect closed at a pinned upstream revision is not a false positive for that revision and should not be recorded as one. That holds only if the finding says which revision it read: a report that cannot name its revision is not an honest witness to a superseded text, it is a stale claim about the current one.
Only the second is an error in the reading itself; the unpinned case of the third is an error in the pipeline that reported it. A record that says which death each finding died can be audited by someone who was not there. A ratio — “we triaged two hundred findings down to six” — cannot, and it is not a quality signal either: a lazy pipeline reaches that ratio by generating rubbish and discarding it. The elimination is not the product. The evidence behind the survivors is.
What an edition is for
The reason to build an edition rather than a list is that somebody has to reach a verdict, and it will not be the reviewer.
A maintainer decides whether to patch. A manufacturer decides what goes into the technical file. An enterprise buyer’s security team decides whether to sign. Each of them is reaching a verdict, and each will be asked later — by someone less friendly, with more time — to show how they reached it. What they need from a reviewer is not one more opinion to weigh. It is the evidence arranged so that their own decision survives the asking.
In Europe this is becoming a legal requirement rather than good practice. The Cyber Resilience Act requires a manufacturer to “apply effective and regular tests and reviews of the security” of their product (Annex I Part II(3)), and the technical documentation must contain “reports of the tests carried out to verify conformity” (Annex VII(6)) — kept for at least ten years after the product is placed on the market, or the support period if longer (Article 13(13)). Article 14 notification of actively exploited vulnerabilities and severe incidents starts 11 September 2026; the Regulation applies in full from 11 December 2027.
Read what that actually asks for. Not a clean scan, and not a low number. A report of tests carried out, retained for a decade, legible to someone who was not in the room. A list of findings wearing confident severities, with no apparatus underneath and no record of what was rejected or why, does not survive that reading. An edition does — because every claim states the evidence behind it, every eliminated reading is recorded with its reason, and every correction is in writing.
The obligations sit on the manufacturer, not on an external reviewer. An external review of the kind described here supports the technical documentation you assemble; it is not the complete file, and it is not a substitute for a notified-body conformity assessment where the Regulation requires one. But it is the part of that file that is hardest to produce about your own work — because the one property it has to carry is that somebody other than you wrote it.
Why we care
At CounterProof we do adversarial review of machine-written code, and these questions are not theoretical to us — they are our operating rules. Findings are adjudicated rather than averaged; every claim names the evidence rung it reached and no higher; a finding already fixed upstream is withdrawn rather than counted; and when one of our own claims does not survive, we retract it in writing, publicly. We are not selling a finding count, and we would tell you not to buy on one. The scanners are scribes, and the scriptorium now runs day and night for pennies. The scarce thing was never the copy. It is the edition.
The longer argument — including the conditional form of every claim above, and the limits of our own method — is written up in our working paper on collation and provenance for machine-generated text, readable in full here. It is a draft preprint, not peer-reviewed, and says so.
Figures above are sourced: the 501-project campaign is documented in our earlier note; the three-system benchmark ratios are from the vendor’s published post; curl’s confirmed-report rate, the phantom-function report, and the closure of its bounty programme are reported by The New Stack, BleepingComputer, and The Register. Citations to the Cyber Resilience Act are to Regulation (EU) 2024/2847 and are the same ones we carry on our homepage. Where a claim is our inference rather than a cited finding — that severity inflates upward, that fluency and validity are independent — we have said so. If we have something wrong here, we will correct it in writing on this page.