In A finding is not a verdict we made one argument: an AI-generated security finding is a witness, not a fact, and counting witnesses is not the same as reaching a verdict. That note ended where the work begins. If a finding is testimony, what turns it into something a maintainer, a buyer, or a regulator can act on?
The answer is shorter than the question, and it has two halves.
Some claims can be settled. Most cannot.
Take a finding apart and it contains claims of two different kinds.
Some of them have an oracle, something that answers the claim without anyone’s opinion. Does this function accept a note the federation never signed? Write the test and run it. Does this build compile with the flag off? Compile it. Does the exploit path reach the sink from a default configuration? Reproduce it. Does the specification permit this enum value? Read the specification. When an oracle exists, it settles the claim, and every opinion about the claim (the reviewer’s, the tool’s, ours) is subordinate to it. Not “weighed against it.” Subordinate. A test that ran outranks a model that reasoned, and it outranks a person who reasoned too.
The rest have no oracle. How severe is this? Is the trust assumption behind it acceptable for this product? Would a real attacker bother? Nothing you can run answers those. They are judgement, and no amount of fluency from a model (or from us) converts judgement into fact. What can be done with judgement is narrower and more honest: weigh the testimony, say whose judgement it is, and put a name on it.
That is the whole method, and it is why every finding in a CounterProof assessment names the evidence rung it actually reached, the ladder the earlier note sets out, from source trace to live reproduction. A finding that reached none of those rungs is graded plausible-only, which is not an embarrassment to be hidden at the bottom of the report. It is the rung that says this one rests on judgement, and here is the judgement’s owner. A report that never prints that rung is not a report with no judgement in it. It is a report that has disguised its judgement as evidence.
What “outranks” costs
Two habits get in the way, and both feel like rigour.
The first is averaging. Three tools flag the same line; two reviewers agree the severity is high; the panel converges. Convergence is comfortable, and it is the wrong instrument for a claim an oracle could settle. If the question is does this path execute, the cheapest honest move is to execute it, and a panel convened on that question is, in our view, the wrong instrument; it produces more testimony about a matter that needed none. We have done this ourselves. A finding in our own register was carried to a four-reviewer panel this month when a ten-line script settled it in two minutes; that too is our own account of an internal record. The panel was not wrong. It was the wrong instrument, and the script is now committed beside the finding so the next reader can run it instead of asking anyone.
The second habit is subtler: trusting a test because it exists. A test is an oracle only if it can fail. A test that has only ever passed has not been shown to discriminate between the code being right and the test being blind: and when the same process wrote the code, the test, and the comment explaining both, all three agreeing is one witness in three copies. The check is cheap and it is the one most often skipped: plant the defect, run the test, watch it fail; remove the defect, watch it pass. A test that cannot be made to fail is evidence of exactly one thing; that the test is vacuous. It is a green light wired to the wall.
The same standard, applied to our own tools
Everything above is easy to say about someone else’s code. The harder question is what happens when the thing under review is a checking tool we built, because a broken verification tool does not report an error. It reports pass, in exactly the voice a working one uses.
We have two documented instances of a verification tool of ours failing, and we would rather you read them here than have to ask.
The first is in our working paper, section 8. We built a tool whose only job was to refuse “not found” claims that lacked a known-positive control, a guard against exactly the false-absence failure we warn clients about. A reviewer we had commissioned, who had not built the tool, constructed six routes to a false “absent,” including one where a quietly-succeeding command reported absence precisely when the target was present. The tool’s own five-case self-test had passed over all six. One of those five controls had been tuned, by its author, to a string that could not trigger the defect it was supposed to catch. The tool is quarantined, its defects encoded as permanently failing tests.
The second is from this month, and it is worse in one specific way. We wrote a check for the register we keep of our own findings (internal tooling, never a client-facing artifact) to enforce the rule in this note: route a decidable question to an oracle, not a panel. We shipped it on 1 September with its self-test defined and never called, and in the commit message quoted an “8 of 8 passing” that came from a prototype rather than the shipped file. A reviewer we commissioned, working from the source alone, named thirteen ways past it. We did what the rule says: ran every one rather than debate them. Every one reproduced. Two of the holes were in the design rather than the wiring: a permitted value accepted unconditionally, which in our reading rewarded the weaker label, and a gate that reported pass while it had evaluated nothing at all. The check was fixed the next day and proven fixed the way this note prescribes: a planted violation now fails under every mode the check runs in, and the check reports itself dormant (and fails) until it has evaluated one real subject. The rule that says the oracle outranks the reviewer was repaired by running the reviewer’s claims as an oracle against the tool that enforces it.
Two things about those reviewers, said precisely because the words matter on this site. They were review seats we commission: models drawn from a different family than the one that built the tool, briefed by us, paid for by us. They are independent of the author. They are not a third party to the firm, and we do not claim they are; what a third party looks like is the subject of section 04 on our homepage, and it is us, for your code, never for our own. And they are not the reviewing bench. On a client engagement the bench is the two named people on our homepage; the seats are instruments that bench runs, and here they are instruments we turned on our own instruments. The paper’s section 8 calls that reviewer “independent” without this qualification. We narrow the word here on purpose, and the paper’s next revision will carry the same narrowing.
Why publish any of this? Because a practice that cannot show you where it was wrong has no standing where it says it is right. A withdrawal log is not an admission. It is the mechanism, and it applies to our instruments the same way it applies to our findings.
What to ask of any report
None of this requires our involvement. When a finding lands on your desk (from a scanner, a consultancy, a model, or us) four questions separate what was settled from what was asserted:
- Which rung did it reach? Traced, compiled, tested, reproduced, or plausible-only. If the report does not say, assume the lowest.
- Could an oracle have settled it, and was one run? A claim that a test could decide, delivered as an opinion, is a reviewer who did not do the cheap thing.
- Whose judgement is it? For the claims no oracle can settle, a name (a person’s, on the signed document) or it is the firm’s mood.
- Can the test fail? For any test offered as evidence, ask whether anyone planted the defect and watched it go red. If not, the green means nothing yet.
Those four are not a standard we hold others to and exempt ourselves from. They are the ones that caught us.
Limits
Two instances are two instances, not a rate; we make no claim about how often verification tools fail, ours or anyone’s. The reviewers who found the failures were commissioned by us, and a commissioned reviewer’s catches track the brief they were given, a limit we discuss at length in the paper. And the rule that an oracle outranks opinion says nothing about the large remainder of questions that have no oracle; there, all we can offer is testimony that knows it is testimony, weighed by a named person, on the record.
The conditional form of every claim above is in the working paper: a draft preprint, not peer-reviewed, and it says so.
The section-8 instance follows the paper’s own account in every particular but one: the paper calls the reviewer “independent” and we say here what that meant, a seat we commissioned, independent of the tool’s author, not a third party. The paper does not name the reviewer and neither do we. The second instance is from our internal register, which is not public; the counts are as recorded there on 2 September 2026, and a reader has only our word for them. Three claims above are stated as rules because we operate by them, but they are our inferences, not recorded facts: that a test which has only ever passed has not been shown to discriminate; that a panel is the wrong instrument for a claim an oracle could settle; and that accepting a permitted value unconditionally “rewarded the weaker label”; the last is our reading of a design’s effect, not a measurement. If we have something wrong here, we will correct it in writing on this page.