Illustrative specimen, not from a client engagement; identifiers are placeholders.
CounterProof reviewed a Bitcoin infrastructure codebase at a pinned revision, using deterministic reproductions against unmodified code. Candidate claims were separated into reproduced behavior, source-only analysis, and refuted hypotheses. Where maintainers supplied better evidence, CounterProof corrected or withdrew claims. The objective was not to maximize a finding count, but to make every surviving conclusion easier to verify. From: [CounterProof home page](https://counterproof.io/#what-you-get) · [Evidence-graded security review](https://counterproof.io/case-studies/evidence-graded-security-review/) ### What does an assessment cost, and how long does it take? We don't publish prices or timelines; both depend on the scope we agree before any work starts. Email us what you need reviewed, what decision the assessment must support, and when you need it. Elenora Soons, our account manager, will arrange a call with the review team. From: [CounterProof home page](https://counterproof.io/#what-you-get) ### Which companies have you reviewed? We name a client only with their written agreement, and otherwise don't discuss engagements in public. Our case study is anonymized, and the finding on our homepage is an illustrative specimen. From: [Evidence-graded security review](https://counterproof.io/case-studies/evidence-graded-security-review/) ### If your review finds nothing, is my code secure? We do not report "we found nothing" as "there is nothing". A clean result says what was examined, not what is absent. A check that was run within the stated scope and did not detect the defect it was looking for, written into the report with its conditions, which is weaker than a statement that the code is sound. From: [What we will not claim](https://counterproof.io/refusals/) · [Glossary: adversarial code review and AI code assurance](https://counterproof.io/glossary/#recorded-negative) ## How we work ### What is a finding? One reviewer's report that a defect exists, a claim to be established against the code, not a conclusion to be counted. A candidate finding is what a reviewer raised; a confirmed finding is one established against your source with exact file and line, a reproduction path, and an impact classification; everything else is graded plausible-only. From: [A finding is not a verdict](https://counterproof.io/notes/a-finding-is-not-a-verdict/) · [Glossary: adversarial code review and AI code assurance](https://counterproof.io/glossary/) ### What does plausible-only mean? The grade for a finding that reached no evidence rung, not a claim that the question is unanswerable, only that no check settled it in this engagement, printed openly rather than buried. From: [Glossary: adversarial code review and AI code assurance](https://counterproof.io/glossary/#plausible-only) ### What is the difference between a finding we read and one we reproduced? We label every finding by how it was established. Reproduced means a test run against the unmodified code at a fixed revision. Read means supported by reading that code, without running it. We never present the second as the first. From: [What we will not claim](https://counterproof.io/refusals/) ### How is a finding closed? We do not close a finding by outvoting the reviewer who raised it. A finding closes when its raiser withdraws it or when it is demonstrated against the code. A closed finding names the revision that closed it, so a later reader can tell what was verified from what was merely asserted. From: [What we will not claim](https://counterproof.io/refusals/) · [Glossary: adversarial code review and AI code assurance](https://counterproof.io/glossary/) ### Why isn't agreement between AI models proof? We do not treat agreement between AI models as proof. Models often make the same mistake: when two are wrong, they tend to be wrong in the same way, and models from different companies are not automatically independent of each other (Kim et al., ICML 2025). A shared answer can be a shared error. From: [Agentic Stemmatics: An Introduction](https://research.counterproof.io/introduction-agentic-stemmatics.html) · [What we will not claim](https://counterproof.io/refusals/) ### What is a recorded negative? A check that was run within the stated scope and did not detect the defect it was looking for, written into the report with its conditions, which is weaker than a statement that the code is sound. It tells a reader what was looked for and how, so someone who was not in the room can judge the coverage. From: [Glossary: adversarial code review and AI code assurance](https://counterproof.io/glossary/#recorded-negative) ### Did your own review method work when you measured it? Two pre-registered, blind-scored comparisons of our full multi-model review against a single cheap pass, with published professional audits as the answer key. The full review lost one target outright; on the other it met its own precision trigger only by deleting the findings the auditor had ruled real (0–1 of 4, depending on the scorer). A later audit found the comparison less clean than first stated: the inexpensive pass's recall figures are retrieval-contaminated, so the direction stands and the margin does not. From: [We measured our own method, and it did not earn its cost](https://counterproof.io/notes/we-measured-our-own-method/) · [Agentic Stemmatics: An Introduction](https://research.counterproof.io/introduction-agentic-stemmatics.html) ### Why did you withdraw a published figure? We shipped a reproduction figure that had no retained artifact behind it. We found it, withdrew it in writing, and replaced it with a narrower claim that is actually measured. This is the withdrawal commitment being exercised, on us. From: [We withdrew our own number](https://counterproof.io/notes/we-withdrew-our-own-number/) ### What do you mean by an oracle? A check that can settle a specified claim under stated conditions: a test, compile or reproduction shown capable of returning the other answer, or a read of a normative text for what that text says. Where one exists and has been run, it outranks any reviewer's opinion on that claim, ours included. Mere execution is not enough: a test that cannot fail settles nothing. From: [Glossary: adversarial code review and AI code assurance](https://counterproof.io/glossary/#oracle) ### What are correlated errors? The tendency of different models to make the same mistake rather than independent ones, measured in some domains and not established for vulnerability review. One study across more than 350 models found that when two models both get something wrong they agree on the same wrong answer far more often than chance (Kim et al., ICML 2025, arXiv:2506.07962), on multiple-choice benchmarks and a résumé-screening task. That this should change how you read two reviewers agreeing on your code is our inference, not the study's finding. From: [Glossary: adversarial code review and AI code assurance](https://counterproof.io/glossary/#correlated-errors) ### What is a lineage? A model family: one developer's line of models, treated as a single witness. It is sampled repeatedly, and each answer is one draw from it. Reviewers from different lineages are not thereby independent; see Correlated errors. From: [Glossary: adversarial code review and AI code assurance](https://counterproof.io/glossary/#lineage) ### What evidence levels do you report for a finding? Every finding names the evidence rung it actually reached: source trace, compile-proof (the build made to fail, then pass, exactly as the finding predicts), test, or live reproduction, and for any probabilistic claim a rate with an interval and the per-trial logs kept, and never implies a higher one. The label recording what evidence was actually obtained for a confirmed finding, never implying a higher one, and describing that finding's claim rather than the codebase generally. Four rungs, weakest to strongest: source trace, compile-proof, test, live reproduction, each defined below; and for any probabilistic claim a fifth, a rate stated with an interval, from enough trials, with the per-trial logs kept, since a measured number with no kept artifact is a withdrawable number. From: [Glossary: adversarial code review and AI code assurance](https://counterproof.io/glossary/#evidence-rung) · [CounterProof home page](https://counterproof.io/) ### Is CounterProof independent, and are your AI reviewers independent of each other? Three different properties, of which we claim one: independence from the code's author. We do not claim measured statistical independence between reviewers' errors, and we do not claim commercial independence from your market: our group builds in payment and digital-asset custody, which is disclosed in writing before any engagement. The independence is structural: we issue no assessment on code authored by CounterProof or by any company in our group, and our own codebase (much of it machine-written) runs through the same adversarial review, continuously. From: [Glossary: adversarial code review and AI code assurance](https://counterproof.io/glossary/#independence) · [The Machines Are Reading the Code](https://counterproof.io/notes/the-machines-are-reading-the-code/) ## Glossary terms ### What does “Adversarial code review” mean? Reviewing source code with an attacker's question in mind (how could this be made to do something it should not), as distinct from reviewing for correctness or maintainability. From: [Glossary: adversarial code review and AI code assurance](https://counterproof.io/glossary/#adversarial-code-review) ### What does “AI code assurance” mean? A descriptive umbrella for practices aimed at establishing what AI-assisted code actually does, not a standard, a certification or a product name. From: [Glossary: adversarial code review and AI code assurance](https://counterproof.io/glossary/#ai-code-assurance) ### What does “AI code review” mean? Automated review of a diff or codebase by a language model, producing candidate findings, usually with severities attached. From: [Glossary: adversarial code review and AI code assurance](https://counterproof.io/glossary/#ai-code-review) ### What does “AI code due diligence” mean? Assessment of an AI-assisted codebase by a party outside the team that built it, commissioned before an acquisition, an investment or a major integration. From: [Glossary: adversarial code review and AI code assurance](https://counterproof.io/glossary/#ai-code-due-diligence) ### What does “Secure code review” mean? Reading source for security defects at a named revision, as distinct from testing a running system. From: [Glossary: adversarial code review and AI code assurance](https://counterproof.io/glossary/#secure-code-review) ### What does “Source trace” mean? The rung recording that the code path the finding describes was read at the pinned revision, with file and line cited, which establishes what the source says, not that the behaviour is reachable in your build. From: [Glossary: adversarial code review and AI code assurance](https://counterproof.io/glossary/#source-trace) ### What does “Compile-proof” mean? The rung recording that the build was made to fail and then pass in the way the finding predicts, which establishes the compile-time claim it exercises, and nothing about runtime behaviour. From: [Glossary: adversarial code review and AI code assurance](https://counterproof.io/glossary/#compile-proof) ### What does “Test” mean? The rung recording that a test which can distinguish the defect from its absence was run with the defect planted and observed to fail, then run with the defect removed and observed to pass. From: [Glossary: adversarial code review and AI code assurance](https://counterproof.io/glossary/#test) ### What does “Live reproduction” mean? The rung recording that the defect was reproduced against the pinned revision under stated conditions. From: [Glossary: adversarial code review and AI code assurance](https://counterproof.io/glossary/#live-reproduction) ### What does “Decidable question vs judgement question” mean? A decidable question has a check that could settle it under stated conditions; a judgement question does not. From: [Glossary: adversarial code review and AI code assurance](https://counterproof.io/glossary/#decidable-question-vs-judgement-question) ### What does “Refusal” mean? A claim we will not make on a client's behalf, published in advance so it can be held against us. From: [Glossary: adversarial code review and AI code assurance](https://counterproof.io/glossary/#refusal) ### What does “Seat” mean? One reviewing position on a panel: one AI model, from one lineage, run with its own brief and tools. A seat that returns nothing usable is recorded absent, never as agreement, and a seat found to share context with another is not counted. From: [Glossary: adversarial code review and AI code assurance](https://counterproof.io/glossary/#seat) ### What does “Scope” mean? What the engagement agreed to examine, named in the report alongside what it did not. From: [Glossary: adversarial code review and AI code assurance](https://counterproof.io/glossary/#scope) ### What does “Positive and negative control” mean? A positive control checks that an instrument can detect the thing it is looking for; a negative control checks it does not fire when the thing is absent. From: [Glossary: adversarial code review and AI code assurance](https://counterproof.io/glossary/#positive-and-negative-control) ### What does “Severity vs evidence strength” mean? Severity is a judgement about impact; evidence strength is a record of what was checked, and a finding never carries a severity its evidence does not support. From: [Glossary: adversarial code review and AI code assurance](https://counterproof.io/glossary/#severity-vs-evidence-strength) ### What does “Fix verification” mean? Re-checking a patch against the finding it targets, at the revision that closed it, recorded as its own result, by the same review team that raised it, unless the engagement says otherwise. From: [Glossary: adversarial code review and AI code assurance](https://counterproof.io/glossary/#fix-verification) ### What does “Adjudication” mean? Deciding the disposition of every candidate finding (confirmed, plausible-only, or discarded) routing it to a check where one exists and to a named person where none does, rather than to a vote count. From: [Glossary: adversarial code review and AI code assurance](https://counterproof.io/glossary/#adjudication) ### What does “Withdrawal log” mean? The written record of findings we retracted under the same engagement identifier, with the reason, issued unprompted to everyone who received the original. From: [Glossary: adversarial code review and AI code assurance](https://counterproof.io/glossary/#withdrawal-log) ### What does “Pinned revision” mean? The exact commit a review was performed against, named in the report. From: [Glossary: adversarial code review and AI code assurance](https://counterproof.io/glossary/#pinned-revision) ### What does “Witness and testimony” mean? A model's output treated as testimony to be weighed and attributed rather than trusted for its manner, because fluency does not establish truth, wrong answers arriving in the same voice as right ones. From: [Glossary: adversarial code review and AI code assurance](https://counterproof.io/glossary/#witness-and-testimony) ### What does “Authenticity” mean? The property of a reading or finding that a witness reached through its own access to the evidence rather than by copying another witness, repeating the brief it was given or following a received consensus; it says where a reading came from, not whether it is true, and is seldom shown in full: an error that a possible source has and the witness lacks can sometimes rule out copying from that source, and how confidently a reading is stated shows nothing about where it came from. From: [Glossary: adversarial code review and AI code assurance](https://counterproof.io/glossary/#authenticity) ### What does “Collation” mean? Comparing multiple witnesses against each other to work out what is reliable, a process that samples and cross-checks, and that cannot recover a defect no reviewer raised. From: [Glossary: adversarial code review and AI code assurance](https://counterproof.io/glossary/#collation) ### What does “Interpretability” mean? Studying what a model does by reading its internal computation, the weights and activations, rather than only its outputs; mechanistic interpretability identifies the features a model represents and tests their role by intervening on them. It needs access to the model's internals, which collation does not, and has no settled way yet to weigh a resemblance between two models' internals as evidence that one descends from the other. From: [Glossary: adversarial code review and AI code assurance](https://counterproof.io/glossary/#interpretability) ### What does “Transmission” mean? The passing of information from one source to the next by copying: in manuscripts, from an exemplar to its copy; among AI models, when one model's output enters another model's training data or working context. From: [Glossary: adversarial code review and AI code assurance](https://counterproof.io/glossary/#transmission) ### What does “Counterpoint” mean? Several melodic lines sounding together, each keeping its own character; used at CounterProof as an image for a review panel, where what matters is whether each reviewer contributes something of its own, which the image itself cannot show. From: [Glossary: adversarial code review and AI code assurance](https://counterproof.io/glossary/#counterpoint) ### What does “CRA test report” mean? Security-testing evidence prepared to support a manufacturer's technical documentation under the EU Cyber Resilience Act: supporting documentation, not a certification of conformity, and not a formally designated document type. From: [Glossary: adversarial code review and AI code assurance](https://counterproof.io/glossary/#cra-test-report) ### What does “Technical debt from AI assistance” mean? Future cost created by shortcomings left unresolved in AI-assisted work, not maintenance in general, and not a claim that every AI-assisted project incurs it. From: [Glossary: adversarial code review and AI code assurance](https://counterproof.io/glossary/#technical-debt-from-ai-assistance) ### What does “Consensus-as-oracle” mean? A benchmark design in which the reference label is the average of other models' answers rather than an independent outcome, so that a mistake the reference models share is scored as truth. From: [Glossary: adversarial code review and AI code assurance](https://counterproof.io/glossary/#consensus-as-oracle) ### What does “Typed-decision model” mean? A model that returns only a choice, a score or a yes/no probability over options the asker supplies, and never text; it cannot name an option the asker left out. From: [Glossary: adversarial code review and AI code assurance](https://counterproof.io/glossary/#typed-decision-model) ## Research ### What is agentic stemmatics? Agentic stemmatics proposes a way to study the relationships among machine-generated texts and the models that produce them. Its central inference is a corrected rule for lineage in model populations: ancestry is carried by significant shared idiosyncrasies, among them a shared error that meets the same condition, and a feature, or a set of features assessed jointly, counts as strong evidence only when independent origin of that agreement would be improbable and descent would make it substantially more probable. The rule is proposed, not established. Its reference site is agenticstemmatics.org. From: [Agentic Stemmatics: An Introduction](https://research.counterproof.io/introduction-agentic-stemmatics.html) · [Agentic Stemmatics: Collation, Provenance, and Conservation for an Emerging Machine Textual Tradition](https://research.counterproof.io/agentic-stemmatics.html) · [CounterProof Research](https://research.counterproof.io/) ### Where can I read the paper, and which version should I cite? Concept DOI 10.5281/zenodo.22030516 resolves to the latest deposit. To cite the exact version you read, use the version DOI given at the top of the paper; it changes with each deposit. From: [Agentic Stemmatics: Collation, Provenance, and Conservation for an Emerging Machine Textual Tradition](https://research.counterproof.io/agentic-stemmatics.html) ### What is Siglum? Siglum is a CounterProof Research project in agentic stemmatics (the philology of machine-generated text). CounterProof intends to offer paid assessments based on Siglum if tests show that it works, so we have a commercial interest in what those tests find. We are building an evidence library and evaluation tools to investigate when AI reviewers miss the same problems, and whether an additional reviewer contributes useful evidence at an acceptable cost. From: [Siglum](https://counterproof.io/siglum/) ### What is the Prolegomena? The Prolegomena to Any Future Agentic Stemmatics (not yet published), a longer companion document, covers the identical argument in more depth From: [Agentic Stemmatics: An Introduction](https://research.counterproof.io/introduction-agentic-stemmatics.html) ## Topics ### Is vibe coding safe? Of twelve agent setups, the one that most often produced working code did so in 57% of tasks; in only 11.8% did its code both work and pass the security test. The 186 tasks were chosen because each is security-sensitive and a human once got it wrong. From: [Is vibe coding safe? What a benchmark of 186 real tasks found](https://counterproof.io/notes/is-vibe-coding-safe/) ### Can an attacker use a different AI model than my reviewer? An attacker is not bound to your reviewer. They choose their own model, harness and budget, and they can iterate until something works, because they need one true positive and you need to have covered everything. You can, and a second reviewer may surface candidates the first did not. What it cannot produce is organisational independence: your team chooses what the reviewers see, frames the questions, and judges the answers, and nobody outside it has signed. From: [The attacker can run any model. How many do you run?](https://counterproof.io/notes/the-attacker-can-run-any-model/) · [Adversarial code review: what the term leaves out](https://counterproof.io/adversarial-code-review/) ### What does a benchmark score tell me about my own codebase? A measurement of performance on that benchmark, under that benchmark's conditions, where transfer to your codebase is a separate question the score does not answer. Contamination, correlated judges and best-of-N submission can each produce correct arithmetic about a question you did not ask. From: [A Clean Benchmark Is Not a Clean Codebase](https://counterproof.io/notes/a-clean-benchmark-is-not-a-clean-codebase/) · [Glossary: adversarial code review and AI code assurance](https://counterproof.io/glossary/) ## What we don't do ### Will you tell me whether a particular project or wallet is secure? Not outside an engagement. We don't give security verdicts on other people's projects, products or protocols; a finding has to be established against the code itself, at a fixed revision. From: [What settles a finding](https://counterproof.io/notes/what-settles-a-finding/) ### Can the search box review my code or give advice? No. The search box only shows verbatim passages from our published pages, with links. It doesn't review code, give security verdicts, or give legal or financial advice. If you have a question about an engagement or your own code, leave your email and we'll reply.