# Our own code: what the second review found, and what found it > Our companies' payment and custody code, reviewed twice. The second review filed 39 Critical findings, 16 of them proven by running them; earlier reviews had described 7. What changed was mostly the procedure, and the record cannot yet say how much the added models contributed. Source: https://counterproof.io/case-studies/our-own-code/ · Published 23 October 2026 · CounterProof is a practice of Clavestra Capital Limited (Malta, C 113987). In 1628 the Swedish warship *Vasa* was tested before she sailed. Thirty sailors ran from side to side across her deck, and after three passes she was rolling so far that the test was stopped. She sailed anyway, and capsized in Stockholm harbour on her maiden voyage. The test had worked. Nobody acted on it. This case study is about our own code, and about a test that did not work as well as it looked. ## What we reviewed Our companies build payment and custody software: modules that mint and redeem eCash against several kinds of asset, a payment gateway, client libraries, monitoring, a proof-of-reserves service and the runbook for setting up a federation. Nineteen repositories in all. Much of the code was written with AI coding agents. None of it has been launched, and no customer funds have ever been held in it. This is the workshop where our method was built. It is not an independent review: the code is our companies' and the review was run by us, which is why we never issue an assessment on code from our own companies (see [the disclosure on our home page](https://counterproof.io/#disclosure)). ## Two reviews **The first review, April to June 2026.** Each repository was audited by one of our founders working with AI coding agents, almost entirely from one model family: a round of findings, a round of fixes, and a check that the fixes closed what they claimed to close. In some repositories there were three rounds. The write-ups are committed in each repository. Before July, at least one repository had also been reviewed by an OpenAI model, and one finding traces back to a review by another AI product. **The second review, from July 2026.** We reviewed the same code again under a written procedure: a security brief for each trust boundary, separate specialist passes over separate parts of the code, a cryptography adviser for the questions a general reviewer gets wrong, proofs of concept run against the unmodified code where that was possible, and models from several other companies, some hunting for defects on their own and some attacking the findings. Every Critical finding records how it was established: by running it, or by reading the source. ## What the second review found The register holds 275 findings: 39 Critical, 76 High, 127 Medium, the rest Low or informational. Of the 39 Critical findings: - **16 were proven by running them**, with a failing test or a proof of concept, run and recorded. The other 23 were confirmed by reading the source, with file and line. - **7 had already been described in an earlier review committed to the repository**, five of them in April and two in June. The other 32 had not. At least one of the seven had been rated Low there and deferred. - **21 are fixed on the branch that ships, 2 partly, and 16 remain open**, as recorded in our register on 7 September 2026. Among those now closed: an OAuth check that skipped the client secret whenever the caller left it out; a key-generation routine that gave every guardian every secret share, so the threshold protected nothing (closed by removing the routine, not by repairing it); an eCash spend key derived from data anyone could read; and a settlement path that accepted underpayment. By our severity rule a Critical is reachable without exotic preconditions. The open ones are described below only by class. Nothing they touch is live. ## What found them We started the second review on a premise: a model reviewing work produced with models of its own family shares their blind spots, so we added models from other companies. The record supports a narrower story than that. (The counts below were extracted from our records by AI agents of the same family; see the limits at the end.) Seven of the 39 Critical findings had already been described in an earlier review. Of the other 32, **23 were first written down by a reviewer from the same model family as the earlier reviews**, now working under the new procedure. **1 was first raised by a model from another company.** For 8 the record does not say who was first. Among the 59 High findings that no earlier review had described, the split is 30, 10, and 18 unrecorded, with one raised by an automated check. "First written down" is not "only found". The other models hunted on their own as well, but we did not record which of them reached each defect before seeing the first list. Without that record we cannot say how many of the defects the other models would have found alone, or missed. What the other models did contribute is on the record: - **Findings they recorded first:** one Critical and ten High findings that no earlier review had described. The Critical: a proof-of-reserves design in which the commitment to liabilities was not cryptographically tied to the attestation of reserves. This counts first records, not catches nobody else would have made. - **Corrections.** A severity raised to Critical once a second family corroborated it. A review that called our own write-up's "5/5 unanimous" framing an oversell, which it was: every reviewer had been briefed by the same coordinating model, so their agreement was weaker evidence than it looked. And the most severe class we found, a way to forge eCash notes in two modules, was found by a specialist cryptography adviser from the same model family, after models from five families, ours and four other companies', had reviewed those modules without reporting it. A different question found it. Another reviewer asking the same questions did not. ## Found is not fixed One Critical finding, a weakness in how deposits were checked, had been described in the April review. Its fix was written. It sat on a branch that was never merged, and the branch that ships still carried the defect when the second review reached it. A review that ends in a document has not ended. Our register records each status against the branch that ships, by name, for that reason. ## What this shows The April review was real work and caught real defects. It missed most of what mattered. A second review under a written security procedure found 32 Critical defects the first had not described, and proved 16 of the 39 by running them. Had the code launched before the second review, those defects would have launched with it. The record does not show that adding model families is what made the difference. The procedure, the effort and the set of reviewers all changed at once, and most first records of the new findings came from the same family as before. The other models added catches, corrections and checks that the main pass would not have made on its own, but this record cannot say how much of the gain is theirs. That is the question [our next comparison](https://counterproof.io/#what-goes-wrong) is designed to answer, with the protocol published before it runs. What the record does show is what a client is buying: not a count of findings, and not a number of models, but the procedure. A brief per trust boundary, specialist passes, execution where it is possible, outside models attacking the findings, every finding labelled by how far it was established, and a register that says what is fixed on the branch that ships. ## Limits - Our own companies' code, and a review run by us. This is not an independent assessment. - The attribution counts were extracted from our review records by AI agents from one model family, the same family as the coordinating model and most first-pass reviewers. We checked the attribution or the earlier-review entry of 11 of the 115 rows by hand against the source and the repository history. Two reviewers from other companies recounted the Critical rows from the extracted table and matched every count; they could not check the extraction itself. Where the records do not say who raised a finding first, we count it as unrecorded, not as anyone's. - "Already described in an earlier review" counts only review documents committed to the repositories before the finding was filed. Reviews held in pull requests or issues are not counted. - "Fixed" means fixed as recorded in our register on 7 September 2026, on the branch that ships. Register statuses can go stale; we re-check them against the repositories, but this page states the register. - The counts cover the 115 Critical and High findings, not all 275.