Work in progress. Siglum is a CounterProof Research project in agentic stemmatics (the philology of machine-generated text). CounterProof intends to offer paid assessments based on Siglum if tests show that it works, so we have a commercial interest in what those tests find. We are building an evidence library and evaluation tools to investigate when AI reviewers miss the same problems, and whether an additional reviewer contributes useful evidence at an acceptable cost. It will record what is known about how models are related, the exact settings used in each test, and every answer given. It will also note where a model’s origins are uncertain. Related methods already exist, among them PhyloLM and Stemma, and we will compare our results with theirs. Its practical value remains to be demonstrated.
Why it matters. Some organisations ask several AI models to check the same work, such as code, security findings or written claims, and treat their agreement as confirmation. That agreement is worth less than it looks. When two models are both wrong, they often give the same wrong answer, and more capable models do so more often (Kim et al., 2025). In one study, nine frontier models labelling the same texts gave about as much information as two independent judges (Kohli, 2026); that task was classifying text, not reviewing work, so the result bears on review panels by analogy rather than by direct measurement. Models from different companies can still have correlated errors, so choosing several vendors does not by itself settle the question. Siglum aims to replace the assumption with a measurement: for a given kind of work and a given set of AI reviewers, how often their mistakes coincide, how many independent checks the panel really adds up to, and whether one more reviewer would catch anything the others miss, at a cost worth paying. If it works, “four AI models agreed” could come with a number attached.