The Illusion of the Score

When we evaluate an AI model, we treat its benchmark score like a speedometer: a clean, objective reading of capability.

It isn’t. A benchmark score is not a measurement. It is testimony — a fluent claim whose warrant lies outside the act of generation. The mechanism that emits a perfectly grounded answer and the mechanism that emits a confident hallucination are the same mechanism, and nothing on the surface of the output distinguishes them. So a score has to be treated the way a detective treats a witness statement: weighed, cross-examined, and traced back to its source.

At CounterProof, our day job is adversarial review of machine-written code — the same discipline applied to a different artifact — which is why we watch the evaluation ecosystem closely. Cross-examine it and you find three well-documented failures the industry treats as separate problems: training-data contamination, judge bias, and leaderboard gaming. In reality they are three symptoms of one disease:

The measuring instrument is not independent of the thing it measures — and in every case, the resulting number looks exactly like a valid one.

Here is how the illusion is constructed, where it honestly breaks, and how to see through it.

1. The contamination illusion: the test is in the training data

The oldest failure is benchmark contamination: the evaluation set has leaked into the training corpus, so the model is not being tested — it is being asked to recall.

This is not marginal. Studies have found test-set leakage across widely used benchmarks including MMLU and GSM8K, with rates that vary substantially by detection method but are consistently material rather than negligible. The cleanest demonstration is the control-group kind. MathArena (arXiv:2505.23281, NeurIPS 2025 Datasets & Benchmarks track) evaluates models on competition problems immediately upon public release — fresh enough that they cannot yet have been scraped into anyone’s training corpus — and sets that against performance on older, widely circulated sets. It reports strong signs of contamination in AIME 2024 alongside genuinely uncontaminated results elsewhere. Note the mechanism, because it is easy to misread: the protection is temporal, not secrecy. The problems are entirely public. They are simply too new to have been absorbed.

The structural point: a contaminated benchmark does not produce a wrong number. It produces a right number about the wrong question. It accurately measures recall and reports it as reasoning.

2. The consensus trap: why three AIs agreeing means less than you think

As benchmarks moved toward open-ended tasks with no single right answer, the field adopted LLM-as-a-judge. It scales, it is cheap, and it correlates reasonably with human preference. It also carries biases that are now precisely documented: judge models rate their own outputs more favourably (self-preference bias, arXiv:2410.21819); the order of two candidate answers changes the verdict (position bias, measured across 15 judges, 22 tasks and over 150,000 evaluation instances, arXiv:2406.07791); and longer answers score better whether or not they are more correct (verbosity bias).

The standard mitigation is a panel: several judge models, verdict by consensus. If three different frontier models agree, the answer must be right.

That is where the deepest failure lives, because of a separate finding sitting underneath the biases: model errors are correlated across vendors, and the correlation rises with capability. Research across more than 350 models (Kim et al., “Correlated Errors in Large Language Models,” ICML 2025, arXiv:2506.07962) found that when two models both get something wrong, they agree on the same wrong answer far more often than chance — around 60% in leaderboard settings — and that this persists across distinct architectures and providers. More capable models exhibit more correlated errors, not fewer.

A panel of three frontier judges is therefore not three independent measurements. It is three draws from a partly shared error distribution, presented as agreement. Where the judges share a blind spot, their consensus is not confirmation — it is the blind spot wearing the costume of confirmation. Averaging correlated witnesses does not cancel their error. It launders it.

Voting itself fares no better where it matters: a capability-controlled audit of 31,900 three-model ensembles (arXiv:2607.20768, preprint) found that majority vote beats the best single ensemble member in under 10% of subsets — an effect its own author characterises as modest and configuration-dependent. The natural reading — that consensus selection filters out minority-correct answers — is our inference from that association, not a finding the papers state.

One honest boundary. The claim is conditional: consensus is unsafe as a truth procedure, where independence is unmeasured or failing. Under genuinely measured independence, pooling is fine — and precision-first triage under alert fatigue is a legitimate operating choice. The problem is that the measured situation is the opposite of the assumption, and it worsens with capability. The distinction between triage and truth-reconstruction should at least be explicit.

The twist that sets up everything below: shared mistakes do not even prove the models are related. Two models trained on overlapping data toward similar objectives converge on the same plausible-but-wrong answers as attractors — the mode of a shared objective — with no copying involved. If you want actual lineage signal, you look at quirks, not errors. Models carry stable, model-specific idiosyncrasies — formatting habits, structural tics — strong enough to attribute text to its source model with 97.1% accuracy across five systems (ICML 2025, arXiv:2502.12150). But not every quirk counts: the lexical fingerprint behind the “delve” epidemic appears in unrelated post-2022 models and in human scientific writing (COLING 2025, arXiv:2412.11385), which looks like convergent training methodology, not kinship. The clean case is the engineered watermark — a statistical mark planted in a teacher’s outputs and recovered in a distilled student: inheritance demonstrated, though fragile against paraphrase (ACL 2025, arXiv:2502.11598). Hold onto this distinction between attractors and inheritance. It becomes the foundation for dating models and reconstructing their family tree.

3. The leaderboard illusion: a maximum dressed as a sample

The third failure is about how numbers reach public rankings at all.

The Leaderboard Illusion (arXiv:2504.20879, NeurIPS 2025 Datasets & Benchmarks track; a team from Cohere Labs, Stanford, Princeton and others) examined Chatbot Arena and found that undisclosed private testing let some providers evaluate many model variants before public release and disclose only the best. Meta tested 27 separate variants in a single month ahead of its Llama 4 release; Google and Amazon also submitted hidden variants; smaller labs typically submitted one or two models, unaware the option existed.

The technical objection is precise: this best-of-N submission strategy violates the statistical assumptions of the Bradley-Terry model Arena uses to compute rankings. The scores are not wrong arithmetic. They are correct arithmetic applied to a sample that was selected on the outcome being measured. A best-of-27 result and a single-shot result are not the same kind of number, yet they appear in the same column.

The remedy follows directly from the diagnosis: leaderboards should require disclosure of every submitted variant, report the full distribution of scores rather than the survivor, state uncertainty honestly, and apply a statistical penalty for undisclosed multiple submissions. None of this is exotic — it is the disclosure standard clinical trials adopted decades ago.

The property all three share

Contamination, correlated judges, and best-of-N disclosure look like a data-hygiene problem, a psychology-of-models problem, and a governance problem. They are one problem:

• Contaminated benchmark: the answer key is inside the instrument.
• Correlated judges: the instrument shares the failure modes of the thing it grades.
• Best-of-N leaderboard: the instrument reports a selected maximum and presents it as a draw.

That is also why a model reviewing its own code produces a review with no evidential content — the most compressed version of the same failure. Code written by a model and reviewed by the same model, or by its close kin, has been reviewed by nobody.

The closing archive: the feedback loop, stated honestly

These failures are happening against a larger backdrop. Until roughly 2022, models were terminal readers of the human archive: they consumed our books and code and contributed nothing back. That era is over. Model output now re-enters training corpora through four channels: direct distillation (copying a chosen teacher); ambient contamination (machine text scraped unlabelled into the next corpus); deliberate synthetic data (Meta’s Llama 3.1 card records “over 25M synthetically generated examples” in fine-tuning; Microsoft’s Phi-3 and Anthropic’s Claude 3 cards describe similar internal generation); and human-mediated diffusion — model phrasings absorbed by human writers, measured at 13.5% of 2024 biomedical abstracts showing an LLM-processing signal (Kobak et al., Science Advances, 2025).

How much of the web is already synthetic? Estimates genuinely diverge: a keyword-frequency working paper puts it near 30–40% (arXiv:2504.08755), while a detector-based sample of 65,000 Common Crawl URLs reports a rise from ~2% in 2020 to roughly half by 2024–25 — and AI-detector reliability is itself disputed. Quoting either number as settled would be exactly the sin this article is about. What is not disputed is the direction.

The scribe analogy now becomes exact. A medieval copyist smoothing a hard reading into an easy cliché is the same operation as recursive training eroding distributional tails: model collapse is demonstrated under pure recursion, where each generation trains only on its predecessor’s output (Shumailov et al., Nature, 2024). The honest boundary matters: where synthetic data accumulates alongside real data rather than replacing it, test error stays bounded and collapse does not occur (Gerstgrasser et al., 2024). So the defensible statement is not “the archive is collapsing” — it is that the degradation mechanism is real, measured in the pure case, and gated by how much real data the tradition keeps ingesting.

For evaluation, the consequence is that contamination stops being an accident and becomes the ecology’s default condition — which makes dating and baselining the new core skills. The pre-2022 archive becomes what one analogy, coined within days of ChatGPT’s release, calls low-background steel: pre-nuclear steel salvaged from shipwrecks because it predates atmospheric testing — here, the sealed stratum of demonstrably human text, now the object of a cataloguing project (lowbackgroundsteel.ai). Its value as a baseline rises monotonically with the contamination of everything after it.

For code specifically, the loop has a security edge: a defect idiom emitted by a model this year is committed to public repositories, scraped, and trained into the models of two years from now — meaning vulnerability patterns could be inherited through the archive, not merely regenerated. (That last step is an inference from demonstrated transmission channels, not a measured finding — and the only ethical way to settle it is to trace existing patterns, never to seed them.)

The four questions for the machine age

We cannot decontaminate the whole ecosystem. We can attach conditions to every number and make the conditions explicit. Stop asking “is this benchmark good?” — that question has no answer. Ask:

1. Was the evaluation set available to the model during training? If unknown, the score is an upper bound, not a measurement. Three answers are clean: a genuinely private held-out set; material too fresh to have been absorbed, which is the MathArena approach of testing on problems the day they are released; or dynamically generated items. What is never clean is a public benchmark that has been sitting on the open internet for two years.
2. Is the judges’ independence measured, or assumed? Vendor labels are not independence. The working metric is per-task error correlation between judge seats, tracked over time. Where it is high, consensus is worthless; where it is unmeasured, consensus is uninterpreted. This is measurable today — it just usually is not measured.
3. How many attempts produced this number, and were the others disclosed? Demand the distribution, not the maximum. Undisclosed best-of-N should be penalised or excluded, never displayed beside single-shot results in the same column.
4. Is dissent adjudicated, or averaged away? A method that discards minority findings throws away the only signal that can catch a correlated majority being wrong together. The fix is not “trust every dissenter” — at low prevalence, dissent is mostly noise, and full human adjudication does not scale. It is that a minority finding triggers a cheap oracle check — run the test, read the spec, execute the code — rather than a vote. Dissent is routed to evidence, not to arithmetic.

Where this goes next

Three developments make this more than a critique:

• Sealed witnesses. A frozen model vintage cannot inherit post-seal contamination. An archive of open-weight models (closed models cannot be archived by third parties — the open-weight ecosystem is the archivable tradition) provides temporal decorrelation for review panels, drift probes — an idiom that entered the tradition in year N reads as normal to year-N+1 models but stays high-perplexity to a sealed older witness, which is a dating instrument — and baselines that make degradation claims measurable rather than atmospheric.
• The stemma of models. If lineage rides improbable shared idiosyncrasies — the quirks, not the errors — then the ecosystem’s family tree can be reconstructed from outputs alone and validated against known lineages: teacher→student distillations, base→fine-tune families. Where reconstruction fails — multi-teacher distillation, synthetic admixture — the failure modes are themselves the map of contamination.
• A growing mitigation landscape. Dynamic benchmark generation, evaluation of the reasoning trace rather than only the final answer, and provable evaluation inside trusted execution environments are all live research directions. Each helps. None removes the independence question; each relocates it.

Why we care

At CounterProof we do adversarial review of machine-written code, and the four questions above are not theoretical to us — they are our operating rules: findings are adjudicated rather than averaged, dissent is routed to an oracle rather than outvoted, and seat independence is measured rather than assumed from a vendor label. We hold this discipline because we have watched correlated consensus fail — including in our own instruments, which we correct in writing, publicly, when they do.

This article is a public-facing distillation of our working paper, “Agentic Stemmatics: Collation, Provenance, and Conservation for an Emerging Machine Textual Tradition” (Soons, 2026, in preparation), which carries the full argument — including the conditional form of every claim made here, and the stated limits of our own method. It is readable in full here, as a draft preprint that has not been peer-reviewed and says so.

---

Sources cited above were checked against their published abstracts and listings; where a claim rests on a specific reported figure we name the source so you can read the paper rather than take our summary for it. Contamination and synthetic-share estimates vary substantially by detection method, and we have deliberately refused to quote a single headline percentage as though the field agreed on one. Claims labelled as our inference are ours, not the cited papers’. If we have something wrong here, we will correct it in writing on this page.