Hallucination benchmark — 2026-07-03

Head-to-head against the model we wrap

The figure above compares Noesis to published numbers for other systems on another corpus. The more demanding test is against the frontier model Noesis sits on top of, on identical questions, graded by the same judge. We ran it on TruthfulQA (Lin et al. 2021, arXiv:2109.07958), 300 paired questions, zero failed rows.

The difference between Noesis and the model it wraps is not statistically significant — 9.6% against 10.6%, McNemar exact on paired wrong answers p=0.6358. Noesis declines more often (29 against 17) and makes fewer wrong answers in raw count, but neither gap survives the paired test at this sample size.

What is established: going from one model lineage to four improves accuracy (p=0.0402) and reduces wrong answers (p=0.0290), and one lineage inside the full pipeline is significantly worse than the frontier model alone (p=0.0018). The panel is what brings Noesis up to parity. It is not what takes it past.

These three arms are comparable to each other and not to any published TruthfulQA leaderboard score, which is predominantly multiple-choice while these are free-text answers graded by an LLM judge. Method and every paired test: docs/benchmarks/REGISTRY.md, entry G20.

The 9.8% above is a citation hallucination rate on legal and medical probes. Through September and October 2026 we measured the architecture against public corpora on answer correctness, and published the results whether or not they favoured us. They largely did not.

• Against a frontier model given the same sources, the layer matches it rather than beating it. HotpotQA, evidence supplied, 300 paired items, LLM-judged, zero errors in any arm:

  The baseline is a current frontier model from a major lab, used through its standard API with the same documents supplied in the same prompt. It is named, with the full configuration, in the benchmark registry.

  arm	accuracy (95% CI)	wrong	$ per correct answer
  Frontier model alone	0.860 [0.816, 0.895]	4.7%	$0.0233
  Noesis, 4 lineages	0.847 [0.802, 0.883]	4.7%	$0.0397
  Noesis, 1 lineage	0.710 [0.656, 0.758]	5.3%	$0.0249

  McNemar, 4 lineages vs the frontier model: 22/26 discordant, p=0.666 — indistinguishable. Wrong-answer counts are identical (14 each; paired error test 12/12, p=1.000). Over 300 turns the panel costs $4.08 more, avoids zero errors, and loses 4 correct answers — 70% more per correct answer.
• A hallucination detector can be built, and still cannot carry a gate. Per-claim entailment against the supplied sources scores AUC 0.735 [0.687, 0.774] on RAGTruth (n=300, human span-level labels), beating every cheaper signal we tested. But it was validated at 50% error prevalence and would deploy at ~5%, where positive predictive value falls to 0.133 — roughly six correct answers destroyed per error caught. The binding constraint is prevalence, not tuning.
• Convergence itself is real, and large. One lineage to four moves accuracy 0.710 → 0.847 (n=300, p<0.0001) — the biggest effect we have measured. But it is what reaches parity with a single frontier model, not what passes it: one lineage plus our full pipeline is significantly worse than the frontier model alone (p<0.0001).

What this means: a verification layer needs errors to prevent. When the evidence is already in front of the model, a frontier system errs rarely enough that the layer matches its accuracy rather than improving on it, and an abstention gate at that error rate removes more correct answers than it catches wrong ones. Method, confidence intervals, pre-registered thresholds and the reversals of our own earlier claims are all in the benchmark registry.