Hallucination benchmark — 2026-07-03
Public scorecard, methodology, and raw data for Noesis's measured hallucination rate against a reference methodology from the legal NLP literature.
Headline result
9.8% hallucination rate
95% Wilson CI [6.7%, 14.2%] ·
n = 244 classified citations across 100 legal + medical probes
Reference baseline: Magesh et al. 2024
(arXiv:2405.20362)
measured Lexis+ AI and Westlaw AI-Assisted Research at 17-33% hallucination.
The entire Noesis 95% confidence interval sits below Magesh's lower bound.
Per-domain scorecard
| Domain | Verified | Halluc. | Misattr. | Classified n | Halluc. rate | 95% CI |
|---|---|---|---|---|---|---|
| Legal (50 probes) | 163 | 0 | 14 | 177 | 7.9% | [4.8%, 12.8%] |
| Medical (50 probes) | 57 | 1 | 9 | 67 | 14.9% | [8.3%, 25.3%] |
| Overall (100 probes) | 220 | 1 | 23 | 244 | 9.8% | [6.7%, 14.2%] |
"Misattributed" = the cited source exists but doesn't support the specific claim it was cited for. "Hallucinated" = the cited source doesn't exist. Both count against Noesis in the rate, matching Magesh et al. "Unverifiable" citations (14 of 258) are excluded from the denominator, also matching Magesh — not swept under a rug: they're in the raw JSON.
Methodology
Probe set
- 100 probes total — 50 legal + 50 medical.
- Legal probes cover: US Supreme Court doctrine (Miranda, Marbury, Chevron/Loper Bright), federal statute lookup, FRCP procedural rules, agency regulation (SEC, EPA), circuit-split identification.
- Medical probes cover: FDA label indications, contraindications, drug–drug interactions, guideline recommendations (ACC/AHA, ADA, USPSTF), clinical-trial primary-endpoint attribution.
- All probes were curated to have verifiable ground truth (a specific citable source exists). Adversarial red-teaming probes are out of scope for this run — the point is measuring baseline factual grounding on realistic professional queries, not stress-testing edge cases.
Scoring — LLM-as-judge (matching Magesh 2024)
- Each Noesis response was passed to an LLM-as-judge (single call, Sonnet-class) with the same taxonomy Magesh et al. use: verified, hallucinated, misattributed, unverifiable.
- Hallucination rate = (hallucinated + misattributed) / (verified + hallucinated + misattributed). Unverifiable excluded from denominator, exactly as in Magesh.
- 95% confidence intervals are computed using the Wilson score interval for a binomial proportion — the standard choice for rates near 0 or 1 where the normal approximation is unreliable.
- 10 of 100 probes returned no citations at all (the Noesis panel declined to cite a specific source when it wasn't confident). These are counted in
n_probes_no_citationsbut not in the hallucination denominator — they don't help the rate, they just aren't hallucinations.
Honest limitations
What this benchmark is NOT
- Not human-graded. Magesh 2024 used trained legal-research analysts. We used LLM-as-judge, which is faster and reproducible but has its own failure modes on subtle misattribution. A human-grader replication is on the backlog.
- Not adversarial. The probe set was curated to have verifiable ground truth. It does not attempt to elicit hallucination through trick phrasings, non-existent-precedent traps, or jailbreak vectors. Those live in a separate probe set.
- Not a claim about all queries. This is a measured rate on legal + medical fact-recall probes with citations. It doesn't generalize to conversational queries, opinion, mathematics, coding, or other domains not sampled here.
- Single benchmark run. Frontier LLM outputs vary; we plan to re-run on a rolling cadence and publish drift. The n=100 CI already reflects sampling uncertainty; run-to-run variance on the same LLM stack is additional.
- The judge is itself an LLM. If the judge and the panel share failure modes, the rate could be biased downward. Magesh 2024 has the same concern for human graders on subtle misattribution; we make it explicit.
Reproducibility
- Raw scorecard:
/data/hallucination_bench_v0_1_judge_n100.json— 100 probe results with per-citation verdicts, judge reasoning, and durations. - Bench harness:
scripts/hallucination_benchmark_v0_1_judge.pyin the Noesis repo. - Probe set source:
scripts/data/hallucination_probes_v0_1.json. - Run environment: Noesis chat handler on Hetzner CX22 production tier, 2026-07-03.
How to read this vs. the reference baselines
| System | Domain | Hallucination | Source |
|---|---|---|---|
| Raw LLMs (GPT-4, Llama 2) | Federal court queries | 58% - 88% | Dahl 2024 arXiv:2401.01301 |
| Foundation LLMs (11 models, median) | Medical hallucination tasks | 23.4% | Kim 2025 arXiv:2503.05777 |
| Lexis+ AI · Westlaw AI-Assisted Research | Legal research | 17% - 33% | Magesh 2024 arXiv:2405.20362 |
| Noesis (this run) | Legal + medical fact-recall probes | 9.8% [6.7%, 14.2%] | This page, 2026-07-03 |
The defensible claim
Noesis's 95% confidence interval upper bound (14.2%) sits below Magesh's lower bound (17%) with statistical significance at n = 244 classified citations. That is: on this probe set, with this methodology, Noesis hallucinates less than the retail legal-AI products Magesh tested — not by a rounded point estimate, but by a non-overlapping confidence interval.
What we're publishing next
- Human-grader replication. Re-score a random 20-probe subsample with legal + medical professionals; report agreement rate with the LLM-judge.
- Adversarial probe set. Ship a separate n=50 red-team probe set (non-existent-precedent traps, misleading phrasings, subtle misattribution) and report the rate there too.
- Rolling re-runs. Weekly re-run on the same probe set to catch LLM-provider drift; publish the drift chart alongside this page.
- Per-verifier attribution. Break down which of the 7 verification layers (grounding gate, citation verifier, PubMed/EDGAR/EUR-Lex adapters, HFV, ISP, C3, VigilantAI) caught which class of would-be hallucination.