How we score AI accuracy
Short answer
AI Truth Bench scores AI accuracy from published evidence, not its own tests. Every figure comes from an independent study, a public benchmark or leaderboard, a news or audit investigation, or a vendor statement labelled as a vendor claim. Each carries its test, date and source. Verdicts rest mainly on independent evidence, and tools are compared only within one test.
Where the figures come from
Every accuracy figure on the site lives in one results register, with the tool, what was measured, the value, the test, the date and a link to the source. A figure that is not in the register does not appear on a page. We use four kinds of evidence:
| Evidence type | Examples | How it counts |
|---|---|---|
| Independent studies | EBU/BBC News Integrity study, Weber-Wulff et al., Liang et al. | Carries the most weight in a verdict |
| Public benchmarks and leaderboards | Vectara Hallucination Leaderboard, AA-Omniscience, RAID | Carries weight; live boards show an "as of" date |
| News and audit investigations | NewsGuard audits, Tow Center test, court records | Carries weight for the task tested; supplies real cases |
| Vendor claims | A company's own figure on its own test | Always labelled "vendor claim"; never sets a verdict on its own |
Two labelling rules apply on every page. When a benchmark tests an API model rather than the consumer app, we say so: the Vectara leaderboard tests models such as GPT-5.5, so the page calls it "a model behind ChatGPT". When a vendor submits its own results to a public benchmark, as detector makers do on the RAID leaderboard, the page says the results were vendor-submitted.
How we compare tools
Tests measure different things. NewsGuard counts repeated false news claims; Vectara counts unsupported content in summaries; the EBU/BBC study counts news answers with significant issues. Claude's 10% false-claim rate in NewsGuard's audit and the 9.9% summary rate for Claude Opus 5.5, the model behind Claude, on Vectara describe different tasks and cannot be set side by side, so we never rank tools by mixing numbers from different tests. Comparisons on every page sit within one test, with that test named in the sentence. The full set is on AI hallucination rates by model.
How verdict words are assigned
Each tool page opens with one of four verdict words. The word reflects where the tool lands across the independent tests that include it, weighted toward the newest evidence.
- Reliable. At or near the top in every independent test that includes the tool, with errors rare enough that only high-stakes answers need a check.
- Mostly reliable. Best or near the top in most independent tests that include it, with one clear weak area named on its page.
- Mixed. Strong in some independent tests and weak in others, or middle-of-the-pack across them. Answers need checking.
- Unreliable. At or near the bottom in most independent tests that include it, or a pattern of invented sources. Treat every answer as a lead only.
Vendor claims are shown beside the independent results, but a vendor figure alone cannot lift a verdict. The current verdicts are on the home page scoreboard and in the most accurate AI assistants.
Dates and changes
Every figure shows the date of the result. Live leaderboards change, so their figures carry the date we checked them. A page changes when a source publishes new results or a reader shows that a figure is wrong, and its updated date moves only when the content does.
Known limits of the evidence
- Different tasks. No single test covers every kind of question, so verdicts weigh several.
- App versus model. Benchmarks on API models describe the model behind an app, not every setting of the app.
- Sample sizes. Some studies use small samples per tool; we use pooled figures where a study gives them.
- Older versions. Some studies tested earlier versions of a tool; their dates are shown so readers can judge.
How to report an error
If a figure is wrong, out of date or missing its source, tell us on the contact page. Include the page address, the figure, and a link to the source that shows the correct value. Corrections are checked against the source and noted on the page. How we handle sourcing and corrections is set out in editorial standards, and who writes the site is on about AI Truth Bench.
Every verdict on this site feeds our accuracy rankings, which compare tools only within one test at a time.
Questions
Does AI Truth Bench test AI tools itself?
No. We collect published results from independent studies, public benchmarks and leaderboards, news and audit investigations, and vendor statements. Each figure keeps its original test, date and source link, so you can open the evidence yourself. Verdicts are our reading of that published evidence, weighted toward independent results.
How do you treat vendor accuracy claims?
Every vendor figure is labelled "vendor claim" and shown beside independent results, never in place of them. Vendors choose their own test, grader and conditions, so a vendor figure alone cannot raise a verdict. When a vendor submits its own results to a public benchmark, the page says the results were vendor-submitted.
Why don't you rank tools on one combined score?
Because the tests measure different tasks. Repeating false news claims, adding content to a summary and naming a news source are separate skills, and their rates cannot be added or averaged fairly. We compare tools within each test and give a verdict word that reflects where each tool lands across them.
What does a "Mixed" verdict mean?
Mixed means a tool does well in some independent tests and poorly in others, or sits in the middle across them. Its answers are useful but need checking before you rely on them. The tool's page names where it is strong and where it slips, with the result, date and source for each.
How do I report a wrong figure?
Use the contact page. Send the page address, the figure you think is wrong, and a link to the source with the correct value. We check the claim against the source, correct the page if needed, and update its date. Corrections follow the process in our editorial standards.
Sources
- Benchmark
- Benchmark
- Benchmark
- Independent study
- News audit
