The Gap Between Plausible and True

In 2025, one of the world's most trusted sellers of expertise got caught selling plausibility.

Australia's government paid Deloitte AU$440,000 for a compliance review. The report looked exactly like what a top-tier firm produces: structured, footnoted, confident. Then a University of Sydney academic checked the citations. Over a dozen were fabricated — papers that didn't exist, phantom footnotes, an invented quote from a Federal Court judgment attributed to a misnamed judge. Deloitte admitted a GPT-4o toolchain helped write it and refunded its final installment.

Note what didn't fail: the prose. Every reader found it persuasive until one human checked. That's the signature failure of the era — not wrong answers that look wrong, but wrong answers that look right.

Not wrong answers that look wrong — wrong answers that look right.

The pattern is measured, not anecdotal. Stanford's RegLab tested 200,000 verifiable legal questions: frontier models hallucinated on 69 to 88 percent of them, and — the finding that should keep executives up at night — the models "lack self-awareness about their errors." Courts worldwide have now documented over a thousand cases where parties relied on fabricated AI content.

Here's the frame that makes it manageable. Truth is aim plus sight. AI delivers sight on demand — fluent, fast, in whatever direction it was pointed. What it cannot do is choose the direction, because choosing requires a stake. It stakes nothing on being right. It has no willingness to be wrong. It can be connected to machinery for checking — but it cannot own the obligation to find out.

Someone in your organization must own that obligation. Not as a vibe — as a design requirement. Two tests to run on any intelligence system before it ships:

• The room test. When the human decides, is the whole room visible, or just the row? A claim, a report, an order can be internally consistent and still false to the situation around it. Clean on paper is plausible. Your job is true.
• The correction test. When this system is wrong, how does the organization find out — and how fast? If the honest answer is "the post-mortem," you've scheduled the discovery of truth's absence.