Every large language model is, underneath the branding, a guessing machine — and the guessing is not a defect that better engineering will someday retire. It is the mechanism.
When ChatGPT or Claude or Gemini produces a sentence, it is not retrieving a fact and reciting it. It is predicting, one token at a time, which word is statistically most likely to come next given everything before it. Most of the time that lines up with reality, because reality is what the training data mostly described. Sometimes it does not, and the model produces a confident, fluent, entirely wrong answer — a fake case citation, an invented statistic, a source that was never written. That is a hallucination. It is not the model lying, because lying requires knowing the truth and choosing to contradict it. The model has no such knowledge to betray. It has only probabilities.
Why the Guessing Doesn't Stop
In September 2025, OpenAI researchers Adam Tauman Kalai and Ofir Nachum published a paper, "Why Language Models Hallucinate," pinning the cause on something more mundane than mysticism: the way models are graded. Part of it is a statistical floor — if a model can't reliably distinguish a correct statement from a plausible-sounding incorrect one, some error rate is mathematically guaranteed. But the sharper point is about evaluation. Nearly every benchmark labs use to rank models scores an answer as simply right or wrong — no partial credit for "I'm not sure." A model that says "I don't know" scores the same as one that's flatly wrong; a model that guesses has a shot at being marked correct. The training loop rewards confident guessing over honest uncertainty, and the model learns exactly that lesson.
That's an incentive problem, not a physics problem — the paper's proposed fix is to change how benchmarks score abstention, not to bolt on a hallucination detector. Hallucination isn't a bug waiting for a patch. It's the predictable output of optimizing a next-token predictor against tests that punish humility.
Where the Glitch Is the Point
The same mechanism that invents a fake court case is what lets a model draft twelve headline options, sketch a plot twist nobody asked for, or connect two ideas that have no business sitting next to each other. Thoughtworks has argued publicly that hallucination should be treated as a feature rather than a defect to engineer away, because the unpredictability that produces a wrong fact in one context produces a genuinely novel idea in another. Research on generative AI in creative and educational settings backs this from a different angle: unexpected, ungrounded outputs helped people break through creative blocks and disrupted conventional thinking during brainstorming, precisely because the model wasn't constrained to say only verified things.
Ask a model for fifty taglines or a wildly speculative business idea, and the exact quality that makes it occasionally wrong — its willingness to generate something not strictly derived from a source — is the quality doing the useful work. You don't want a brainstorming partner that only offers phrases it can cite. You want one that guesses well.
Where It Wrecks You
Take that same guessing instinct into a courtroom brief, a tax filing, or a diagnosis, and it stops being a feature. Legal researcher Damien Charlotin maintains a public database tracking AI-hallucination cases in courts worldwide; by May 2026 it had logged roughly 1,490 decisions globally — more than 1,000 in the United States alone — where a court responded to a party's reliance on fabricated AI material, with new entries added at five or six a day. The penalties have stopped being nominal. A federal judge in Oregon sanctioned two attorneys a combined $110,000, the largest AI-hallucination penalty in U.S. legal history, after they submitted 23 fabricated citations and eight invented quotations. An Omaha attorney was suspended after a Nebraska divorce-appeal brief turned out to have 57 of its 63 citations defective, twenty of them hallucinated cases outright. Courts have been unambiguous: there is a non-delegable duty to verify every citation regardless of what tool produced it, and no bar permits blind reliance on AI output.
The same logic extends past law. A model asked for a drug interaction, a tax position, or a financial projection is being asked for something that must be true, not something that merely sounds plausible — and plausible is all the underlying mechanism is built to deliver.
The tool that helps you brainstorm a business plan is the same tool that will invent a court case with total confidence. The difference isn't the model. It's what you're asking it to be right about.
The discipline isn't complicated, but it takes actually doing it. Before trusting a model's output, decide which category the task falls into.
- If the answer must be factually true — a citation, a number, a medical claim, a legal standard — verify it against a primary source before it leaves your hands. Independently. Every time.
- If the task is generative — ideation, drafts, brainstorming, fiction, alternative phrasings — let the model guess freely, and treat the output as raw material, not the finished product.
- Never let a model's confidence stand in for your verification. Fluency and accuracy are unrelated. A hallucinated citation reads exactly as smooth as a real one.
Take a position: the industry's fixation on "solving" hallucination targets the wrong thing. You cannot train the guessing out of a next-token predictor without training out the quality that makes it useful for anything beyond lookup. The fix isn't a hallucination-free model. It's a user who knows which job they hired the tool to do — and checks its work on the jobs where being wrong has a cost.
