If you’ve ever chatted with an AI assistant, you’ve probably noticed how confidently it speaks—even when it’s dead wrong. Why does this keep happening? A new study from OpenAI and Georgia Tech researchers claims the problem lies less in the technology itself and more in how these systems are trained and tested.
Guessing Pays Off
According to the team, today’s large language models (LLMs) are built to guess. The way benchmarks are designed, refusing to answer is treated just as badly as staying silent. So models learn the obvious lesson: never say “I don’t know.”
“Think about a school exam,” the researchers explained in a blog post. “If you don’t attempt the question, you get zero. If you take a wild guess, you might score. The same logic applies to language models—they’d rather gamble than leave a blank.”
This explains why we often see AI respond with polished, confident sentences, even when the facts are shaky.
Errors by Design
The study stresses that hallucinations are not freak accidents. They’re baked into the math. Models learn by predicting the next word in a sequence. No matter how big the dataset, mistakes are inevitable.
One concept the researchers highlight is the “singleton rate”—how often a fact shows up only once in the training data. If 20% of birthdays appear only once, then the model is almost guaranteed to mess up about 20% of birthday questions. It’s like trying to memorize an encyclopedia but forgetting the rare entries.
The Calibration Dilemma
Another challenge is calibration. A good model, in theory, should reflect probabilities—like a weather forecast saying there’s a 60% chance of rain. But a chatbot that constantly refuses to answer would feel useless. To stay helpful, the system has to take risks. And risks mean errors.
The irony? Post-training methods like reinforcement learning, meant to reduce hallucinations, can sometimes make them worse. Because benchmarks still punish uncertainty, models continue to act bold even when they shouldn’t.
The Leaderboard Effect
The obsession with leaderboards only deepens the problem. Popular tests such as MMLU-Pro or GPQA don’t reward caution. They score models in black-and-white: right or wrong. So an AI that blurts out answers, correct or not, will outrank one that plays it safe.
This creates, in the researchers’ words, an “epidemic of overconfidence.”
In practice, companies optimizing for these scores end up with chatbots that bluff more often than they should.
A Different Way Forward
The paper proposes shaking up the scoring system. Imagine a benchmark where:
- Correct answers earn points.
- Wrong answers cost double.
- Saying “I don’t know” doesn’t affect your score.
Suddenly, the incentive flips. Now the smart move is to admit uncertainty unless you’re genuinely confident.
This is what the researchers call behavioral calibration—teaching models to express doubt the way humans do. Instead of hiding behind probability scores, the AI would plainly say when it’s unsure.
What the Study Doesn’t Solve
Of course, the framework isn’t perfect. It deals mainly with plausible falsehoods, not the random nonsense models sometimes spit out. It also doesn’t cover subtler behaviors, like asking clarifying questions.
Even retrieval-based models, which use search engines to double-check answers, aren’t immune. If the rules still punish uncertainty, they too will guess when their search fails.
Looking Ahead
The next step, researchers say, is to test whether people actually prefer an AI that admits ignorance more often. Will users trust it more? Or will they get frustrated by frequent “I don’t know” replies?
Another avenue is to experiment with new benchmarks that use negative scoring for wrong answers. If those catch on, we may see a cultural shift in how AI progress is measured.
Bigger Picture
The real story here isn’t just about algorithms—it’s about incentives. For years, the field has rewarded models for looking smart under pressure. That chase for leaderboard glory may have come at the cost of honesty.
As one researcher put it, “We’ve built machines that bluff because we told them bluffing was the way to win.”
The lesson? If we want AI that acts like a trustworthy colleague rather than a cocky student, the rules of the game need to change. Until then, hallucinations will remain part of the package—no matter how advanced the technology gets.