Does AI Hallucination Really Get Better With Newer Models?

The belief that AI hallucination gets rarer with each new model release didn't come from nowhere. Every major lab frames its launches that way: benchmark charts showing improved factuality scores, release notes citing fewer ungrounded answers, safety sections describing new mitigation techniques. Scale itself seemed to back the story — bigger models, more training data, and reinforcement learning from human feedback all correlated loosely with better performance on most benchmarks through 2023 and 2024, so it was reasonable to extend that line forward and assume hallucination would keep shrinking the same way. Reasoning models arrived on top of that assumption, marketed as the most careful, most deliberate class of model yet — the ones that check their own work before answering. That branding is exactly what makes the actual benchmark numbers, run on the same test across every major provider, worth checking against instead of taking on faith.

Vectara's Hallucination Leaderboard is the closest thing to an apples-to-apples test across labs: every model gets the same articles and the same instruction — summarize using only the provided text — and Vectara's own detection model scores each summary for factual consistency, flagging anything below a 0.5 confidence threshold as a hallucination. In November 2025, Vectara overhauled the test set from roughly 1,000 short articles to more than 7,700 longer ones, up to 32,000 tokens, spanning law, medicine, finance, technology and several other domains, and re-scored every model — old and new alike — against the harder set. That re-scoring matters here: the 9.6% carried by GPT-4o and the 13.6% carried by Gemini 3 Pro preview were both measured on the identical current benchmark, not two different eras of test.

Run the same before-and-after test on every major lab's newest flagship against the one it replaced, and the results split into two groups rather than the clean downward line the myth predicts. OpenAI's GPT-5.5 (9.3%) and Anthropic's Claude Sonnet 4.6 (10.6%) and Claude Opus 4.7 (12.0%) all landed within half a point of their predecessors — GPT-4o at 9.6%, Sonnet 4 at 10.3%, Opus 4 at 12.0% exactly. Nothing moved enough to call it progress, but nothing collapsed either.

Google and xAI · The Two Real Outliers

Google and xAI are the two providers where the myth doesn't just fail to hold — it inverts. Gemini 3 Pro preview scored 13.6% against Gemini 2.5 Pro's 7.0%, nearly double, and didn't crack Vectara's top 25 despite being Google's newest and most capable model. Grok-4-fast-reasoning scored 20.2% against Grok-3's 5.8%, more than tripling it and landing among the highest hallucination rates of any frontier model tested. Both companies shipped reasoning-heavy successors — which, as the next section shows, is very likely not a coincidence.

DeepSeek sits between the two groups: DeepSeek-V4-Pro scored 8.6% against DeepSeek-V3's 6.1%, a real but smaller regression than Google's or xAI's. Across all five providers checked, the tally is zero meaningfully improved, three held roughly flat or worsened slightly, and two got substantially worse — not the picture 'each generation hallucinates less' predicts.

Reasoning models were supposed to be the safer category — models that think through a problem step by step before answering should, in theory, catch their own errors. The Vectara data says otherwise. OpenAI's o3-pro scored 23.3% against its standard-tier sibling GPT-5.4-pro's 8.3%, and o4-mini-high scored 18.6% against GPT-5.4-mini's 5.5% — a gap of up to 3.4x, the widest reasoning-versus-standard split in the dataset. DeepSeek's R1 scored 11.3% against DeepSeek-V3's 6.1%, nearly double.

OpenAI's Own Numbers Say the Same Thing

OpenAI's own system card for o3 and o4-mini, published alongside the models in April 2025, runs a separate open-domain test called PersonQA and finds the identical pattern on its own benchmark: o1, the previous reasoning generation, hallucinated on 16% of questions, and o3-mini on 14.8%. o3 jumped to 33%, roughly double o1's rate, and o4-mini reached 48%, close to triple it. OpenAI's stated explanation is that reasoning models make more claims overall — more correct ones, but also more hallucinated ones — and the company writes plainly that more research is needed to understand why the rate keeps climbing.

The effect isn't just 'reasoning mode' as a universal on-off switch, either. xAI's Grok-4-fast is the same underlying model tested with reasoning turned off and on, and the two scores are close: 19.7% without reasoning, 20.2% with it. The largest gaps in this dataset all come from comparing separate, reasoning-specialized model lines — OpenAI's o-series, DeepSeek's R1 — against their standard siblings, not from flipping a switch inside one model.

None of this means every new model hallucinates more than the one before it — the trend is genuinely non-monotonic, not a uniform decline. On OpenAI's own PersonQA benchmark, GPT-4.5, a standard non-reasoning model released shortly before o3, scored 19% against GPT-4o's 30%, an improvement of more than a third and the best result in that entire sequence. It's only when o3 and o4-mini launch right after that the numbers jump back up to 33% and 48%. The honest read of the data isn't 'AI hallucination always gets worse' — it's that the recent regression is concentrated specifically in reasoning models, while non-reasoning progress kept moving in the expected direction right up until it didn't.

Every number in this piece names its benchmark for a reason: there is no single 'hallucination rate' that applies to a model across every kind of task. Vectara's leaderboard measures something narrow and specific — whether a model stays faithful to a document it was told to summarize. PersonQA measures something completely different — whether a model gets open-domain factual questions about real people right without a source document to lean on. Those are different skills, and they produce very different absolute numbers for the same models.

The spread can be dramatic. Gemini-2.0-Flash reportedly scored 0.7% on Vectara's leaderboard but around 7.6% on Vectara's own harder FaithJudge variant — more than ten times higher on the same model, under a harder test. GPT-5 has been reported scoring 47% on the unrelated SimpleQA open-domain benchmark, a number that would look alarming set next to its low single-digit Vectara summarization score, if the two were ever compared directly rather than treated as measuring different things. Quoting a hallucination percentage without naming the benchmark behind it is one of the easiest ways to make AI look better, or worse, than the data actually supports.

Run the numbers the way the myth predicts — pick any provider's newest flagship, compare it to the one it replaced, expect a lower hallucination rate — and it works for exactly zero of the five providers checked here. OpenAI and Anthropic held flat. DeepSeek got slightly worse. Google nearly doubled its rate and xAI more than tripled it. On the benchmark built specifically to make this comparison fair, 'newer is safer' does not survive contact with the data.

The sharper story sits inside that result, not around it: reasoning is the real driver. Strip out everything except reasoning-versus-standard pairs from the same provider, and the gap runs up to 3.4x higher, confirmed independently by OpenAI's own published system card. That's the part of this myth-bust that should worry anyone deploying these models for anything involving names, dates, or citations — and it's the part OpenAI itself says it can't yet explain.

The one thing this data doesn't support is the flip side of the myth — that AI hallucination is now uniformly worse, full stop. GPT-4.5 improved. Anthropic's flagships barely moved either direction. What changed is specifically tied to reasoning-model releases, on specific benchmarks, and any claim that skips naming which benchmark it's quoting is telling less than the full story.