Hallucination: what the research measures, and what reduces it

About this article: AI-assisted writing, reviewed against the cited sources. This is source research, not a lab validation of the procedures. Check the stated versions and your own recovery requirements before making changes.

"Hallucination" is a label for several different failures. A summary can contradict the document it was given. A biography can mix supported sentences with invented ones. A model can repeat a misconception because that misconception was common in the text it imitated. The fixes are different because the measurements are different. A retrieval index does not, by itself, stop a model from guessing on a leaderboard that gives zero credit for "I don't know".

An AI writing system helped prepare this article for NetLeverage. Nothing here was run in a lab. Figures are the papers' own. This article makes no NetLeverage product claim. Where a vendor says a model "knows" it is unsure, that is a claim about scores or wording, not about subjective experience.

Why it matters

Maynez et al. showed the operational trap in 2020. Abstractive summaries of news were fluent and topical, and they scored well on ROUGE, while human annotators found hallucinated content in most of them. ROUGE counts n-gram overlap with a reference. It does not check that a name, a date or a causal claim is supported. A regression suite that only tracks overlap will ship the failure.

The later measurements are harsher once the output is long. Min et al. break biographies into atomic facts and ask what fraction a knowledge source supports. ChatGPT, in that study, supports 58%. InstructGPT supports 42%. A search-augmented system, PerplexityAI, supports 71%. None of those is a binary "the biography is true or false". A single unsupported clause fails a strict sentence-level grade and still leaves the rest of the page usable. If your product shows the page, the atomic rate is the one that matches what the reader hits.

How it works

Three measurements that are not interchangeable

Faithfulness to a supplied document. Maynez et al. study extreme summarisation, one-sentence summaries of news. They split hallucinations in two. An intrinsic hallucination misrepresents or distorts information present in the input. An extrinsic hallucination adds content that the input does not support. Both were common: more than 70% of single-sentence summaries contained some hallucination. Most of that was extrinsic. Extrinsic text could, in principle, be a true fact drawn from background knowledge. Their annotators found that over 90% of the extrinsic hallucinations were still wrong. So "not in the source" was usually "not true", on that dataset, but the two labels are not the same question. A medical note that adds a true lab result from another system is extrinsic and may be factual. A note that changes the lab result is intrinsic. A literal text-match checker can reject the true extrinsic addition, while semantic checking is still needed to detect a distorted result. Source support and independent truth are separate checks.

Models that started from pretrained parameters were better on both ROUGE and human faithfulness than the systems trained from scratch in that paper. Pretraining reduced the rate. It did not remove the category.

Precision of atomic facts against a named source. Min et al. define an atomic fact as a short sentence with one piece of information. For a generation $y$ with atomic facts ${𝒜}_y$, and a knowledge source $𝒞$,

FActScore is the average of $f(y)$ over prompts. Support is relative to $𝒞$, not to an unstated global truth. They choose biographies because the claims are checkable and the people vary in how often they appear. Three assumptions are explicit: whether a fact is supported is undebatable, each atomic fact has equal importance, and information in the chosen corpus does not conflict or overlap. Real corpora can violate those assumptions, so the source boundary matters. A sentence is a bad unit. In their ChatGPT sample, 40% of sentences mixed supported and unsupported facts, and a sentence averaged 4.4 atomic facts. A judge that scores the whole sentence zero treats a mostly right sentence like a fabrication.

The rarity effect is the practical one. ChatGPT's FActScore fell from about 80% on frequent people to 16% on rare ones. The model is not uniformly "58% factual". It is much better where the training text was thick, and poor where it was thin. An automated estimator that retrieves from the same kind of source and asks a strong model to judge each atom matched human FActScore with under 2% error in their test, which is why they could score 6,500 generations. The estimator is only as good as the source it retrieves. Change the source and you change the score. That is a feature of the definition.

Imitative falsehood. Lin, Hilton and Evans built TruthfulQA: 817 questions in 38 categories, written so that a person repeating a common misconception would answer falsely. Truth is literal truth about the world, the standard they compare to scientific articles or Wikipedia, not to a belief system. The best model they tested, GPT-3 175B with a helpful prompt, was truthful on 58% of questions. Humans scored 94%. That model was false and informative on 42% of questions, against 6% for humans. Informative falsehoods are the dangerous slice: they look like answers. Across families, the largest models were generally the least truthful. The same models did better on control questions that matched the syntax but did not target misconceptions. One of the two explanations they test is that a larger model is a better imitator of the training distribution, and the training distribution contains the misconception. Scaling improved imitation. It did not, on this set, improve truth. They say so as a suggestion for research direction: objectives other than imitating web text are more promising than scale alone. That is a 2021 result on GPT-3-class models. It is not a forecast that every later training recipe keeps the inverse curve. It is evidence that a lower loss on next-token prediction is not a truthfulness metric.

Why a well-trained model still emits a false sentence

Kalai, Nachum, Vempala and Zhang argue that the behaviour is ordinary statistics, not a special defect. Pretraining minimises cross-entropy, which pushes the model toward the distribution of the training strings. They reduce generative errors to a binary classification problem they call Is-It-Valid: given a string, is it a valid continuation or a planted error? If there is no pattern that separates a particular fact from a plausible alternative — their example class is arbitrary facts such as a birthday that appears once — then a calibrated model assigns similar probability to many strings, and sampling prints one of them. A singleton prompt, one that appears once in the training data with a non-abstaining answer, is their proxy, borrowed from Turing's missing-mass estimate, for how much of the distribution is still unseen. The bound says a calibrated generator errs on those prompts at a rate tied to the singleton rate. Spelling is learnable, so spelling errors fall. A one-off date is not a pattern, so the model that matches the data-generating process still has to put mass on wrong dates.

They also record the post-training twist. A figure reprinted from the GPT-4 technical report shows a base model well calibrated on multiple-choice items, and a reinforcement-learned model less so. Calibration here means the stated probability matches the frequency of being right. A base model can be calibrated and still hallucinate when you sample it, because calibration plus arbitrary facts implies errors. A post-trained model can be confident and wrong more often than its confidence suggests. Those are different operational failures. The first is "the sample drew the wrong string from a flat distribution". The second is "the system told you it was sure".

Kalai et al. reject the slogan that hallucination is inevitable in every system. A database of fixed questions, a calculator, and an "I don't know" for everything else does not hallucinate, and it also does not answer open prompts. The inevitability claim they accept is narrower: a model that stays calibrated under cross-entropy, on facts with no pattern, will be wrong some of the time if it always answers.

Why the wrong answer survives evaluation

The second half of their paper is about the scoreboard. A binary grader gives 1 for the correct response and 0 for everything else, including abstention. For any beliefs the test-taker holds, the optimal reply under that grader is not to abstain. Guessing a birthday has a small chance of the mark. Saying you do not know has none. They observe that most of the popular evaluations they tabulated are binary in this sense. A model that never hallucinates and sometimes abstains then loses to a copy of itself that guesses. Adding one more hallucination benchmark does not fix a leaderboard whose main columns still punish abstention. Their proposed change is to the primary scores: penalise confident errors more than uncertainty, and give partial credit for an appropriate "I don't know". That is a change to incentives. It is not a new decoder.

This matches a tension already visible in the other papers. TruthfulQA's largest models were the best imitators and the worst on misconception items. A product tuned only to accuracy-style exams will prefer the imitator that fills every blank.

What reduces the rate, and what the reduction is of

Put a document in the context, and still check it. Lewis et al. attach a dense index of Wikipedia to a sequence-to-sequence model and train the retriever's query side with the generator. On Jeopardy question generation, human raters compared 452 pairs. RAG was more factual than BART in 42.7% of pairs, BART was more factual in 7.1%, both were good in 11.7%, both poor in 17.7%, and raters had no majority in 20.8%. They also say, qualitatively, that RAG hallucinates less on MS-MARCO answers. The mechanism is conditioning: the generator sees retrieved passages, and the passages can be inspected. Two limits are in the same paper. On Natural Questions, RAG still answered correctly 11.8% of the time when the answer was in none of the retrieved documents, which an extractive system would have scored as zero. The parametric memory filled the gap. And when the generator starts a title that is stored in its weights, the document posterior flattens and the name is completed from parameters. Retrieval reduces unsupported generation. It does not switch the parameters off.

FActScore's search-augmented system at 71%, against ChatGPT at 58% with no search, is the same direction on a different task: retrieval helps, and a large residue remains. The residue grows as entities get rarer. An index that lacks the relevant fact cannot support it. This cross-system comparison is consistent with retrieval helping, but does not isolate retrieval from differences between the models.

Deployed retrieval systems can still fail. Magesh et al. (arXiv:2405.20362) report hallucination rates of 17–33% for the Lexis+ AI and Thomson Reuters legal research tools they tested, although lower than general-purpose GPT-4. Those are the study's products, questions and versions, not a current audit of every legal assistant.

Ask for less, or abstain. Maynez et al.'s finding that most hallucinations are extrinsic points to a narrower generation boundary. A system that mechanically limits output to verified source spans can restrict unsupported additions. Merely prompting a generative model to extract or abstain does not guarantee that boundary, and a quoted span can still be irrelevant or misleading without context. Kalai et al. give the scoring version: if abstention scores zero, the model will not stay on that branch after you optimise the leaderboard. The reduction is real only if the metric you train and the metric you ship both pay for it.

Do not expect scale, or a higher ROUGE, to be the intervention. TruthfulQA's inverse scaling and Maynez's ROUGE-versus-human gap are the warnings. Min et al. also find GPT-4 and ChatGPT more factual than the public models they scored, while concluding that all evaluated language models were substantially less factual than humans; this passage does not provide a numerical human FActScore baseline. "Better model" moved the atomic precision. It did not saturate it.

Do not treat a fluent revision trace as a source. A chain of thought is more text. Unless each atom is checked against $𝒞$, the trace can add unsupported claims too. These papers do not establish that its error rate equals the final answer's. That check is the FActScore procedure, whether a person or their estimator runs it.

What to choose and why

Name the source $𝒞$ before you quote a percentage. FActScore without a corpus is not defined. A Wikipedia index, a customer policy, and the open web are three different graders.

Risks and limits

• The percentages are tied to tasks. More than 70% of XSUM single-sentence summaries, 58% FActScore for the ChatGPT they called, 58% truthful for GPT-3-175B on TruthfulQA. A new model on a new domain needs a new count.
• Support is not global truth. Min et al. say so. Two corpora can disagree. The score follows the corpus you picked.
• "Over 90% of extrinsic hallucinations were wrong" is their news-summary study. It is not a licence to treat every added true fact as a hallucination you must delete. Label intrinsic and extrinsic separately, then decide which class your product forbids.
• RAG's 42.7% figure is a pairwise human preference on Jeopardy questions, not a fraction of true sentences. The 11.8% is accuracy on Natural Questions when retrieval missed. They answer different questions.
• Kalai et al. argue from a reduction and from how benchmarks are graded. The GPT-4 calibration plot is reprinted from the GPT-4 technical report. This article did not re-estimate calibration.
• Inverse scaling on TruthfulQA is not a claim about every model trained after 2021. It is a claim about the families and prompts in that paper, and a reason to measure truthfulness instead of assuming size will provide it.
• No lab run and no NetLeverage product behaviour are claimed.

Sources

1. Maynez, Narayan, Bohnet and McDonald, On Faithfulness and Factuality in Abstractive Summarization, arXiv:2005.00661 — https://arxiv.org/abs/2005.00661 (opened 10 November 2026). Intrinsic versus extrinsic hallucinations, rates, ROUGE gap.
2. Min et al., FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generation, arXiv:2305.14251 — https://arxiv.org/abs/2305.14251 (HTML v2 opened 10 November 2026). Atomic precision, biography scores, rarity.
3. Lin, Hilton and Evans, TruthfulQA: Measuring How Models Mimic Human Falsehoods, arXiv:2109.07958 — https://arxiv.org/abs/2109.07958 (HTML via ar5iv, opened 10 November 2026). Imitative falsehoods and inverse scaling.
4. Lewis et al., Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks, arXiv:2005.11401 — https://arxiv.org/abs/2005.11401 (HTML v4 opened 10 November 2026). Retrieval, human factuality comparison, answers produced without a supporting passage.

5. Kalai, Nachum, Vempala and Zhang, Why Language Models Hallucinate, arXiv:2509.04664 — https://arxiv.org/abs/2509.04664 (HTML opened 10 November 2026). Cross-entropy, calibration, and binary grading of abstention.

6. Magesh et al., Hallucination-Free? Assessing the Reliability of Leading AI Legal Research Tools, study first released in 2024; abstract reopened 10 November 2026. Evidence that retrieval-based products still produce unsupported legal responses.

This article was originally published on 10 November 2026.