What are AI hallucinations? Why LLMs make things up

Key takeaways

• An AI hallucination is a fluent, confident answer from a language model that is false, made up, or not supported by the input it was given.
• OpenAI researchers argued in September 2025 that models hallucinate because pretraining cannot separate rare facts from plausible fiction, and because most benchmarks reward a guess over "I don't know".
• In one OpenAI comparison on the SimpleQA test, a model that abstained 52% of the time made errors on 26% of questions, while an older model that almost never abstained was wrong on 75%.
• NIST's Generative AI Profile (NIST AI 600-1, July 2024) calls the problem "confabulation" and lists it among 12 risks specific to generative AI.
• No technique removes hallucinations completely. Retrieval (RAG), required citations, letting the model abstain, and regular evaluation reduce them, and human checking is still needed for high-stakes use.

What are AI hallucinations? They are fluent, confident answers from language models such as ChatGPT, Claude or Gemini that are false, made up, or unsupported by their input. A September 2025 OpenAI paper traces them to how models are trained and scored.

This explainer covers what counts as a hallucination, why large language models (LLMs) produce them, the main types researchers recognise, and the practical methods that reduce them in real applications.

What are AI hallucinations? A working definition

There is no single legal or technical definition, but the main reference documents agree on the core idea. The survey by Lei Huang and colleagues, first posted in November 2023 and later accepted by ACM Transactions on Information Systems, describes the problem as LLMs producing plausible yet nonfactual content. The OpenAI paper "Why Language Models Hallucinate" (Adam Tauman Kalai, Ofir Nachum, Santosh Vempala and Edwin Zhang) defines hallucinations as plausible but incorrect statements a model makes when it guesses under uncertainty.

The US National Institute of Standards and Technology prefers a different word. Its Generative AI Profile, published on 26 July 2024 as a companion to the AI Risk Management Framework, uses "confabulation": confidently stated content that is erroneous or false, which the document notes is colloquially known as hallucination or fabrication. NIST treats it as a risk because users may be misled or deceived, especially when they rely too heavily on the system.

Two features separate a hallucination from an ordinary mistake. First, the output is fluent and confident, so it does not look like an error. Second, the model has no built-in signal telling the reader which sentences are grounded and which were filled in.

Why do LLMs make things up?

1. Next-token prediction learns patterns, not a list of facts

An LLM is pretrained to predict the next token (a word or word fragment) across a very large body of text. That objective teaches spelling, grammar and common facts well, because those patterns repeat thousands of times. It is much weaker on facts that appear rarely. The OpenAI authors use a person's birthday as the example: nothing in the surrounding text lets a model infer a random date, so if the fact was seen once or not at all, the model has no pattern to learn.

The paper puts a number on this with the "singleton rate", the share of facts that appear exactly once in the training data. Its illustration: if 20% of birthday facts appear only once in pretraining data, a base model should be expected to hallucinate on at least about 20% of birthday questions. The authors also prove that generating valid text is at least as hard as classifying whether a given statement is valid, and that the generative error rate is roughly at least twice the classification error rate. In short, some hallucination is a statistical consequence of training, not a rare bug.

The paper gives a concrete case. Asked for Kalai's birthday in DD-MM format and told to answer only if it knew, the open-weight model DeepSeek-V3 gave three different wrong dates across three attempts. Asked for the title of his PhD dissertation, three popular chatbots each produced a different invented title, and none was correct.

2. Benchmarks reward guessing

Pretraining explains where errors come from; post-training and evaluation explain why they persist. Most leaderboards grade answers as simply right or wrong. Under that scoring, "I don't know" earns zero, the same as a wrong answer, so a model that guesses will always score at least as well as one that admits uncertainty. The authors compare it to a multiple-choice exam with no penalty for wrong answers: the best strategy is to never leave a question blank.

They reviewed 10 widely used benchmarks. Nine use binary grading as the main metric, and none gives full credit for an honest abstention. The tenth, WildBench, gives partial credit, but its rubric can still score "I don't know" below a fluent answer that contains errors.

3. Generation choices add risk

The Huang survey lists further causes beyond training data: outdated or long-tail knowledge, fine-tuning on facts the model never really learned, a tendency to agree with the user (sycophancy), sampling randomness at decoding time, and failures in multi-step reasoning. Long prompts add another risk, because models can lose track of material buried in the middle of a long context window and then fill the gap with something plausible.

What the SimpleQA numbers show

OpenAI's 5 September 2025 post on the research illustrates the trade-off with SimpleQA results taken from the GPT-5 System Card. Accuracy alone makes the older model look slightly better; the error rate tells a different story.

Source: OpenAI, "Why language models hallucinate" (5 September 2025), citing the GPT-5 System Card. Figures as published by OpenAI.

The lesson for anyone reading a leaderboard: a model can gain a couple of accuracy points by guessing more, while tripling the number of confident wrong answers its users will see.

Types of AI hallucination and how to reduce each

The Huang survey splits hallucinations into two families. Factuality hallucinations conflict with real-world facts. Faithfulness hallucinations diverge from the user's instructions or the supplied context, or contradict themselves. The table maps each subtype to the mitigations that research and standards bodies point to.

How to reduce AI hallucinations in practice

Ground answers with retrieval (RAG)

Retrieval-augmented generation fetches relevant passages from a document store and places them in the prompt, so the model answers from evidence instead of memory. In the 2020 paper that named the technique, Patrick Lewis and colleagues reported that human raters judged RAG output more factual than a comparable BART model in 42.7% of comparisons, against 7.1% the other way. Retrieval quality depends on good vector embeddings and sensible chunking; our step-by-step guide shows how to build a RAG app in Python. The Huang survey also warns that RAG has limits: if retrieval returns the wrong passage, or the model ignores it, errors still get through.

Require citations, then check them

Asking for a source for every claim makes errors easier to spot, but a citation is only useful if someone opens it. Fabricated references are among the most common hallucinations. In Mata v. Avianca, a federal judge in the Southern District of New York imposed a $5,000 sanction on 22 June 2023 after a legal brief cited court decisions that ChatGPT had invented. The judge wrote that using AI was not in itself improper, but that existing rules give lawyers a gatekeeping role over the accuracy of their filings.

Make abstention acceptable

The OpenAI authors propose stating an explicit confidence target in the instructions: answer only if more than t confident, with a wrong answer costing t/(1−t) points, a correct one earning 1, and "I don't know" earning 0. At t = 0.75, a wrong answer costs 2 points; at 0.9, it costs 9. Guessing then pays only when the model is more likely right than the threshold. Application builders can apply the same idea in system prompts and in how they score their own tests. Well-written prompt engineering helps, but it does not replace grounding.

Measure hallucinations with evaluation

NIST's profile pairs each generative AI risk, confabulation included, with suggested actions for measuring and managing it. For a production app, that means a fixed test set of questions with known answers, scoring that counts wrong answers separately from abstentions, and checks that each answer is supported by the retrieved context. Our tutorial on how to evaluate an LLM app in Python walks through building such a test harness.

Keep a human in the loop where it matters

For medical, legal, financial or news content, treat model output as a draft. NIST notes that the harm from confabulation grows with overreliance, so review effort should match the stakes.

What it means

Hallucinations are not a temporary glitch that the next model release will fix. The OpenAI research argues that some rate of error follows from the statistics of pretraining, and that the industry's own scoreboards have pushed models to bluff. The authors' proposed fix is socio-technical: change how mainstream benchmarks score uncertainty, so that models are rewarded for knowing what they do not know.

For users, the practical rule is simple. Trust a model most when it is working from material you gave it or it retrieved, and least when it recalls rare specifics such as names, dates, figures and citations from memory. For builders, the combination of retrieval, citations, permission to abstain and continuous evaluation is the current best practice. NIST's framework gives organisations a common vocabulary for documenting that risk.

Sources

• OpenAI: Why language models hallucinate (5 September 2025)
• Kalai, Nachum, Vempala, Zhang: Why Language Models Hallucinate (arXiv:2509.04664)
• NIST: AI RMF Generative Artificial Intelligence Profile (NIST AI 600-1), publication page
• NIST AI 600-1 full text (PDF)
• Huang et al.: A Survey on Hallucination in Large Language Models (arXiv:2311.05232)
• Lewis et al.: Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks (arXiv:2005.11401)
• LawSites: Court imposes sanctions in Mata v. Avianca (June 2023)