Why AI gets things wrong: hallucinations, bias and limits
Why fluent answers can be false, where bias comes from, the common failure patterns of language models, and a simple routine for checking AI output before you rely on it.
🎯 By the end of this lesson you will be able to:
- Explain why language models hallucinate
- Recognise the most common failure patterns in AI output
- Apply a verification routine matched to how risky the task is
In this lesson
Fluent is not the same as true
In Lesson 3 you saw that a language model predicts plausible next tokens. Plausible usually lines up with true, because most of the text it learned from is accurate. But the model has no built-in fact checker. When it does not know something, it does not stay silent by default: it produces the most plausible-sounding continuation, which can be completely made up.
This is called a hallucination: output that sounds confident and specific but is false or unsupported. Hallucinations are not a bug in one product; they come from how these models work, and every team using generative AI has to plan for them.
A typical hallucination
Give me three academic papers about AI in airport baggage handling, with authors and years.
The model may return three neat references with realistic titles, author names and years, some or all of which do not exist.
Specific details such as citations, statistics, quotes, URLs, version numbers and legal or medical facts are where hallucinations do the most damage.
Common failure patterns
| Pattern | What it looks like | Typical cause |
|---|---|---|
| Made-up facts | Invented sources, numbers or features | No reliable knowledge, still produces plausible text |
| Outdated answers | Old prices, policies or versions | Knowledge cut-off |
| Missed details | Ignores part of a long document | Very long context, unclear instructions |
| Maths and counting slips | Wrong totals or dates | Text prediction is not calculation |
| Agreeing too easily | Changes a correct answer when you push back | Trained to be helpful and agreeable |
| Following hidden instructions | Obeys text inside a pasted document | Cannot reliably tell instructions from data (Lesson 6) |
Bias: learning our patterns, good and bad
Models learn from human-written text and human decisions, so they can repeat human biases, for example assumptions about gender in certain jobs, or giving poorer answers in some languages and dialects. Model providers work to reduce this, but it never goes away completely. Bias matters most when AI influences decisions about people, such as hiring, lending, grading or support priority.
How teams reduce errors
- Give the model the facts: paste the source document or connect it to trusted data, and ask it to answer only from that (this is the idea behind RAG, Lesson 9).
- Allow "I don't know": tell the model it is fine to say it cannot find the answer.
- Ask for sources and quotes: then check that the quotes really exist in the source.
- Use tools for calculation: let a calculator or code do maths instead of text prediction.
- Keep a human in the loop for anything important, and test the AI feature on many examples before trusting it (Lesson 11).
A verification routine you can use today
- Classify the risk: Who is affected if this is wrong, and how badly?
- Check the specifics: every name, number, date, quote, link and claim that matters.
- Check against the source, not against another AI answer.
- For high-risk work, have a second person review, and keep a record of what you checked.
What this means for your role
Developer
AI-written code can call functions that don't exist, use outdated APIs or contain security holes. Read it, run it, test it, and never paste secrets into prompts.
Tester / QA
Hallucination testing is a new core skill: ask questions with no answer in the source, mix in misleading context, and check whether the system admits uncertainty.
Business Analyst / PO
Define what "correct" means for each AI feature and what must happen when the AI is unsure, for example hand over to a person or say it doesn't know.
Tech Lead / Architect
Design for failure: grounding in trusted data, output validation, human review for risky actions, and logging so problems can be investigated.
Project / Delivery Manager
Put a risk rating on each AI use case. High-risk uses need review steps, sign-off and more testing time in the plan.
Scrum Master / Agile
Encourage people to share AI mistakes openly in retrospectives. A team that talks about failures learns safe habits faster.
Try it: catch a hallucination
- Ask an AI assistant for five statistics with sources on a topic you know well.
- Check two of them against the original source. Note anything invented or misquoted.
- Ask again, this time adding: "If you are not sure, say so, and only give sources you are confident exist." Compare the answers.
Key takeaways
- Language models produce plausible text, not verified facts, so they can hallucinate confidently.
- Specific details such as citations, numbers, dates and legal or medical facts need checking.
- Bias comes from training data; take extra care when AI affects decisions about people.
- Give the model sources, allow it to say "I don't know", match checking to risk, and keep humans accountable.
Key terms
- Hallucination
- Confident, specific AI output that is false or not supported by any source.
- Bias
- Systematic unfairness or skew in outputs, often learned from training data.
- Human in the loop
- A design where a person reviews or approves AI output before it has an effect.
Check your understanding
Answer all 5 questions. Score 3 or more to complete the lesson.
1. Why do language models hallucinate?
2. Which AI output most needs checking before use?
3. Where does bias in AI models mainly come from?
4. Which technique most directly reduces made-up answers about your company's policies?
5. An AI drafted a customer reply containing a wrong refund amount. Who is accountable?