AI can sound incredibly confident, even when it is completely wrong.
That is one of the biggest challenges when it comes to understanding AI hallucinations.
A chatbot might give you an answer that sounds polished, professional and convincing, while the information behind it is completely inaccurate. And sometimes, the problem starts before the AI even generates its answer.
The information we put into AI matters too.
What is an AI hallucination?
An AI hallucination is when a generative AI system produces information that is inaccurate, misleading or made up, while presenting it as though it is correct.
This can be particularly difficult for students to recognise because AI-generated answers often sound authoritative.
The problem isn't necessarily that the AI is trying to deceive us. Generative AI systems produce responses based on patterns in the data they were trained on and the context of the conversation. They can generate plausible-sounding information without actually understanding whether that information is true.
That means an answer can sound right without being right.
And that is where critical thinking becomes so important.
AI can be confidently wrong
One of the most useful ways to teach students about AI hallucinations is to show them how convincing an incorrect answer can sound.
Sometimes the mistakes are obvious and funny. Other times, they're much harder to spot.
Imagine asking an AI chatbot a question based on a false assumption. Instead of challenging the premise, the chatbot may accept the information in the prompt and build an answer around it.
The result can sound perfectly reasonable.
This is why students can't simply judge an AI response by how confidently or professionally it is written.
A convincing answer still needs to be checked.
We need to talk about the prompts we put into AI
Conversations about AI hallucinations often focus on what the AI gets wrong.
But there is another part of the conversation we need to have with students:
What are we putting into the AI in the first place?
If we give an AI chatbot a prompt containing misinformation, an incorrect assumption or something completely absurd, we shouldn't automatically expect it to correct us.
This is particularly important as students increasingly use AI to find information, explain concepts, brainstorm ideas and answer questions.
Students need to understand that the quality of the conversation doesn't depend entirely on the AI.
The information, assumptions and questions they bring to the conversation matter too.
Are we fact-checking ourselves before asking AI?
This is an interesting question for students to consider.
Before putting something into an AI chatbot, ask:
Do I know that this information is actually true?
Am I making an assumption?
Where did this information come from?
Could there be another explanation?
Am I asking the AI to confirm something I already believe?
What would happen if my prompt was wrong?
This changes the way students think about prompting.
Instead of treating prompting as simply knowing the right words to get the best answer, students start thinking about the information and assumptions behind their prompts.
That is a much more useful AI literacy skill.
AI literacy isn't just about getting better answers
There is a tendency to talk about AI literacy as though it is mainly about learning how to use AI tools effectively.
Knowing how to write a good prompt can certainly be useful.
But students also need to understand the limitations of the technology.
They need to know that AI can:
produce incorrect information
accept false assumptions
sound confident when it is wrong
reflect problems in the information or context provided
give different responses to similar questions
require human judgement and verification
Most importantly, students need to develop the habit of questioning the answer rather than automatically trusting it.
Critical thinking still matters
AI doesn't remove the need for critical thinking. If anything, it makes it more important.
Students are going to encounter AI-generated information in search results, social media, schoolwork, videos, images and everyday conversations.
They need to be able to stop and ask:
Does this actually make sense?
How do I know it's true?
What evidence supports it?
Could the AI be wrong?
Do I need to check this somewhere else?
These are skills that extend far beyond AI.
They are the same critical thinking skills students need to navigate misinformation, social media and the wider online world.
Teach students to question both the answer and the prompt
One of the most valuable lessons we can teach students about AI is that they shouldn't outsource their thinking to the technology.
That means questioning the answer.
But it also means questioning the question.
Before trusting what AI tells us, we should consider whether the information we gave it was accurate in the first place.
AI literacy isn't simply about learning how to get AI to produce better answers.
It's about understanding when to question the technology, when to check the information and when we need to think for ourselves.
That's the kind of critical thinking students will need as AI becomes an even bigger part of their everyday lives.
Want to teach students how to question AI?
Paris McNeil works with schools across Perth and Western Australia to help students build practical AI literacy, critical thinking and online safety skills.
Her sessions help students move beyond simply learning how to use AI and start thinking more critically about what AI produces, what they put into it and when they should question the result.
