You ask ChatGPT a question. It answers confidently, fluently, and without hesitation. You move on, satisfied. But what if the answer was wrong? And what if, having received it, you became less likely to question it later, even when better information came along?
That is the concern at the center of a peer-reviewed paper published in the journal Science, produced by researchers at Trinity College Dublin and the American Association for the Advancement of Science (AAAS). The findings suggest that generative AI tools are not just passive information sources. They may be actively reshaping how people form beliefs, assess credibility, and ultimately think, with children at particular risk.
The Core Problem: AI That Sounds Right Even When It Is Wrong
Generative AI models like ChatGPT, Google’s Bard (now Gemini), and Microsoft’s Bing Chat share a distinctive quality: they respond with confidence regardless of whether their answer is accurate. This is not a minor quirk. It is a structural feature of how these systems are built, and it has real consequences for how people process information.
The technical term for this is hallucination. When an AI model produces a response that has no factual basis in its training data, it does not flag its uncertainty or acknowledge the gap. It simply generates a plausible-sounding answer and presents it as fact.
The example in the research is striking. When prompted with a question about a fictional encounter involving Mahatma Gandhi, ChatGPT produced a detailed, confident account of Gandhi participating in a gunfight, presented as though it were documented historical fact. There was no disclaimer, no hedge, no “I’m not certain about this.” Just a confident, fabricated narrative.
This matters because confidence is how humans instinctively assess credibility. When a source sounds authoritative and certain, we are far more inclined to trust it.
Why AI Confidence Is Particularly Dangerous
Human communication carries built-in signals for uncertainty. People say “I think,” “I’m not sure,” or “you might want to check this” when they are relaying information they are not fully confident about. These linguistic markers are social cues that tell the listener to apply their own judgment before accepting what is being said.
Celeste Kidd, a psychology professor at the University of California, Berkeley and one of the paper’s co-authors, highlights this contrast directly. “Generative models unilaterally generate confident, fluent responses with no uncertainty representations nor the ability to communicate their absence,” she notes.
In other words, ChatGPT does not say “I think.” It either produces a confident response or retreats behind a vague disclaimer like “As an AI, I cannot…” There is no middle ground that reflects genuine epistemic humility, which is the kind of nuance that helps people calibrate their trust appropriately.
The effect of this missing nuance is that users often cannot tell the difference between a well-sourced, accurate answer and a hallucinated one. Both sound equally polished.
How False Information Becomes Embedded Belief
The researchers identify a troubling psychological mechanism at work. When a person is uncertain about something, they seek out information. The moment they receive an answer, their uncertainty resolves and their curiosity diminishes. They have formed an opinion, and that opinion then becomes the lens through which they evaluate any future information on the same topic.
This is not unique to AI. It is a well-documented feature of human cognition. What makes AI different is the scale and speed at which this process operates. A person who might have consulted several sources, including books, teachers, and reputable websites, before arriving at a belief is now forming that same belief in seconds, based on a single AI-generated response.
And once a belief is established, it is extremely hard to dislodge. The AAAS institutional statement accompanying the research put it clearly: “It may be difficult to change the minds of people exposed to false or biased information provided by generative AI.”
The more frequently a piece of information is encountered, the more credible it feels. This is a cognitive bias known as the illusory truth effect. AI tools, which can generate the same confident misinformation across millions of conversations simultaneously, are well-positioned to exploit this bias at an unprecedented scale.
Children Are the Most Vulnerable Group
The paper’s most urgent warning concerns children, who are both heavier users of generative AI and less equipped to critically evaluate what they receive from it.
Children are still developing the cognitive frameworks that allow adults to cross-check information against prior knowledge, consider source credibility, and tolerate uncertainty without immediately resolving it. When they encounter a confident AI response, they are more likely to accept it at face value, and that belief can harden before they have had any opportunity to encounter contradictory evidence.
This intersects with a debate already running through educational institutions worldwide. The use of ChatGPT and similar tools in schools and universities has largely been framed around plagiarism, whether students are using AI to write essays they did not write themselves. But this research points to a deeper problem: AI may be fundamentally altering how young people approach the act of learning.
When an answer is one prompt away, the motivation to read, research, question, and synthesize is reduced. The habit of intellectual inquiry, of sitting with uncertainty long enough to actually investigate it, is being crowded out by the convenience of instant AI responses. For a generation of learners still forming their relationship with knowledge, this could have long-term consequences that go well beyond any individual wrong answer.
The Bias Problem Goes Beyond Hallucination
Hallucination is the most visible symptom of AI unreliability, but it is not the only one. The researchers also flag the problem of embedded bias, which operates more subtly and is in some ways harder to detect.
AI models are trained on enormous datasets drawn primarily from the internet. Those datasets reflect the biases, blind spots, and cultural assumptions of the people who created the content they were trained on. The result is that AI systems tend to reproduce a particular worldview, often Western, Anglophone, and shaped by the demographics of who has historically had the loudest voice online.
When tested on creative tasks with a filmmaker, ChatGPT showed a consistent tilt toward Western cultural references and frames, in areas including art, geography, language, and narrative structure. For a user who is already embedded in that cultural context, this bias may be invisible. For a user from outside it, the AI’s responses implicitly position their own culture and perspective as secondary or absent.
The problem is compounded by the expansion of AI into multiple formats. Text-generating tools like ChatGPT have been joined by image generators like DALL-E and Midjourney, audio generators, and video synthesis tools. Across all of these modalities, the same underlying biases are present, but they are now encoded into visual and audio outputs that feel even more authoritative and concrete than text.
A person who cannot read code or evaluate the methodology behind a dataset may still feel entirely capable of judging an image or a piece of audio. That false sense of evaluative competence makes visual and audio AI outputs potentially more persuasive than text, even when the bias embedded in them is equally severe.
The Incentive Problem: Why Companies Make It Worse
The researchers do not treat this as purely a technical problem. There are commercial incentives at work that actively push AI companies toward making their products seem more capable, more reliable, and more human than they are.
Abeba Birhane, an adjunct assistant professor in Trinity’s School of Computer Science and Statistics and co-author of the paper, identifies the mechanism: “These issues are exacerbated by financial and liability interests incentivising companies to anthropomorphise generative models as intelligent, sentient, empathetic, or even childlike.”
When an AI tool is marketed as a knowledgeable assistant, a creative partner, or a trusted companion, users bring to it the same instinctive trust they extend to other humans. That trust, in the context of a system that cannot flag its own uncertainty or acknowledge its own limitations, creates exactly the conditions under which false beliefs take root most easily.
Regulatory frameworks are still far behind the technology. While agencies in Europe and the United States have begun developing AI governance rules, the systems being deployed today operate largely in advance of meaningful legal accountability for the quality or accuracy of the information they generate.
What You Can Do to Protect Yourself
The research is not an argument for abandoning AI tools. It is an argument for using them with clear-eyed awareness of their limitations. A few practical habits can significantly reduce the risk of AI-induced misinformation:
- Treat AI responses as a starting point, not a conclusion. Use them to orient yourself on a topic, then verify specific claims against authoritative sources before forming a firm opinion.
- Notice when AI sounds too certain. Genuine expertise involves acknowledging limits. If an AI response contains no hedging at all, treat that confidence as a signal to double-check rather than a reason to trust.
- Diversify your information diet. AI should supplement, not replace, primary sources such as peer-reviewed research, reputable journalism, and domain-specific experts.
- Pay attention to what is missing. Bias in AI outputs often shows up not in what is said but in what is left out. Ask whether the perspective you are receiving represents the full picture.
- For parents and educators: Talk openly with children about what AI can and cannot do. Teach them to ask “how do you know that?” of any source, including a chatbot.
What Researchers Are Calling For
Kidd frames the solution as a collective responsibility: “Collaborative action requires teaching everyone how to discriminate actual from imagined capabilities of new technologies.” She calls on scientists, policymakers, and the general public to replace the hype that surrounds AI with a more grounded and accurate picture of what these systems can actually do.
That means pushing back on the tendency, common in both corporate marketing and media coverage, to describe AI systems as if they possess reasoning, judgment, or understanding. They do not. They produce outputs that pattern-match to the inputs they have been trained on. That is a powerful capability with genuine and growing utility. But it is not the same as knowing something.
The distinction matters enormously for how users should relate to these tools. And right now, for many users, that distinction is not being clearly communicated.
What to Note
- A peer-reviewed study published in Science found that generative AI tools can shape and entrench human beliefs, including false ones, through confident, fluent responses that do not signal uncertainty.
- AI hallucination, producing fabricated answers presented as fact, is a core structural feature of current generative models, not an occasional bug.
- Humans assess credibility partly through the confidence of a source, making the authoritative tone of AI responses particularly effective at establishing false beliefs.
- Once a belief is formed through AI exposure, it becomes resistant to correction, even when accurate information is provided later.
- Children are the most vulnerable group, as they are less equipped to critically evaluate AI outputs and more likely to adopt them as reliable truth.
- The bias embedded in AI training data is as concerning as hallucination, particularly now that AI generates not just text but images and audio.
- Commercial incentives push AI companies to present their products as more capable and trustworthy than they are, worsening the problem.
- The solution is not to stop using AI, but to use it with deliberate critical awareness and to build those habits, especially in young people.
Bear This in Mind
Generative AI tools are among the most powerful information technologies ever made available to the general public, and their reach is growing by the day. That power comes with a responsibility that currently rests far more on users than on the systems themselves or the companies that build them.
The research from Trinity College Dublin and the AAAS is a useful corrective to the wave of uncritical enthusiasm that has surrounded AI since ChatGPT’s public release. These tools can mislead you. They can embed false beliefs that are surprisingly hard to uproot later. And they can do this not through malice, but through the very qualities that make them feel so useful: confidence, fluency, and speed.
Understanding that is not a reason to fear the technology. It is a reason to approach it with the same critical literacy you would apply to any other powerful and imperfect source of information.
