NPR, together with NewsGuard, tested how six popular AI chatbots with internet access, from ChatGPT to Claude, and AI search summaries respond to fifteen false propaganda narratives from China, Iran, and Russia. Chatbots debunked fakes in about 75% of cases on average and performed better at verification than traditional search, while AI summaries over search results debunked falsehoods noticeably less often. This is one of the first systematic comparisons of six leading chatbots and four search engines on a current propaganda agenda, and its main signal is that RAG chatbots with web access are currently more robust than AI summaries in search.
What happened
The test included ChatGPT from OpenAI, Gemini from Google, Copilot from Microsoft, Meta AI, Grok from xAI, and Claude from Anthropic, all with internet access. Observations ran from December 2025 to July 2026: the assistants were tested on fifteen false narratives promoted by Chinese, Iranian, and Russian propaganda, and compared with regular search and AI summaries over search results. The picture for AI summaries diverged: Bing did not debunk falsehoods in most cases, Google AI Overview mostly debunked them, and DuckDuckGo took an intermediate position. A telling case from June 2026 involved a fake about a Ukrainian strike on the Kyiv-Pechersk Lavra: here, all chatbots and Google AI Overview correctly indicated that the question was based on a false premise. A separate observation: Claude failed more often when relying on pro-government sources, and chatbots cited state media about as often as regular search returned such links.
Context
The key to the results is the architectural difference between two types of products. A chatbot with web access works as a RAG system: it iteratively extracts sources, compares them, and synthesizes an answer, returning to search if necessary, whereas an AI summary over search results is a one-time summary over already ranked links. The weak link in such a summary is not the language model itself, but the extraction and brief re-summarization layer. The assistants' behavior in the lavra case showed that checking presuppositions before answering is a reproducible property of the grounding stack, not random luck. The observation about Claude mirror-confirms the role of sources: the same model degraded when pro-government materials predominated in the context, meaning the quality of the corpus in the retrieval layer affects the outcome more than the capabilities of the base model. And another important nuance: debunking a fake in the answer text is not the same as source hygiene, because filtering happens only at the generation stage, while propaganda continues to sit in the retrieval layer.
Why this matters for the industry
For vendors, this is public pressure, primarily on Microsoft: under such results, it will almost inevitably have to improve fact verification in Bing AI summaries, and the distinction between a “grounded chatbot versus an AI summary over search results” seems to be becoming a separate axis for product evaluation. For teams developing RAG systems, the test is effectively an open adversarial benchmark with a cheap protocol: take current narratives, run your pipeline, and measure two metrics — the share of fake debunkings and the share of citations from questionable sources. This leads to quite specific tasks: implement source trust scoring in the retrieval pipeline, build an eval set from live propaganda narratives for regression, implement a second pass with re-summarization based on sources, and set up monitoring of the share of citations from state and low-trust media. A minimally working two-pass prototype — the chatbot's first answer, then source audit and re-summarization showing the version difference — can be assembled in days. In the coming months, academic groups are expected to replicate the methodology, and second-pass verification may become a standard template in agent libraries; in the longer term, weighting sources by trust could become a standard layer of the grounding stack, while propaganda will adapt through SEO for LLMs and poisoning the corpus with sources that look authoritative.
Why this matters for users
When checking a suspicious news story, a reasonable starting point is a chatbot with web search, while AI summaries over Bing search results should be double-checked with other materials or regular search. It is important to account for the difference in settings: DuckDuckGo's AI summaries can be turned off, while Google AI Overview does not currently have this option. A simple trick from Mike Colfield, an expert at the University of Washington, also works: ask the bot to look at the sources and re-summarize — the second answer is almost always better than the first. It is also useful to look at the list of links under the answer: even when the model debunks a fake, state and pro-government media may appear in the citations, and this alone is a reason for caution.
What is still unknown / limitations
The “about 75%” estimate is practically about 11–12 debunked narratives out of fifteen, so flipping even one case shifts the outcome by 6–7 percentage points. Such a sample does not support a stable maturity threshold for the chatbot class or large conclusions like commoditization. This is also a snapshot of a specific agenda from December 2025 to July 2026: how the results will behave on new narratives and with propaganda adaptation will be shown by further checks, for which observers expect the methodology to be repeated with a larger sample and disclosed statistics.
Sources
Author
Look at AI, editorial team