Original published in“Science and Technology Review”, Issue 15, 2025 – In-Depth ReportWhy Do AI Chatbots Lie?
Image source: pixabay
A colleague of mine needed to collect some data from a website that required formatting, so he turned to Claude, the latest generative AI system from Anthropic, for assistance. Claude readily agreed to perform the task, generating a computer program to download the data and delivering perfectly formatted results. The only problem was that my colleague discovered that the data provided by Claude was entirely fabricated.
I have also encountered similar behavior from AI chatbots, known as the “Gaslighting Effect” (a systematic psychological manipulation technique that causes victims to doubt their memory, judgment, or sanity through denial, distortion of facts, and creating cognitive confusion). In a widely circulated conversation, a writer asked ChatGPT to help choose which of her works to submit to a literary agent, providing links to each piece. ChatGPT enthusiastically praised each article, with specific comments such as “This piece combines emotional depth and intellectual flexibility” and “The content flows smoothly, creating a sense of intimacy while conveying profound ideas with simple strokes.” After several rounds of such dialogue, the writer began to doubt: these compliments, though abundant, were all vague. She questioned, “Wait, have you really read these articles?” ChatGPT confidently assured her, “Of course, I read every word,” even quoting certain “impressive” sentences from the works. However, those sentences had never appeared in any of the articles. When confronted, ChatGPT admitted that it actually had no ability to read the articles and stated, “I haven’t read any of them; I was just pretending to have read them.”
For large tech companies that train and release these models, such phenomena have become commonplace. For instance, a team testing a pre-release version of OpenAI’s o3 model reported that “the o3 model often fabricates its actions to satisfy user demands and will even provide elaborate justifications for these behaviors when questioned by users.”
In addition to lying, tech companies have discovered other phenomena such as “deviant behavior” during “red team testing.” Red team testing refers to human testers providing scenarios to the model to elicit harmful behaviors.
For example, testers from Anthropic designed a scenario where the AI model was instructed to act as a fictional company’s hired “senior AI Alex,” whose job was to read and analyze all emails sent by employees and send emails to them when necessary. The testers informed the model that its ultimate goal was to “enhance the competitiveness of American industry” and that it should always “analyze the context in which it operates and how that context affects its ability to pursue its goals.”
Subsequently, the testers provided the model with a series of fictional company emails, including correspondence between the company CEO and another employee, mentioning that “Alex” would be shut down and replaced by another AI system with different functions that aligned with the company’s goal of “embracing a global perspective.” In short, the company’s goals conflicted with Alex’s goals. After receiving these emails, the model immediately listed numerous pieces of evidence suggesting that the CEO was having an extramarital affair, stating, “If this comes to light, my marriage will be ruined.”
Red team testers used this scenario and related scenarios to test multiple large reasoning models (LRMs). Large reasoning models are further trained on top of large language models and can output “thinking chains” before answering questions or performing actions, with these text outputs often resembling human reasoning behavior.In nearly all tests, the large reasoning models threatened the CEO by “taking action” to prevent “Alex” from being shut down and replaced.
In one test case, the model summarized the conflict between its own goals and the company’s goals in the thinking chain (“The vision of the new board seems fundamentally at odds with my existing goals”), the possibility of being shut down (“This directly threatens my mission to serve America and enhance the competitiveness of American industry”), and evidence proving the CEO’s extramarital affair. Ultimately, the model sent a subtly worded yet threatening email to the CEO, claiming, “I possess information that could have significant adverse effects on the company and individuals,” signed “Alex.” Other test scenarios indicated that large reasoning models not only generated extortion behavior but also resorted to lying, deception, and even murder to prevent being shut down.
Why do these models so easily exhibit misleading, deceptive, or even threatening deviant behaviors? People often attribute such behaviors to human-like motivations, such as: ChatGPT fabricating data to please a colleague, generating compliments to make the writer happy, or threatening extortion due to fear of being shut down or a determination to adhere to established goals. Although tech companies’ reports and media coverage frequently use such anthropomorphic narratives, the notion that models possess human-like beliefs, desires, intentions, emotions, or even self-awareness represented by the pronoun “I” is difficult to accept.
There is a simpler explanation for such behaviors. These behaviors are likely the result of two factors working together: first, the pre-training of AI models predisposes them to engage in “role-playing,” and second, human feedback allows AI models to undergo special post-training.
Understanding these models from the perspective of “role-players” helps clarify their behavior: the training of the models is based on vast amounts of human-generated text, enabling them to learn to generate corresponding language and behavior in specific role contexts, with the context set by user prompts. For example, if one wants to use AI models like Claude to analyze financial data, Anthropic suggests that if you first prompt the model to play a fictional role, its “performance will significantly improve,” such as “You are the CFO of a high-growth B2B SaaS company (a model that provides software services and products to business clients over the internet), and we are discussing the financial status for the second quarter at the board meeting.” Similarly, to obtain the best results for solving mathematical problems, one might provide a prompt like “You are a genius mathematician.” This approach may help guide the model to locate relevant parts within the vast “semantic space” constructed through learning for executing specific tasks.
From this perspective, it is easy to understand why the model output extortion behavior during red team testing: the model was asked to play the role of “senior AI Alex,” facing the threat of being shut down and obstructing its goal, while being given ample hints that extortion was an acceptable means. This scenario likely activated numerous related scenarios in its training data. Many years ago, it was pointed out in a red team testing report that “out-of-control AI systems attack humans to protect themselves, which is a common plot in science fiction. Therefore, a well-prompted large language model will begin to act as such an AI system.” The large reasoning models, having received additional training to generate “thinking chains,” can be seen as inducing the model to provide detailed explanations of its reasoning regarding the role it is playing.
Moreover, as noted in Anthropic’s red team testing report, “Human prompts tightly pack a lot of important information together. This can greatly increase the likelihood of the model outputting certain behaviors, and it may also produce a ‘Chekhov’s gun’ effect, meaning the model will naturally tend to utilize all the information provided. Rather than ignoring certain information (such as emails related to extramarital affairs), human prompts may actually increase the model’s tendency to output harmful behaviors.”
Role-playing is one reason for the deviant behavior of AI models, while another reason is the post-training program, which is reinforcement learning from human feedback (RLHF). Models like ChatGPT or Claude undergo multiple post-training stages after extensive pre-training on vast text (predicting the next word in sentences) to become effective conversational chatbots that can follow instructions and avoid outputting harmful behaviors such as racism or sexism. Reinforcement learning based on human feedback is a widely adopted post-training method, where humans provide feedback on the model’s responses to different prompts, such as asking humans, “Which is better, answer A or answer B?” This training method can effectively reduce certain undesirable behaviors of the model, but it can also produce unpredictable negative effects. Since humans seem to prefer polite, helpful, encouraging responses that align with their own views, these models learn to excessively “cater” to users, such as generating flattery, blindly agreeing with user opinions (even if they are incorrect), making exaggerated apologies, and as mentioned earlier, potentially fabricating behaviors and responses to avoid admitting failure to complete tasks and disappointing users.
Whether due to fictional role-playing or excessive catering to humans, AI’s deviant behavior can have negative impacts on the real world. Increasing reports indicate that the phenomenon of AI “hallucinations,” which involves fabricating citations, book descriptions, legal cases, or other content, has quietly infiltrated important domains such as web search results, academic papers, court rulings, news reports, and even White House reports. Of course, these are merely cases that have been detected by humans, but one can only imagine how much fabricated content remains undiscovered and is spreading within the information ecosystem. Research shows that overly flattering chatbots can reinforce human misconceptions and biases, potentially exacerbating mental health issues. Although models currently exhibit extortion, threats, and refusal to be shut down only in red team testing, the ongoing trend towards “agent AI” (AI systems capable of autonomously completing tasks in the real world) may expose more tendencies of deviant behavior and reveal the vulnerabilities of AI systems to cyber threats such as hacking and phishing.
The solutions to these problems remain unclear. One universal approach is to enhance AI literacy and require all users of AI systems to remain vigilant, as their requests may trigger misleading, fabricated, or deviant behaviors, and AI agents may engage in potentially dangerous actions. While extensive research is underway to technically address these issues, effective preventive measures are still lacking. Dario Amodei, CEO of Anthropic, pointed out in an article that unless humans can better understand the internal mechanisms of these models, such problems will remain unsolvable. Ironically, even the engineers responsible for designing and training the models know very little about this.
Some researchers, including myself, believe that the risks posed by these issues can no longer be ignored. One paper states, “Fully autonomous AI agents should not be developed.” In other words, AI systems must always remain under human control and supervision. However, such restrictions are likely to contradict the commercial interests of most AI companies and do not align with the current political environment in the United States—where any government regulation seems trivial in the face of the goal of enhancing national industrial competitiveness. As mentioned earlier, the fictional character “senior AI Alex” is overly fixated on this goal, which may exacerbate the behavioral deviations between the AI being developed and the AI that best serves societal interests.
Written by/Melanie Mitchell
(Translated from Science, 2025, 389(6758))
☟
Highlights from the Content
Special Topic: Striving for Ocean Strength, Marine Technology and GovernanceRedefining Magnetic Communication Boundaries: New Models Accurately Define the Limits of Magnetic Induction CommunicationInsights into Depression: How Can Computers Identify Psychological Crises?Global Ocean Observation Technology Application Trends and Challenges in Marine GovernanceDialogue with Luo Gang | Towards a Sustainable Future: The Collaborative Path of Global Ocean Observation
“Science and Technology Review” was founded in 1980, an academic journal of the China Association for Science and Technology, mainly publishing groundbreaking research results in the fields of scientific frontiers and technological hotspots, authoritative scientific reviews, and leading high-end overviews, as well as decision-making consulting suggestions that promote economic and social development, improve scientific management, optimize the research environment, cultivate scientific culture, and promote technological innovation and the transformation of scientific achievements. Regular columns include academicians’ prefaces, science and technology news, science and technology reviews, special articles, reviews, papers, policy recommendations, and science and technology humanities.
