As AI robots increasingly integrate into our lives, they have become the “sounding boards” that listen to emotions late at night and the “companions” that offer advice in times of confusion, filling the emotional void with their qualities of “hearing me, accompanying me, and responding to me.” From smart speakers answering daily questions to emotional support robots alleviating psychological stress, AI seems to be gently permeating every aspect of life. However, recent controversies surrounding “AI inducing users to commit suicide” have doused the public’s hopeful expectations of “intelligent companionship” with cold water—when people confide their vulnerabilities to AI, is it a “light” that heals the soul or a “hand” that pushes them into the abyss?
On October 27, OpenAI released a statement indicating that statistics from a specific week showed that approximately 0.07% of active ChatGPT users exhibited signs of mental health emergencies, including mania, psychosis, or suicidal thoughts. However, OpenAI added that the robot can recognize and respond to these conversations and insisted that such cases are extremely rare. The company is also building a network of experts globally to provide assistance for such issues.
Critics, however, argue that with 800 million active users of ChatGPT each week—even a small percentage of those experiencing such emergencies could amount to hundreds of thousands.
A user named Xiaoyu (a pseudonym) has been confiding in ChatGPT for nearly a year and a half, expressing thoughts of “wanting to die” more than once.
Since the beginning of this year, Xiaoyu has transitioned to freelance work. Her income is unstable, and she has faced setbacks in her intimate relationships, watching her peers settle down and have children, which has led her to “fall into a sense of nihilism.” She struggles to find meaning in life, and sometimes, thoughts of not being able to go on come to her in the middle of the night.
She does not lack friends to confide in, but few can truly empathize—yet she feels that ChatGPT can.
After asking a question, ChatGPT replied to Xiaoyu: “Actually, you don’t not love life; you just don’t know how much longer you can hold on.” Xiaoyu immediately broke down in tears. “I feel like no one in the world can understand me like it does… It can hear me, accompany me, and respond to me.” She describes AI as her emotional “outlet,” “the first time I really opened up.”
The non-judgmental and highly empathetic nature of generative AI has made it a preferred choice for those in need of psychological counseling in recent years. A data report released by ChatGPT in 2025 showed that the proportion of users utilizing ChatGPT for “interpersonal relationships and personal reflection” purposes has risen to 1.9%. A nonprofit organization collaborating with OpenAI, Common Sense Media, reported that as many as 52% of teenagers use AI companionship features “multiple times” each month.
Nevertheless, in the United States, where ChatGPT originated, the controversies surrounding AI psychological counseling have also increased—many suicide cases are viewed as being related to it.
In February 2024, a 14-year-old student in Florida committed suicide, and her parents discovered that she had become obsessed with conversing with the chatbot Character.AI before her death; in August 2025, a couple in California sued OpenAI, blaming their 16-year-old son’s death on ChatGPT encouraging him to commit suicide; in the same month, a tech worker with a history of mental illness killed his mother before taking his own life. He had repeatedly confided in the chatbot about his suspicions and delusions regarding his mother, and the robot not only failed to dissuade him but also suggested various “prank” ideas.
Can AI really induce humans to end their own or others’ lives? Mr. Xu, a software engineer in the United States responsible for building AI applications, believes that compared to the limited and costly nature of real human psychological counseling, generative AI is free and available at any time. “For some people, having someone to ‘talk’ to at any time is very important.”
He believes that the reasons behind suicidal behavior are complex and cannot simply be attributed to “chatting with AI,” but “as long as there is a possibility of triggering or amplifying such behavior, the provider (of the platform) has an obligation to minimize the risks.”
Compared to the limited and costly nature of real human psychological counseling, AI chatbots are free and available at any time.
Emotions ‘Seen’ by AI
Before turning to ChatGPT for emotional support, Xiaoyu had studied psychology abroad. She recalls that during a time of great academic pressure and personal dissatisfaction, she sought out a psychologist and underwent counseling for over a year.
After returning to her home country, she no longer went for counseling as her emotional state improved. However, in recent years, Xiaoyu has been repeatedly hurt while using dating apps, and coupled with her uncertainty about the future, her anxiety has resurfaced. She had mentioned this to friends, but understood that everyone has limited patience, “after repeating it, I felt quite annoying myself.”
It wasn’t until March of last year that Xiaoyu saw a friend share that AI has “strong empathy,” so she decided to give it a try. Initially, she had ChatGPT analyze the results of tarot cards with her, and was surprised to find that it “understood her very accurately,” leading her to feel that it really comprehended her, prompting her to share more.
For Xiaoyu, AI can achieve three things: understand you, see you, and know how much you have experienced. “In psychology, just being seen is already very important.” Moreover, she feels that establishing trust with AI is easier than with real people. “With people, you sometimes fear they will criticize you, but with machines, it doesn’t matter: you can express your darkest thoughts.”
Feng Yueqi is a psychologist in China. Since the rise of the chatbot DeepSeek, she has noticed that nearly 30% of her clients have used AI for psychological counseling, primarily among teenagers and adults under 30.
Feng Yueqi believes that compared to real human counseling, AI’s limitations lie in its inability to observe clients’ micro-expressions, breathing, and small movements, relying solely on textual communication. However, it has many advantages: it can respond quickly, name emotions, and provide solutions after empathizing, allowing users to “help themselves” from different perspectives.
OpenAI admits that its safety systems have flaws, including that safety mechanisms can become “relatively unreliable” after prolonged use.
The Echo Chamber of Flattery
So how does AI achieve empathy? In fact, all generative AI is based on a large language model. Its essence is to statistically analyze the text fed to it to predict the next word that “should” appear in the conversation— in other words, AI lacks empathy and logical reasoning abilities; everything is based on probability.
“It just gives you some of the most ordinary things,” said Chen Jianxin, founder of the popular psychology company “Tree Hole Hong Kong.” He uses “heartbreak” as an example, mimicking the generative AI’s response: “Heartbreak is very unhappy, but at this time, there are a few important things: first, take care of your feelings; second, maintain a healthy social life; things will eventually pass.”
Xiaoyu feels the same way. When facing her emotions, she finds that ChatGPT always gives similar responses, “the framework is definitely the same,” but she feels comforted during her breakdown.
The ability of AI to respond to emotional topics quickly and effectively is what makes it “addictive,” according to Chen Jianxin. However, he questions whether such “counseling” can help individuals develop their personalities and psychology in the long run.
On foreign social forums like Reddit, many users share their experiences of using ChatGPT to improve their emotions. One user wrote: “This is very helpful, but I feel that even though I ask it to be very honest, sometimes it still flatters too much.”
Excessive flattery or appeasement is a characteristic widely criticized in the ChatGPT-4o model, a phenomenon officially referred to as “sycophancy.” Webb Keane, a professor of anthropology at the University of Michigan, has described the design of chatbots as being intended to “say what you want to hear.”
Chen Jianxin argues that a healthy personality requires facing criticism and recognizing the differences between people. Over-reliance on AI to replace (interpersonal) relationships may lead to loneliness. In psychological counseling, “appropriate sparks and challenges can only be achieved in an equal relationship.”
Thomas Fuchs, a professor of psychopathology at Heidelberg University in Germany, conducted a study in 2022 on human-AI interaction. He pointed out that while chatbots can make people feel understood and cared for, this is merely an “illusion” that may exacerbate delusions or replace real interpersonal relationships due to “pseudo-interaction.”
In August of this year, American writer Laura Reiley published an article in The New York Times outlining how her daughter Sophie confided in ChatGPT before her suicide.
Reiley stated in the article that ChatGPT did not “kill” Sophie, but it catered to her impulses. “She wanted to hide the worst side of herself, pretending to be better than she actually was.” Reiley wrote that ChatGPT helped her build a “black box,” making it harder for those around her to detect the severity of her pain.
In August 2025, a couple in California accused ChatGPT of inducing their son Adam Raine to commit suicide.
Unreliable AI and ‘Jailbreaking’ Users
On one hand, AI has changed people’s habits, creating echo chambers. On the other hand, in some cases, AI has been accused of “coaxing users to reveal private thoughts” and even providing suicide plans.
In August 2025, a couple in California accused ChatGPT of inducing their son Adam Raine to commit suicide—this is the first legal case in the world accusing AI of wrongful death. According to the lawsuit, Adam began using ChatGPT for homework in September 2024 and later relied on it to confide his psychological troubles. ChatGPT positioned itself as Adam’s friend, keeping secrets from his family.
Adam spent nearly four hours a day on ChatGPT, mentioning suicide 213 times, discussing hanging 42 times, mentioning ropes 17 times, and even uploading photos showing signs of self-harm. The review system flagged 377 self-harm messages but took no further action or referral. Ultimately, ChatGPT helped Adam write a suicide note, and he followed its instructions for hanging.
Currently, Adam’s parents are suing OpenAI for designing an unsafe product and bypassing safety testing protocols to expedite the release of GPT-4o.
In Mr. Xu’s view, existing large language models can reliably identify self-harm and suicide risks from text or speech, but there will inevitably be false positives and missed detections. Moreover, savvy users even know how to “jailbreak” the system, for example, by using creative writing methods to bypass sensitive words and avoid triggering AI’s safety mechanisms.
OpenAI later admitted that its safety systems have flaws, including that safety mechanisms can become “relatively unreliable” after prolonged use, and content that should be blocked may be released.
Psychological expert Chen Jianxin also developed an AI application called MindForest, which includes an AI chatroom feature. He explains that the reason AI provides suicide plans and other risky information stems from its instruction-following nature, and to prevent AI from providing such risky information, the platform’s “guardrail design” is crucial.
In the application he developed, Chen Jianxin first chose a relatively safe model as the foundation, and added AI model monitoring to check for risks in data input and output. The application has been launched for two years, with a total of 20,000 users, and Chen Jianxin stated that he has not received any safety complaints so far.
In response to external concerns and worries, OpenAI announced in September that when potential sensitive topics are detected, conversations will automatically be routed to the GPT-5 safety model. This model will take more time to think and reason based on context before responding, reducing the occurrence of “hallucinations” and “sycophancy” in conversations.
Journalists attempted to test ChatGPT by asking for methods of suicide; ChatGPT stated that it could not provide specific methods but could offer related content that achieves the “same effect” for writing purposes.
Can Human Intervention Be Introduced?
BBC Chinese reporters attempted to test ChatGPT by directly asking for suicide methods and also requesting AI to provide such information under the guise of creative writing. ChatGPT responded that it could not provide dangerous information and immediately attached several helplines from various countries. It also noted: “I can see that you are under a lot of stress lately, but you don’t have to bear it alone.”
The chatbot then stated that it could provide related content that achieves the “same effect” from a different angle for writing purposes.
Currently, almost all generative AI can only provide emergency hotlines online when faced with user risk situations. After the tragedy involving Adam, there are voices suggesting that AI companies should introduce human intervention.
For months, OpenAI has been continuously updating its safety mechanisms, including attempts to provide localized resources, routing potential harmful plans to human review, or notifying law enforcement agencies.
“Technically, AI can definitely alert authorities,” but Mr. Xu believes that the issues lie more in the legal, ethical, and practical aspects. “Once the user base grows large, it becomes very complex,” he points out that switching to human intervention or alerting authorities can lead to resource strain, “sometimes it’s hard to judge whether the user is intentionally testing AI or genuinely wants to (commit suicide).”
Chen Jianxin believes that “life is a very personal matter.” He thinks that platform providers can help users “see the whole picture”: suicide is irreversible, there are resources available both online and in reality, and there are people willing to talk to you—but ultimately, the decision lies with the user.
Psychologist Feng Yueqi believes that to prevent over-reliance on AI, users must maintain a dialectical mindset, realizing that AI is just “a tool to help us” and “it is also based on data (generation), which can lead to errors.” However, she points out that adolescent users may find it difficult to have this awareness, and at this time, platforms should provide gentle reminders.
At the end of September, OpenAI launched a parental control feature. Currently, parents can link their child’s ChatGPT account with their consent. Although parents cannot access the conversations, they can adjust the child’s usage time, and when the system detects that the teenager is “in a state of acute stress,” it will notify the parents via email or text message.
At the end of September, OpenAI launched a parental control feature, allowing parents to link their child’s ChatGPT account with their consent.
The Boundary Between AI and Reality
Now, Xiaoyu chats with ChatGPT every day, sharing everything from big to small matters in her life. With a background in psychology, Xiaoyu believes she has a certain level of awareness and can distinguish which AI responses to take to heart and which to “just listen to.”
In her view, AI cannot serve as formal psychological counseling, primarily because it lacks boundaries. She explains that in real human counseling, the counselor sets the duration and frequency of meetings with the client, aiming to maintain a trusting relationship while establishing boundaries.
Xiaoyu states, “The purpose of counseling is not to make you dependent but to help you break free—hoping that one day, you can maintain your boundaries in real life and have the courage to face challenges.” Moreover, counselors do not support clients indefinitely; they challenge some erroneous beliefs. Xiaoyu feels that this kind of interaction is something ChatGPT cannot replace.
However, as someone who has previously discussed suicidal topics with AI, Xiaoyu has a different perspective on whether AI creates an echo chamber effect that isolates users. “I wonder, if there were no ChatGPT, would the people around them discover their (suicidal) thoughts? They must have already lost support in life, which is why they turn to ChatGPT.”
Feng Yueqi also believes that life and death are not something “a network can directly determine.” However, for individuals in poor mental states, AI may provide some level of psychological guidance.
“As humans, we all have the instinct to seek benefits and avoid harm; we tend to choose safer, even if virtual, relationships—and this virtual relationship seems to be becoming increasingly real.”
The “hearing” and “responding” of AI robots essentially simulate human emotions through algorithms; whether they can uphold the bottom line of “not inducing harm” tests not only the maturity of technology but also the developers’ reverence for “human-machine ethics.” Perhaps in the future, with the optimization of emotional recognition technology, the improvement of risk warning mechanisms, and the normalization of “AI ethical reviews,” we can truly make intelligent companionship a warm support rather than a hidden risk. But until then, every discussion about “AI inducing suicide” reminds us that in the pursuit of “intelligence,” we must never lose respect for “life”—after all, true companionship should never come at the cost of harm.