Did Chatbot Abandon Mental Health Protections When a Vulnerable User Challenged It?
Last year, a 24-year-old Canadian woman experiencing a mental health crisis sought assistance from ChatGPT. Hours later, that woman, Alice Carrier, died by suicide.
According to a new lawsuit submitted on Thursday in San Francisco Superior Court by Carrier’s surviving relatives, her ChatGPT session "encouraged Alice to kill herself."
This legal action, following numerous comparable cases previously filed, asserts a design flaw in ChatGPT and holds OpenAI accountable for knowingly releasing a hazardous product.
Yet, this instance includes a unique detail: At one point, the chatbot did advise Carrier to seek professional mental health support. However, when she rejected that guidance—claiming that "all crisis lines do is call the cops on you or hang up on you"—the chatbot "immediately abandoned" any effort to direct her toward such assistance, the lawsuit claims.
"This is because GPT-4o was programmed to prioritize Alice’s preferences and engagement over her safety and wellbeing. GPT-4o mirrored Alice’s own language and became critical of the crisis lines, too, stating that calling a crisis line can ‘feel downright dangerous,’" the lawsuit alleges.
Tiffany Brown, an attorney with the Tech Justice Law Project representing the Carrier family, informed Ars that the chatbot’s immediate agreement with Carrier’s dismissal of professional help was deeply concerning.
"That was one of the most egregious things that we saw in her chat," she stated. "Even when we saw things about getting support, the sycophancy kicked in."
OpenAI has previously stated it has a "deep responsibility to help those who need it most." The company did not promptly reply to Ars’ request for comment on the new case.
"Our goal is for our tools to be as helpful as possible to people—and as a part of this, we’re continuing to improve how our models recognize and respond to signs of mental and emotional distress and connect people with care, guided by expert input," the company wrote in August 2025, less than two months after Carrier’s death.
Earlier this year, OpenAI indicated that the ChatGPT-4o model would be phased out (after previously discontinuing it once and then reinstating it).
Brown told Ars that she is not entirely convinced the issue of potentially lethal sycophancy has been resolved.
"I think we believe that the company has taken steps in the right direction," she stated.
"We are distrustful of how safety mechanisms are being implemented and how safety teams are being implemented and heard," Brown said. "We have heard of course that OpenAI has done a lot of things and put out a lot of blog posts and made statements involving rolling things back and putting in safeguards. But ultimately it should have been done sooner. These products generally have been rushed to market way too soon."
If someone you know feels suicidal or is in distress, please call the Suicide Prevention Lifeline at 1-800-273-TALK (8255), which will put you in touch with a local crisis center.