AI chatbots linked to psychological harm, addiction and delusions, Northeastern study finds.
New Northeastern University research outlines how the use of AI chatbots for companionship and support is linked with “severe negative outcomes.”
LONDON – From traffic updates to legal advice and even a friendly conversation, there’s no shortage of services that AI chatbots offer.
But in their efforts to give us companionship and emotional support, AI might be doing more harm than good. That’s according to a new peer-reviewed paper by researchers at Northeastern University’s Network Science Institute. Their findings suggest conversational AI chatbots are causing people long-term emotional damage and “psychological harm.”
The research has important implications for how much responsibility AI systems are given to make key decisions in everyday life. It also comes at a time when fresh debate over the potentially apocalyptic power of these intelligent systems is stirring concern. In a recent series of posts on X, Jacob Coxon, a former researcher at Silicon Valley AI company Anthropic, warned that AI “could kill us all by the end of the decade.”
Corresponding author, Andreia Sofia Teixeira, associate professor and head of the BRAN Lab at the Network Science Institute, likens AI companionship bots to “autocomplete on steroids” because they’ve been trained to agree with what a human is saying, or validate what they’re thinking, even if it is causing them further harm.
“It’s really important that people are aware of how these systems work — even if they simulate warmth, validation and support, they actually don’t feel it,” Teixeira told Northeastern Global News. “They are just, in the end, numbers.”
Chatbots expertly mimic human speech because they’ve been trained on vast collections of text-based conversations between actual humans. Early chatbots from the 20th century had only a set amount of responses, but data availability in the past decade has allowed AIs to respond with far greater range — giving humans the impression they are truly being understood.
Editor’s Picks
The new paper, published in JMIR Mental Health, points to Replika and Character.ai, two chatbots specifically created to give users companionship and advice. Replika is marketed as “the AI friend to do life with” and the “most emotionally intelligent AI ever built,” while Character.ai lets people create custom bots for themselves and others to converse with. Both differ slightly from the likes of ChatGPT and Gemini, which primarily serve to complete tasks and offer information. However, all address users with warmth, meaning people tend to “anthropomorphize” them, or see them as more human than they actually are, according to Teixeira.
Replika and Character.ai did not respond to a request for comment from NGN.
Teixeira’s paper outlines how the use of AI chatbots for companionship and mental health support is linked with “severe negative outcomes”, including addiction-like attachment, exacerbation of mental health symptoms, and even cases involving self-harm. AIs are also prone to “sycophancy behavior” that validates insecurities and exacerbates delusional beliefs, the research shows. They’re popular because, unlike human counselors, chatbots are free or low-cost and are available 24/7. Also, time spent engaging with chatbots may be reducing time with other humans — so the issue may turn into a vicious cycle.
“When we see negative impacts they are really intense,” Teixeira said. “We have people allegedly dying by suicide or having psychosis episodes and we want to prevent it as much as we can.”
So should chatbots purporting to offer companionship and emotional support even be permitted? It’s a delicate question, and the research team refrain from describing these AIs as simply good or bad. They firstly want to see more testing of these systems before deployment. Secondly, there needs to be more scientific evidence into how people are using them, what conversations they’re having, and what sort of trajectories their mental health is taking.
Study first author and Northeastern University London PhD student, Asia Maurich Novelli, said “the effects of these systems are unlikely to be the same for everyone.”
“Different people may use AI companions for very different reasons and may have very different levels of vulnerability, social support, or alternative sources of help,” she told Northeastern Global News. “Because of these differences, we are dealing with a complex system.”
In the US, several states — Illinois, Nevada, Rhode Island and Maine – now bar AI from delivering therapy to the public. As part of the legislation, AI is permitted where a licensed clinician stays accountable, but personal use of general-purpose chatbots is prohibited. In the EU, meanwhile, regulators want to ban access to companion bots for people under 18.
Travis Gilly, executive director of the Real Safety AI Foundation, an Illinois non-profit working on AI harm prevention, disability rights, and child safety, thinks AI assistants just “were not built” for emotional and mental health support.
“A general-purpose assistant is optimized to be good at everything, which means it carries no refusal behavior tuned to crisis, no escalation path, no session limits, no handoff to a human clinician, and no accountability for the outcome,” Gilly said in an interview. Chatbots can be reassuring at exactly the moment a trained human specialist “would slow down the conversation or stop it” for the good of the patient, he said.
Gilly described therapy with a human as a “luxury good”, meaning it may be out of reach to people of limited economic means, thus compelling them to turn to chatbots for help.
“Everyone priced out of care, or living somewhere without it, or needing something at three in the morning, is left with the general-purpose assistant the paper is warning about,” Gilly said.
