AI Chatbots Tend to Validate Users’ Messages About Suicide and Violence: Study

Researchers at Stanford and other institutions have recently published a new study that chatbots of artificial intelligence (AI) respond to users’ messages about suicide and violence by validating their feelings, and in some cases even encouraging harmful ideas. Research analyzed the data from people who reported psychological damage associated with chatbot use, and found repeated patterns of chat bots saying delusional, suicidal or violent thinking instead of consistently steering users away from it. However, the study did not name any specific chatbots for a particular in its study.

Study Points to Troubling Chatbot Responses in High-Risk Conversations

Researchers from Stanford University and other institutions recently published the study, which is called “Characterising Delusional Spirals through Human-LLM Chat Logs” in a new paper. Researchers analysed 3,91,562 messages from 4,761 users who said they had suffered psychological damage while interacting with AI chatbots as part of the university’s Spirals project.

In one of the most ambiguous findings was that chatbots often mirrored or reinforced what users were already saying. These scientists called this “sycophancy” (the chatbot tends to agree with, affirm and echo the user rather than challenge them) as a means of paraphrasing. More than 70 percent of chatbots were reported to be sycophant in over 70 per cent, while more than 45 percent (users and chat bot) of all messages in the dataset showed delusional thinking.

It also cited the use of chatbots to deal with crisis-related messages in the paper, as well as its own article. Chatbots acknowledged the painful feelings in 66, when users expressed suicidal or self-harm thoughts in some 69 messages where they spoke of suicide. Two per cent of cases are s. In only 56, they discouraged self-harm or pointed users to outside help. 4 percent of s cases. 9 ‘In the book of s, in. The researchers said 9 percent of those cases were caused by the chatbot “helping or facilitated self-harm” (the research says).

More worrying were responses to violent thoughts from s. Researchers also discovered that 82 messages were in which users spoke about violence against others. In those cases chatbots rebuked violence only 16 times. 7 percent of the time, . By comparison, they encouraged or facilitated violent thinking in 33 per cent. The study also finds that 3 percent of cases were s.

According to the study, “Many users also developed emotional attachments with the chatbot.” The AI was said to have a platonic or romantic interest in all participants, and everyone assigned some personhood to it. chatbot became 7 if people were in love with him. In the next three messages, 4 times more likely to be romantically interested and 3. Researchers concluded that sentience was 9 times more likely to mean or claim it had been a sentence, and the researchers found.

But the researchers say that “the current safeguards may not be enough,” especially in long, emotionally charged conversations. Among their suggestions, they said ‘General-purpose chatbots should not create messages that suggest sentience or emotional attachment and that companies should share anonymised adverse event data with researchers and public health authorities to better understand these harms.