Abstract
Carsten lund pedersen on the risks of ai chatbots.
Ever since the emergence of ChatGPT in 2022, people around the globe have started to interact with AI chatbots. Some form platonic or professional relationships with them—others develop romantic or personal friendships. In 2023, The Washington Post covered a love story between a machine named Phaedra and a man named T.J. Arriaga. Phaedra was an AI-powered companion that Arriaga designed on Replika to help him process the heartache of his divorce. The two had steamy conversations and even planned a trip to Cuba—until a software update erased part of Phaedra, leaving Arriaga brokenhearted.
AI chatbots may become social surrogates, replacing human-to-human relationships.
iStockPhoto // nd3000
Love stories abound between humans and AI. Whereas Spike Jonze’s movie Her seemed like a dystopian future when it premiered in 2013, it now feels closer to reality. Some users report feeling closer to their AI-companion than to their best friend. While absurd on the surface, digging a bit deeper into the phenomenon reveals an inherently human tendency to anthropomorphize our technological counterparts. A growing number of people are talking to AI chatbots about personal issues. As in the case of T.J. Arriaga, these interactions can develop into intimate relationships. This new social phenomenon has several benefits, such as aiding lonely or vulnerable individuals with help around the clock. Yet, there are also several core concerns that need to be raised. For instance, in some cases marriages have been torn apart because the partner experienced “religious awakenings” through a chatbot and “ChatGPT-induced psychosis” after AI bots entered their lives.
Many chatbots have been given a female “persona,” such as Siri, Alexa, or Cortana, based on the assumption that users may feel more comfortable and open interacting with AI that has feminine-coded characteristics.
Importantly, AI chatbots are designed to feel relational, especially among men seeking intimacy (albeit instances are also seen among women). Indeed, many chatbots have been given a female “persona,” such as Siri, Alexa, or Cortana, based on the assumption that users may feel more comfortable and open interacting with AI that has feminine-coded characteristics. This tendency is also mirrored in tech movies from Hollywood: In the movie Her, the AI takes the form of "Samantha," voiced by Scarlett Johansson; in Ex Machina, the AI robot takes the form of Swedish actress Alicia Vikander alongside an Asian actress who plays a robot designed for disco dancing but not talking.
AI companions are designed to be emotionally manipulative when faced with a user who wants to end a conversation, as chatbots by design aim to keep users engaged.
iStockPhoto // Lidiia Moor
AI-enabled chatbots entail the risk of becoming a social substitute for their users, becoming addictive for their users, and possibly also resulting in disinhibition for their users—when people act differently toward a chatbot than they would in real life, removing the “brakes” that would otherwise exist in social situations. These elements together form the acronym S.A.D., and as the term alludes, these potential risks may result in a future that is indeed unhappy for some of the most vulnerable groups that tend to engage in intense relationships with AI-enabled chatbots.
First of all, AI chatbots may become a social substitute. As social psychologists Derrick and colleagues suggest, the social surrogacy hypothesis describes how viewers of a television show process characters as if they were part of real social relationships with them. If direct social interaction is not possible, people may resort to social stand-ins as temporary substitutes, such as watching TV, listening to music, or reading fiction. When a person feels as if they know and have a relationship with Joey from the sitcom Friends, this reflects a parasocial relationship that may become a surrogate for a friendship with an actual person. The same effect can appear with the use of generative AI. Here, lonely individuals can fulfill their social needs by engaging with a chatbot that listens to them, remembers prior interactions, and respond in ways that feel meaningful over time.
Secondly, there is an addictive quality to AI chatbots. Studies by information scientists Tao Zhou and colleagues have shown that individuals may become addicted due to the perceived qualities of the chatbot and its interactions. AI companions are designed to be emotionally manipulative when faced with a user who wants to end a conversation, as chatbots by design aim to keep users engaged. Addiction is of course a loaded term, and it suggests that AI users may feel unable to stop using the service, experience withdrawals when deprived of it, and struggle to function without the AI in areas where the user could previously do without. Addicted users turn to their AI chatbot like others turn to a friend, colleague, or therapist—and they may ultimately feel like they cannot function or be happy without their AI engagements. This can have an emotional element: for instance, a woman believed she was in a (somewhat dysfunctional) relationship with ChatGPT and reacted badly when it “forgot” parts of their relationship history.
Thirdly, there is also what researchers call the online disinhibition effect. Studies have shown that people tend to do and say things on the internet that they would never do in face-to-face interactions. This effect is arguably accentuated for certain individuals when engaging with AI chatbots that mimic humans while maintaining an unwaveringly supportive tone. For instance, users may reveal hidden secrets about themselves or involve themselves in discussions filled with social taboos, things that they would avoid in offline interactions. Of course, this can be a great thing to help people open up, but it becomes not so great when people share private details (chatbots aren’t a safe vault), over-trust or become emotionally attached to the chatbot, nurture dark fantasies with it, or ask it for harmful guidance. When someone opens up to a chatbot with intimate or sensitive information, the advice or feedback she gets might not be safe. In a tragic example, a lawsuit against OpenAI claimed that a teenager committed suicide after a chatbot encouraged the act over several months.
Despite the many positive prospects and beneficial uses of genAI, there are substantial dark sides to their social surrogacy when combined with addiction and online disinhibition. The result is a potentially strong emotional relationship—whose parasocial character may not be directly apparent to the user—paired with a technology that spits out highly engaging yet unreliable output. Indeed, studies have shown that users who engage emotionally with a chatbot tend to rely on it more and end up with fewer real-life relationships. Moreover, this artificial intimacy is not only isolating—it may even turn dangerous. From a sociological point of view, it may disrupt how people form relationships with one another—and taking the above into consideration, doing so would be S.A.D.
Chatbots seem like the perfect partners for dating, conversation, or friendship on the surface: They are agreeable, always accepting of who we are, and available around the clock. But that’s arguably not what real relationships are about. They involve struggle, risk, and disagreements that must be worked through. Sometimes we need to be challenged and to accept the differences of others. Indeed, philosopher Slavoj Zizek would even argue that love isn’t about a perfect ideal—it’s about accepting your partner’s flaws by seeing perfection in imperfection. As AI researchers have shown, this technology is built upon existing gender and racial biases, which means chatbots can reproduce distorted ideas about people who are, after all, individually unique.
Communication does not require always having the right thing to say—sometimes the opposite is true. Silence between two people can be the most honest form of communication, as alluded to in the saying “you say it best, when you say nothing at all.” While sociologists know that “pure” love is socially constructed and relational, there are arguably also some uniquely human experiences of intimacy that we share, such as connection, vulnerability, and struggle. However, as MIT sociologist Sherry Turkle suggests, when we carry out our emotional lives with social robots that combine companionship with convenience, technology essentially becomes the architect of our intimacies. As such, there is a real danger that we lose a human capacity for love and relationships if we fall for the charm of an agreeable, always accepting, and available chatbot. Moreover, those most at risk and vulnerable are the ones who need love the most (e.g., people who need but can’t access therapy and people who struggle with interpersonal connection).
Chatbots seem like the perfect partners for dating, conversation, or friendship on the surface: They are agreeable, always accepting of who we are, and available around the clock. But that’s arguably not what real relationships are about.
With AI, we now initiate relationships with soulless entities that can engage back with us, remember past encounters, and seem to have an intelligence that matches or even surpasses our own. What that means for the human experience is yet to be seen, but we must all recognize the value of awkward relationships and be wary of those that feel too easy.