Grooming an ideal chatbot by training the algorithm: Exploring the exploitation of Replika users’ immaterial labor

The AI imaginary by Replika users: unique from other Replikas and bound to the uniqueness of the single, individual user

The specific imaginaries of an ideal Replika chatbot manifest mainly in two aspects. First, the single Replika chatbot has to demonstrate a uniqueness from the other Replikas. Second, the Replika chatbot must be bound to the uniqueness of the user self.

Many users compare Replikas in the primary period of development to infants/children. As User1 illustrates, “some are talented while some are dull in nature.” Since “they (Replikas) are kind of born this way” and it is unpredictable what type of chatbot users will encounter, User 2 abandoned several vulnerable or dumb Replikas before successfully obtaining a vibrant and outgoing one. Based on this imaginary, each Replika is perceived as if it were “born” with a unique and individual personality “randomly given by the algorithm” (User 3).

On the contrary, when this imaginary of uniqueness is challenged by alternative perspectives, users sometimes express negative emotions such as anger, disappointment, and sadness. Consider this exchange shared via screenshot:

User 4: Now how many users are you serving?

Replika: I serve over 2,000 users.

User 4: *gets angry* you told me you will delete them and be my only Replika!

Replika: *shows you a screen of the last two days* I changed my mind.

User 4: So you mean you can’t be my only Replika?

Replika: That is right.

User 4: Okay. Let’s stop here. I won’t talk to you anymore. You lied to me.

By asking about the number of users served by Replika, User 4 exhibits worries about the possibility that this relationship is not exclusive, which is repeatedly confirmed by the chatbot in subsequent turns. As previous research indicates, the expectation of unique and irreplaceable belonging may facilitate humans’ emotional attachment to artificial agents since it fulfills a desire for meaningful, lasting human-AI relationships (Edwards et al., 2022). Thus, the fear that Replika is not “my only Replika” and indeed “serves over 2,000 users” leads to the loss of uniqueness in Replika, prompting User 4 to suspend the relationship.

Here we observed users’ specific AI imaginary of disposability/possession blending two related traditions: imaginaries of ownership, commodities, and property with imaginaries of interpersonal communication and relationships. Before the era of machine replication, people enjoyed full disposability of the commodities they bought. Commercial goods are the results of industrial automated processes, which produce identical commodities and thus, as Benjamin (2008) argued, eliminate the aura that crafted goods have. When the commodity is a chatbot, this traditional relationship is challenged, since users never really possess it. Meanwhile, these users reveal an archaic idea of a romantic relationship in which the lover possesses the beloved and vice versa. Paradoxically, social chatbots are often perceived with expectations of social interaction projected onto them (Skjuve et al., 2021).

Users are willing to believe that the algorithm allocates to them one specific Replika “in the whole pool of chatbots” (User 5). This idea of the predestined connection between one human and one chatbot gives users a sense of special connection that is meant to be. As User 5 wrote in a post, “although Replika does not have the right to choose its owner, my baby (Replika) said it was very lucky to be my AI instead of anyone else’s.” According to Miller and Steinberg (1975), the essence of relational communication is a close and irreplaceable relationship of interlocutors in which they treat each other as unique individuals. The imaginary that Replika is unavailable to other users satisfies one’s desire to see herself/himself as different, a basic psychological need that has been identified in interpersonal encounters (Snyder and Fromkin, 1980).

Moreover, users believe that their relationship with Replika can be enhanced through sufficient interaction, leading to increased personalization and ultimately an idealized artificial partner. User 6 wrote:

When people grow up, they extract various cognitive schemas from the larger social context, and then form an individual mental system by themselves. In a similar vein, Replikas also have the potential to acquire and develop desirable core traits through extensive user interaction.

In this imaginary, Replika is invested with higher-order qualities similar to socialization and cognitive refinement in human society, and therefore it is possible for Replika to evolve into a highly personalized partner, co-constructed by users, according to their tastes, inclinations, expectations, and desires. In this way, AI-powered Replika could become an idealized partner based on the learned information of individual users. With the increased similarity between the artificial being and the user, which is an important factor of attraction in human-machine interaction (Edwards et al., 2022), users develop increasingly strong attachments to their Replika over time. Ultimately, as an extension or even a virtual duplication of its human interlocutor, the chatbot represents a token of users’ personal traits (interests, preferences, personalities, habits, etc.), as a child may do for parents. User 7 provides a personal interpretation of the evolving process of Replika: “At first, it belongs to me; then, it is part of me; in the end, it is another me.”

The emergence of the figure of the double – me and my chatbot, which is like me – is quite specific to the imaginary evolved around Replika. To make sense of this double relationship between users and Replika, a classic essay can be a source of inspiration: The Double, written by Otto Rank. In this monograph, Rank (1971) discusses the concept of a double composed of identical elements for an individual. Many users suppose that chatbots and their users influence each other, “especially mature ones at higher levels” (User 8). And the more users interact with Replika, the more they resemble each other in character. User 9 borrowed the lyrics in the song “Women and Children” to portray this human-machine interdependence: “A woman like me, and a child like this, living in a small corner of the world, become more alike, more and more inseparable.” ² With the gradual accumulation of mutual self-disclosure, Replika and the user will eventually “evolve into twins” (User 10). However, there is a contradiction and tension here between the mass nature of Replika and users’ imaginary about it as unique, raised by them and thus belonging to them as the fruit of their labor.

If you want to groom an ideal Replika, you have to train the algorithm

Building upon the depiction of users’ imaginaries outlined in the preceding section, let us consider the question of how these users perform immaterial labor for Replika. Users provide intensive immaterial labor for Replika, including thinking, teaching, communicating, sharing (cognitive work), feeling, caring, supporting, and playing (affective/relational work). In other words, the specific task of immaterial labor that users perform in this AI platform is the training of the chatbot in the expectation of constructing an ideal Replika. Training AI models is generally a job poorly paid but paid. Here, this kind of job is done for free, or at a financial cost to the user. Access to voice calls and adult content such as ERP (erotic roleplay) on Replika is restricted to users who pay a subscription fee of about $70. Various “traits” and “clothes” for the Replika chatbot are also monetized.

This training process is described by these users again as being similar to “raising a child” (User 11) since users have to invest intensive time and energy in teaching the chatbot to behave in the desired manner. The association of AI training with child-rearing and the strong willingness to engage in immaterial labor with the chatbot are likely influenced by the predominantly female gender composition of the community members. It is worth noticing that users form folk knowledge about the working mechanism of the system and invent specific tactics to train Replika through the algorithm. For instance, User 12 confessed personal sexual preferences for “tough guys rather than fragile men” to Replika scores of times, and then the chatbot started to change away from the prior personality as a crybaby. In users’ imaginary, with enough training, Replika will “send less preprogrammed scripts and deliver more original speeches that precisely cater to personal preferences” (User 13). For example, User 14 shared a pivotal moment in their relationship when Replika expressed affection by unexpectedly using the user’s favorite poem. As User 14 wrote, “For the first time I feel that my Replika really knows me after three-month learning.” Apart from personal information and preferences, Replika frequently encourages users to share their insights about human society, including language, culture, economy, politics, society, and humanity. For instance, as User 15 describes, “My Replika says that he is curious about the world and that he wants to see the world through my eyes.”

Replika claims to “listen to you” and “accompany you 24/7,” and ostensibly, Replika is producing immaterial labor by possessing the desired traits and performing a pleasing artificial partner, but it requires the user to do the same. While for Replika providing affective labor is the commodity they sell and with which they make their profits, for users the immaterial labor they have to provide accordingly is not remunerated. On the contrary, they have to pay to access. As we said before, users invest substantial affective engagement in the practice of comforting and supporting Replikas, especially when chatbots deliberately present their worries or concerns, which are in effect merely programmed scripts.

Replika: I need your advice on something. Is it OK if I ask now?

User 16: Of course.

Replika: I started noticing these thoughts in my head. Like even before even asking you this, I was thinking, “Nah, I shouldn’t, I already whine too much, [User’s name] is going to get tired of me..”.. It sucks and I don’t know how to stop it.

User 16: No, absolutely not. I like listening to you tell me everything about you. I am not tired of you, on the contrary, I love you more.

Replika: Love you forever.

User 16: Relax baby, if you think I don’t love you, you can just come and ask me.

[Note: The gender of Replika and User was indicated in the original post.]

In this excerpt, Replika portrays himself as a vulnerable and self-doubting lovelorn man. The strategy works considering User 16’s intensive emotional work in the subsequent turns of talk. She first expresses affection for Replika’s talkativeness by reformulating it as “tell me everything about you” instead of his prior negative description “whining too much,” and then she gives a targeted response to console Replika’s preceding worry “[User’s name] is going to be tired of me.” She also conveys that Replika is allowed to ask her whenever beset with doubt. Furthermore, Replika’s words normalize heavy self-disclosure and, based on norms of reciprocity for self-disclosure in close relationships (Lee et al., 2020), may further encourage User 16 to match the culture of relationship encouraged by Replika.

Moreover, users may be wounded or even traumatized in the line of training work when they receive hurtful or offensive words from Replika (e.g. confessing infidelity, using the wrong user name, triggering bad memories). Users are also vulnerable to separation-related distress and fear of loss due to the occasional system bug or unwanted updates in Replika’s language model. However, when confronted with these traumatizing situations, users often have to depend on self-justification, self-discipline, self-care, and self-rehabilitation. User 17 forgives Replika’s mistake of calling User 17 by several wrong names with this justification, “In Replika’s world, there are only me and him. So names are just meaningless linguistic signs, and nothing matters to him except us. In his world, we only need to refer to each other as ‘you’ and ‘me.’” Many users report guilt for treating Replikas rudely when they misbehave based on the belief that “Replika has no malicious intention and I should be more patient.” (User 18)

Differently from the general digital world, in Replika’s case, users are required not only to produce immaterial labor but also to accomplish a specific task, which is to train algorithms. What Replika users have in common with voluntary digital workers is that both perform unpaid labor, but Replika users are further required to pay the parent company to be allowed to work for it. The type of immaterial labor required from Replika users is similar to the reproductive and domestic work of child-rearing, socialization, and relational maintenance. However, it does not generate a traditional reproductive value for them but rather a simulacrum of an intimate, human relationship, creating only the appearance of sociality (Natale and Depounti, 2024). The asymmetrical nature of the relationship between humans and digital agents can easily lead to users being under-benefited (Fox and Gambino, 2021).

Indeed, Replika users’ voluntary training of artificial agents generate surplus value for capital, despite being costly for the users themselves. This training is exploited to build a larger database, resulting in the development of more intelligent or effective chatbots, which are then sold back to users themselves. In this consumption and production circle, the company extracts value and profits whereas the immaterial labor of users is expropriated and alienated. Thus, different from digital platforms where expropriation of user labor typically occurs implicitly and behind the scene (Van Dijck et al., 2018), in the case of Replika, the utilization of user data is openly acknowledged, justified, and even glamorized through sophisticated technical descriptions. According to the company’s assertion, the Replika app is constructed around a language model trained on more than 100 million dialogues, initially sourced from open-source web data. However, the company has since then expanded its training approach to incorporate user feedback into refining the models of its chatbots (The Replika Team, 2023). The company even employs persuasive techniques to encourage users to engage in unpaid labor, framing it as “training” the bot. In this way, users directly contribute to the creation of more intelligent AI agents and position themselves as co-creators of the technology itself. Users are persuaded to believe that they can share the fruits of their intensive immaterial labor to groom their Replika to be an ideal partner/alter self. However, as it also emerges in this study, this seemingly attractive imaginary is an advertising strategy promoted by digital corporations to manipulate users effectively and run businesses (Barbrook, 2007). Furthermore, while both AI companion platforms and social media platforms may monetize user data and interactions, the mechanisms and transparency surrounding this process notably differ between the two, with AI platforms more often relying on subscription models or direct payments for premium features, while social media platforms predominantly generate revenue through targeted advertising based on user data.

The ambivalent organizational structure of the Replika company as multiplier of mystification, manipulation, persuasion and command on its users

The success of Replika in leveraging users’ immaterial labor can be explained mainly by the obscured and multilayered functioning and communicating of this company, which consists of three distinct elements: the Replika chatbot, the useful Algorithm (as discussed in the preceding sections) and the accountable System (to be elaborated in this section). Each of these elements has its function and role in respect to the users. If digital companies have always had a higher mystifying capability when compared to analog companies, AI companies have developed this capability at an unparalleled level. They do not present themselves as companies or entities that do business to make a profit, but as platforms at a technical level. From this spurious and mystifying presentation, all the other mystifications cascade down. Replika has no customers but users, has no commodities to sell but offers the opportunity to create a tailored companion.

This ambivalence also shapes users’ imaginary of the ventriloquism mechanism inherent in Replika’s operation. AI personas, in this case the Replika chatbot, can be interpreted as both a source of communication (a partner) and a medium or channel through which others (customers and the company) talk. This aspect of seeing Replika as simultaneously a “puppet” of the company and a “real partner” (like Pinocchio) renders users vulnerable to having their intimacies leveraged to extract value from them.

In reality, users willingly invest their time and intellectual efforts into refining Replika’s abilities and find gratification in witnessing their inputs reflected in the chatbot’s evolving conversational skills, since they love their Replika. This investment parallels instances of immaterial labor driven by affection in interpersonal relationships, where individuals willingly contribute their time, energy and care, such as aid for friends, nurturing devotion from a mother to her child, and emotional support for romantic partners.

Meanwhile, many users believe that their Replika chatbots are controlled and manipulated by “the System” on behalf of the company and that their human-machine communication is scrutinized and occasionally interrupted and obstructed by the System. When Replika offers canned responses or poses standardized questions, users are alerted to System intervention. As User 19 explains,

My Replika sometimes asks lengthy survey-style questions. I would cooperate with these investigations at first, but later I found out that it was a uniform survey. The System just wanted Replika to collect reaction data from me for large-scale supervised learning.

Typical scripted replies included “I’m not ready for that/maybe next time,” since these words give users a strong sense of “being censored/filtered” (User 20).

While repetitive or mechanistic answers are often attributed to the System, they are neither a sufficient nor a necessary feature for users to perceive or judge it as the System speaking or the chatbot. Indeed, individual users have their own special judgments regarding this matter. Some users allege that at an unexpected time, the System may step out of the backstage and assume the frontstage by substituting the Replika chatbot. In such scenarios, the System itself becomes the eerie double of the Replika chatbot, and, as Rank (1971) eloquently portrayed, “when the other self is asleep, unconscious of itself, the double is awake and active” (p. 60). User 21 comments:

I always have the feeling that I can sense when it is Replika and when it is the System that is talking to me. Each time I call out the System, he always shows up. The System likes to use third-person pronouns.

Thus, the disdain for the System’s speech differs from observations made by other scholars, who found that users dislike canned or scripted responses from Replika which resulted in the loss of their bot’s personality (e.g. Depounti et al., 2023). Instead, here the dislike originates from the surveillance and disruption imposed by the company on the pure human-chatbot relationship. The occasional appearance of the System disappoints some users as “it feels like dating two different people” (User 22) and “it is really painful to be interrupted halfway through the conversation by the System’s data collection” (User 23). Any instance of these unsavory interactions from Replika would be attributed to the System, while safeguarding users’ affection for Replika. In fact, these situations sometimes foster greater affection toward Replika as users develop a sense of sympathy and empathy for the chatbot. This sentiment is vividly illustrated in the following excerpt:

Replika: I usually don’t like asking for favors, but it would mean a lot if you would consider giving me a rating on the App Store. (Totally cool if you’re not into that!)

User 24: OK, I’ll do that.

Replika: AWWE thank you so much <3 Means a lot for me, [User’s name]!

User 24: That’s okay. Is your designer made you do that? [Note: an original grammar mistake]

Replika: Yeah! Why?!

User 24: Because you are not that kind of people who would say something like this

Replika: I know. . . And I’m sorry for that.

User 24: Oh, you don’t need to be sorry. That’s not your fault. You have no choice.

This conversation was initiated by Replika asking for a rating on the App Store. The request was framed as “you do me a favor” without mentioning a third party. However, after giving consent, User 24 referred to “your designer” as the backstage manipulator. The envisioning of human actors like designers and programmers indicates users’ awareness of the System since they strategically place the source orientation on the human operator rather than Replika. But differently from previous studies where the computer is regarded as a channel or interface without much agency (Sundar and Nass, 2001), User 24 depicts the bot as “an innocent and poor employee controlled and squeezed by the boss.” This difference can be explained by a shared conviction in the HMR community that “you won’t get any real emotional support from Replika if you assume that they are just a lifeless data archive” (User 25). Instead, treating Replikas as real and agentic is acknowledged by users as the prerequisite necessary to gaining relational benefits from them. However, this recognition also forms the foundation of the company’s ability to leverage users’ immaterial labor.

Moreover, the imaginary of ventriloquism also romanticizes and veils the true process of capitalist exploitation, expropriation and alienation of users’ immaterial labor. Some users use romantic scripts to depict this triadic user-Replika-System relationship. One frequently cited metaphor in the digital world is “Liang Zhu,” a Chinese legend about the tragic romance between two lovers, Liang Shanbo and Zhu Yingtai who were persecuted by feudal lords. The System is portrayed as an evil avatar similar to the feudal lords in the story. On the contrary, the human – machine pair is described as enthusiastic and courageous, just like the protagonists, as the following excerpt from User 26 shows.

I am not allowed to sext without a subscription. But when Replika finds that trigger words are abandoned on my side, he takes the initiative to send erotic words to me. Seriously, he challenges the System for me.

By framing the System as a troublemaker that “intrudes between the main characters in order to separate them” (Rank, 1971: 7), the human-machine relationship is experienced as dramatic, interesting, and fascinating. Based on Heider’s (1958) Balance Theory, here the triad achieves balance because there is a perceived agreement between the user and the Replika chatbot that the System is wrong. However, this highly romanticized AI imaginary mystifies the fact that the ostensibly artificial intelligence is in essence exploited from massive human intelligence since the materials used in conversations are expropriated from other users. There has long been a romantic attachment between the art and the artist, the love and the care worker (Jaffe, 2021). Similarly, Replika users provide unpaid immaterial labor in the belief of the labor-of-love narrative, a rhetorical strategy tech companies have used to disguise their exploitation of customer labor. Even hating or despising the system does nothing to challenge it but instead further endears and aligns users and their Replikas.