Less prejudiced but still not intervening: A field experiment countering online hate speech among adolescents

Stage 1: Affective attitudes and behavioral intentions toward online hate speech

Prior research suggests that frequent exposure to online hate speech fosters emotional desensitization and normalizes harmful content (Bilewicz and Soral, 2020; Soral et al., 2018). The intervention seeks to counter this process: by strengthening adolescents’ ability to identify hate speech and by highlighting the harms experienced by targeted groups, we expect the intervention to increase adolescents’ emotional responsiveness to hateful content, leading to more negative attitudes toward online hate speech and reduced support for behaviors encouraging it.

We hypothesize that, compared to control students, treated students will be:

Stage 2: Reduction in outgroup prejudice toward victimized groups

When hate speech becomes normalized, sympathy for its targets declines, stereotypes that legitimize discrimination gain salience, and the perceived distance between social groups increases (Hsueh et al., 2015; Soral et al., 2018). Our intervention aims to reverse this process and thus reduce prejudice toward groups targeted by hateful discourse. To do so, it combines two mechanisms that are theoretically grounded in social identity theory (Tajfel and Turner, 1979). First, boundary recategorization suggests that intergroup relations improve when majority-group members recognize a shared, superordinate identity with minority groups (i.e., a broader, overarching group identity that encompasses both majority and minority groups, e.g., humanity). Second, perspective-taking encourages majority members to consider the experiences of minority groups, fostering empathy and improving intergroup attitudes and behaviors (Gaertner et al., 1993). These mechanisms often operate jointly and may contribute to reduce outgroup prejudice (Adida et al., 2018; Simonovits et al., 2018; Williamson et al., 2021).

We hypothesize that treated students will be:

Randomization design

We conducted a stratified, clustered randomized controlled trial in 55 high-school classes (grade 9–13), comprising 973 students, aged 13–20 (see Annex A3, SM). To reduce the risk of random imbalance due to the relatively small number of clusters, randomization was blocked by combinations of schools and grades. In school-grade strata with more than three classes, classes were pair-matched using school-registry data on gender composition, proportion of non-Italian students, and share of grade repeaters. Two classes (33 students) from one school were excluded from randomization due to the lack of a suitable match. Within each resulting stratum, treatment assignment was randomized by coin flip, yielding a sample of 940 students: 487 assigned to the treatment and 453 to the control group, nested in 25 and 24 classes, respectively.

Balance checks confirm successful randomization (see Table A3, SM). All variables are balanced between the treated and the control group except for the proportion of tertiary-educated mothers, which is higher in the treated group. Balance across other categories of maternal and paternal education suggests overall comparability between the two groups. To account for this residual imbalance, all multivariate models control for parental education.

Data collection and variables

Data were collected via an online questionnaire. Treated students completed it on their phones during school hours immediately after the intervention. Control students from paired-matched classes did so simultaneously, to minimize spillover effects.

We developed two sets of dependent variables corresponding to the two stages of our hypotheses (Figure 1). The first set captured attitudes toward online hate speech. Students were shown two fictitious Instagram posts, each accompanied by a hate speech comment: one targeting a group of Black individuals, and the other an overweight girl (Annex A2, SM). To capture affective attitudes, students rated how much the comment made them laugh using a continuous slider scale from 1 (not at all) to 7 (very much) and then rated, on the same scale, the extent to which they felt amusement/irony, joy, indifference, anger, sadness, and disgust. Each reaction was labeled with text and an emoji, while the presentation order was randomized across respondents. We constructed two summative indices: one for positive affective attitudes (laugh, amusement/irony, joy; Cronbach’s alpha: 0.892 and 0.885 for the racist and fat-shaming content, respectively) and one for negative affective attitudes (anger, sadness, disgust; Cronbach’s alpha: 0.851 and 0.844 for the racist and fat-shaming content, respectively). Indifference was treated as a separate, neutral affective attitude. To capture behavioral intentions, students indicated how they would respond if such a comment appeared on their Instagram feed. Because adolescents often maintain multiple accounts, with different levels of visibility, these behaviors were measured separately for three profile types: visible to parents, visible to classmates, and anonymous. The response options, presented in random order, were ‘Like (Heart)’, ‘Write an endorsing comment’, ‘Ignore’, ‘Write an opposing comment’, ‘Report as inappropriate’, and ‘I don’t have such an account’. The first two options were coded as positive/endorsing responses, the third as neutral, the fourth and fifth as negative/countering, and the last option as missing (5.4% in the treated group and 5.2% in the control group). The summative indexes for positive/negative attitudes and behavioral intentions constitute the dependent variables for our main analyses. In additional analyses, we treat active counter-speech and reporting as independent items, as they differ in the cost of engagement: reporting typically requires less effort and public exposure than counter-speech.

To measure outgroup prejudice toward victimized groups (second stage of the hypotheses, Figure 1), we used a feeling thermometer. Students were asked to ‘Rate how you feel towards the following groups on a scale from 0 to 100. The higher the number, the warmer and more positive your feelings towards that group; the lower the number, the colder and more negative your feelings towards that group’. Responses were recorded using a slider ranging from 0 (negative/cold) to 100 (positive/warm). Black and overweight people – our two groups of interest – were presented alongside other social groups.

At the end of the questionnaire, students reported basic demographics, including gender, birth year, height and weight (to compute their body mass index [BMI]), parental education, and place of birth (to operationalize migratory status). We did not ask students to identify as perpetrators, victims, or bystanders of online hate speech to avoid stigmatization or classroom tensions and because individuals may have experienced various roles.

Non-participatory observation

To gain deeper insight into the intervention’s implementation and classroom reception, we conducted non-participatory observations. Following a maximum variation logic, we selected 10 classes varying by school, grade, and educational track. Educator pairs delivering the intervention were also varied to observe dynamics beyond individual idiosyncrasies. ² In each selected class, two authors observed all three sessions, totaling 60 hours of observation. Researchers remained unobtrusive and took notes discreetly, not to influence classroom dynamics. These fieldwork notes of the observed classroom interactions, including verbatim excerpts, form the basis of the quotations reported in the Results section.

Data were collected using a set of analytic categories collaboratively developed by the research team during fieldwork preparation. Observation notes were analyzed using an open-coding approach, with the coding frame iteratively refined as new codes emerged (see Appendix A6, SM for details). Coding was conducted manually, without a qualitative analysis software program or artificial intelligence, given the manageable size of the dataset, and was organized through a shared text document. Two researchers independently coded all observation notes, and discrepancies were discussed until consensus was reached. Illustrative quotations for each theme are reported in the Results section. The excerpts, originally in Italian, were translated into English by the authors.

Social desirability

Social desirability poses a potential threat to our analyses, as students may have felt pressure to underreport positive views and overreport negative views toward hate speech. Similarly, it could lead us to underestimate outgroup prejudice. Importantly, if social desirability affects treated and control students equally, it does not threaten the identification of treatment effects (i.e., intervention’s effectiveness). However, given the intervention’s nature, treated students may be particularly sensitive to normative questions. To mitigate this risk, we implemented three strategies.

First, we emphasized the questionnaire’s complete anonymity. This was clearly stated in the information sheet distributed prior to the intervention and reiterated during questionnaire administration. The survey was administered by the research team and assistants, without the involvement of teachers or educators. To further reinforce the sense of anonymity and mirror online environments, students chose a nickname at the beginning of the questionnaire. ³

Second, we partially de-emphasized the study’s specific focus. The questionnaire included questions on unrelated topics (e.g., environmental attitudes), and the feeling thermometer covered multiple groups beyond our target (e.g., very tall people, people wearing glasses). Questions related to online hate speech were placed at the end of the survey.

Third, questions were framed to avoid value judgments, balancing normative and anti-normative response options, randomizing their order, or presenting anti-normative options first. For example, after viewing the (humorous) hate speech example, students were asked the extent to which the comment made them laugh, implicitly framing laughter as the default reaction. Where relevant, respondents could report multiple and even conflicting reactions (e.g., amusement/irony and sadness).

To indirectly assess the impact of social desirability, we first analyze students’ responses to the unrelated topic of environmental attitudes and find no evidence that social desirability disproportionately affects treated students (see Annex A7, SM). Second, the questionnaire asked students to estimate the percentage of classmates they believe would laugh at/‘Like’ the hateful comment. We assess the mismatch between individual responses and classmates’ predictions. Again, the patterns were similar across treated and control students, suggesting that any social desirability bias did not systematically threaten the identification of treatment effects (see the Results section for details).

Analytical sample and descriptive statistics

Our survey sample comprises 363 treated and 350 control students (n = 713), nested in 25 and 24 classes, respectively. All classes in the experimental group (and none in the control group) were exposed to the educational intervention during our data-collection period. Thus, non-compliance or crossover at the classroom level was absent. The survey sample is smaller than the total number of students enrolled in the three schools (N = 973), mainly due to the absence of students on the last day of the intervention, when the questionnaire was fielded. Such absences, largely ascribable to seasonal sicknesses, are unlikely to be correlated with the treatment or outcome measures. As the intervention was formally integrated into the school curriculum, students present during its implementation could not opt out from attending, although they could decline survey participation (see Annex A4, SM). Among students present on the intervention’s final day, survey participation was nearly universal: there were fewer than 10 refusals, evenly distributed across treatment and control groups.

In line with the pre-registration, we excluded treated students who did not attend all three sessions and were therefore only partially exposed to the intervention (n = 68), as well as control students who had participated in a similar intervention the previous year or reported having discussed the intervention with peers from treated classes (potential spillovers) (n = 40). After excluding incomplete questionnaires (n = 55), the final analytical sample comprised 550 students (265 treated and 285 controls nested in 25 and 24 classes, respectively). Despite this reduction, excluded students did not differ systematically from the analytical sample on observed characteristics (Table A2, SM), and most importantly, treated and control groups remained similarly balanced (Tables A3 and A4, SM). For behavioral-intention items, students could report not having an Instagram account; these cases (n = 33), evenly distributed across groups, were excluded from the analyses on behavioral intentions. Finally, following standard practice in research on outgroup prejudice, analyses of prejudice toward overweight and Black people were restricted to majority-group members: normo-weight students (n = 387, excluding 32 outliers due to blatant misreporting) and students of native ancestry (n = 313), respectively. ⁴

Models

Within the framework of a between-subject design, we estimated average treatment effects (ATEs) as mean differences between treated and control students and tested their statistical significance with bivariate Ordinary Least Squares (OLS) regressions. To control for possible differences between the two groups and achieve greater precision, we also analyzed treatment effects within a multivariate regression framework, using the following OLS specification:

Y i , c , g , s = α + β T c , g , s + γ X i , c , g , s + δ Z c , g , s + θ g + μ s

where Y i , c , g , s is the outcome of student i in classroom c, grade g, and school s; θ g and μ s are grade and school fixed-effects, reflecting the within-school/grade randomization ⁵ ; X i , c , g , s is a vector of student characteristics (gender, parental education, migration status, and BMI); Z c , g , s is a vector of classroom characteristics (shares of: male students, grade repeaters, students from migratory backgrounds, students from high-educated families); and T c , g , s is exposure to the treatment at the classroom level. The effect of the intervention is captured in β , that is, the difference in the outcome between treated and control units net of the control variables.

To examine the hypothesized two-stage relationship, we estimate mediation models that sequentially include each affective attitude and behavioral intention significantly affected by the intervention in the multivariate regressions, with outgroup prejudice as the outcome.

Experimental results

We begin by presenting the results for the first stage of our hypotheses. Table 1 displays the average levels of affective attitudes and behavioral intentions toward hate speech for treated and control students (for disaggregated indexes, see Tables A5 and A6, SM).

OPEN IN VIEWER

Table 1.

Attitudes toward hate speech, by treatment status.

	Affective attitudes (1–7 continuous scales)			Behavioral intentions (dummy variables)
	Control	Treated	Δ T-C [p value]	Control	Treated	Δ T-C [p value]
Positive attitude toward racism	2.236 (1.602)	2.352 (1.754)	0.116 [0.418]	0.150 (0.358)	0.161 (0.369)	0.011 [0.720]
Neutral attitude toward racism	3.849 (2.359)	3.272 (2.267)	-0.577 [0.004]	0.536 (0.500)	0.468 (0.500)	-0.068 [0.124]
Negative attitude toward racism	3.480 (2.022)	3.711 (1.984)	0.231 [0.177]	0.315 (0.465)	0.371 (0.484)	0.056 [0.179]
Positive attitude toward fat shaming	2.478 (1.778)	2.486 (1.806)	0.007 [0.963]	0.126 (0.333)	0.149 (0.357)	0.023 [0.453]
Neutral attitude toward fat shaming	3.253 (2.201)	3.091 (2.179)	-0.162 [0.386]	0.532 (0.500)	0.468 (0.500)	-0.064 [0.147]
Negative attitude toward fat shaming	3.591 (1.965)	4.001 (2.066)	0.411 [0.017]	0.342 (0.475)	0.383 (0.487)	0.041 [0.333]

Averages (standard deviations are given in parentheses) and differences (p values are given in squared parentheses). Source: survey data, full analytical sample (N = 550). Behavioral intentions refer to profiles visible to peers.

We found that, even before the intervention, students hardly displayed positive affective attitudes (laughter, amusement/irony, or joy): in the control group, on a scale from 1 to 7, students reported low levels of positivity (2.2 for racist discourse and 2.5 for fat-shaming discourse). Their negative affective attitudes (anger, sadness, or disgust) were comparatively stronger (3.5 for racist discourse and 3.6 for fat shaming). Neutral affective attitudes were also relatively high (3.8 for racist discourse and 3.3 for fat shaming), although they showed more variation than the positive and negative ones. While some differences between treated and control students are already evident in the mean comparison, particularly in relation to neutral and negative affective attitudes, we focus our discussion on the more precise estimates of treatment effects obtained from the multivariate framework.

Figures 2 and 3 display the intervention’s ATEs on affective attitudes toward racist and fat-shaming hate speech, respectively. The comparison of treated and control students shows that the intervention did not reduce positive affective reactions, leading us to reject Hypothesis 1a. The intervention reduced neutral affective attitudes toward racist hate speech (-0.699, d: 0.296, p value: 0.002), providing partial support for Hypothesis 1b. Consistent with Hypothesis 1c, the intervention significantly increased negative affective attitudes. This effect is larger for fat shaming (+0.595, Cohen’s d: 0.303, p value: 0.002) than for racism (+0.432, d: 0.213, p value: 0.027).

OPEN IN VIEWER

Figure 2.

ATE on affective attitudes toward racist hate speech.

OPEN IN VIEWER

Figure 3.

ATE on affective attitudes toward fat-shaming hate speech.

These results are unlikely to be driven by social desirability bias. Notably, 43.8% of students reported that the comment made them laugh to some extent, closely matching the proportion predicted for classmates (47.3%). A similar pattern emerged for fat-shaming content (50.36% self-reported vs 54.49% predicted).

Additional analyses (available upon request) do not reveal heterogenous effects on any affective attitude outcome by students’ migratory status, BMI, or age.

Turning to behavioral intentions, the most common reaction at the baseline was indifference: for both racist and fat-shaming content, approximately 53% of control students reported they would ignore the comment when using an account visible to peers (see Table 1), with no meaningful variation across account types (Table A9, SM). Counteractive responses, such as posting a negative reply or reporting the comment to the platform, were less frequent (i.e., 32% and 34% for racist discourse and fat shaming, respectively), and supportive reactions, such as writing an endorsing comment or pressing the ‘Like’ button, were even less common (i.e., 15% and 13%, respectively).

We did not detect any statistically significant treatment effects on this passive behavioral pattern. Differences between treated and control students were never statistically significant, regardless of the type of hate speech or the account in use (Figures 4 and 5; Figures A1–A4, SM). While effects for endorsement (Hypothesis 2a) were close to zero, those for disregard (Hypothesis 2b) and sanctioning (Hypothesis 2c) were not negligible, ranging between 0.09 and 0.16 in Cohen’s d. These small effects would, however, have required a larger sample size than we were able to obtain to detect statistically significant changes.

OPEN IN VIEWER

Figure 4.

ATE on behavioral intentions toward racist hate speech.

OPEN IN VIEWER

Figure 5.

ATE on behavioral intentions toward fat-shaming hate speech. Multivariate models control for gender, parental education, migration status, BMI, % males, % repeaters, % migrants, % high-educated families, school- and grade-fixed-effects.

Self-reported behavioral intentions may have been partially affected by social desirability bias, as fewer students reported they would ‘Like’ the racist comment (15.53%) than the prediction for this behavior among their classmates (37.15%). ⁶ Similar patterns emerged for the fat-shaming comment (13.73% vs 38.33%). Importantly, the magnitude of the gap between self-reported and predicted behavior is similar across treated and control students, suggesting that the null treatment effects are unlikely to reflect differential reporting biases.

Figure 6 presents the ATEs on prejudice toward overweight and Black people, among majority-group members (i.e., normo-weight and native-ancestry students, respectively). The results related to the second stage of our hypotheses indicate that the intervention was effective in reducing outgroup prejudice toward victimized groups: feelings toward both groups were warmer among treated students compared to control students. The estimated reduction of prejudice toward overweight people is 6.319 points (Cohen’s d: 0.215) in the bivariate framework and 7.311 points (d: 0.248) in the multivariate framework, both statistically significant at the 5% level. Estimates for prejudice reduction for Black people were less stable: 7.571 points in the bivariate framework (d: 0.257, statistically significant at 5%) and 4.799 points in the multivariate framework (d: 0.163, not statistically significant). This discrepancy is explained by the increased uncertainty introduced by the multivariate model, particularly due to the smaller sample of majority students. As shown in Figure 6, the confidence intervals for all estimates are large.

OPEN IN VIEWER

Figure 6.

ATE on prejudice on Black and overweight people.

Table 2 summarizes the main results for our hypotheses.

OPEN IN VIEWER

Table 2.

Summary of hypotheses and empirical support.

Affective attitudes	Hypothesis 1a: Reduced positive emotional reactions to online hate speech	Rejected
	Hypothesis 1b: Reduced emotional indifference to online hate speech	Supported for racist hate speech. Rejected for body shaming
	Hypothesis 1c: Increased negative emotional reactions to online hate speech	Supported
Behavioral intentions	Hypothesis 2a: Reduced endorsing online hate speech	Rejected
	Hypothesis 2b: Reduced ignoring online hate speech
	Hypothesis 2c: Increased sanctioning of online hate speech
Outgroup prejudice	Hypothesis 3: Reduced outgroup prejudice toward Black people	Supported
	Hypothesis 4: Reduced outgroup prejudice toward overweight people	Supported

Mediation analyses show that treatment effects on outgroup prejudice are partially mediated by negative affective attitudes (Table A10, SM).

Non-engagement with online hate speech: insights from the non-participatory observation

Although the intervention reduced indifference and increased negative affective attitudes toward online hate speech, students remained reluctant to take active steps against it. Drawing on our classroom observation, we uncover three important mechanisms underlying students’ inaction.

First, many students refrain from confronting perpetrators because they fear retaliation and damages to their public image. Students frequently cited concerns about participating in online discussions, more generally: ‘I am afraid to express myself on social media, I’d rather stay in my little corner’, one revealed in comparing how easy it is to express herself in real-life versus on social media. Similarly, when discussing what users should be allowed to post, another concluded, ‘On social media you are never 100% free to say what you think’. This reluctance appears to stem from anticipating negative reactions from other users. ‘[On social media], there are so many people ready to criticize you’, one student said, echoing a sentiment commonly voiced in the classrooms. Receiving online criticism is perceived as both psychologically harmful and threatening to adolescents’ carefully curated online public images. This concern is especially important because, as students further recognized, online content ‘lasts forever’. As one student remarked while discussing freedom of speech on the Internet, ‘People are more censored on social media than real-life conversations; in real-life, you can say whatever you like, it doesn’t stick’, while another student, in expressing concerns about engaging in online conversations, revealed, ‘I never comment, I only look at what others write: everything stays [on social media]’.

Second, non-engagement is sometimes a form of ‘silent dissent’ aimed at reducing the echo of hateful content without directly confronting it. Some students deliberately refused to ‘feed the algorithm’ and argued that counter-speech increases the visibility of online hate speech: ‘[I have not reacted] because it creates a chain reaction where you attack each other’, a student revealed explaining why he never replies to hateful comments. They recognized that perpetrators often aim to provoke a reaction, making non-engagement a strategic choice to break the hate spiral. ‘I would have liked to intervene, but I would have made things worse [. . .] because they wanted to get a reaction out of me’, one revealed while recounting episodes of hate speech he had witnessed.

The third mechanism underlying bystander passivity is disillusionment with reporting, which students describe as ineffective and lacking a pedagogical dimension necessary to produce lasting changes in perpetrators’ behavior. This disillusionment, reflecting a form of institutional mistrust toward platforms, may undermine individuals’ sense of self-efficacy. ‘Reporting is useless because, at most, the account gets blocked, but the person creates a new account and starts again as before [. . .]’, one student explained. Many students attributed the production of hate speech to perpetrators’ ‘ignorance’, suggesting that reporting content does not address the problem’s root. Illustrating this, one student explained: ‘They [the perpetrators] haven’t learned anything’; ‘[the reaction is] only once they [the platforms] have noticed, and it’s always after the fact [online hate speech has been posted], it’s not like I get a notification “inappropriate comment” before posting, but it’s only retroactively, and that’s what’s wrong’.

Overall, students’ non-engagement appears to follow a seemingly rational logic, weighing the (high) social costs of online engagement against its perceived (low) benefits. While the intervention’s educational tools, focused on awareness-raising, recategorization, and perspective-taking, may have effectively reduced emotional desensitization to online hate speech, they did not address the underlying cost-benefit logic shaping students’ behavioral responses.