What are the effects of different elements of media on radicalization outcomes? A systematic review

The media-radicalization relationship

Since the early days of terrorism research, scholars have consistently emphasized the integral role of the media as a key factor in promoting terrorism. The media's reporting on and portrayal of terrorist attacks and groups provides them with “free advertising.” This exposure provides terrorists, groups, and their proponents with a vehicle for highlighting their grievances and spreading their message. In doing so, terrorists aim to attract new supporters and recruits, and strengthen their image among their current supporters (Weimann, 2005). The media also reports on and portrays a range of social issues that may highlight forms of discrimination or violence against different groups—both locally and abroad—which can increase feelings of anger, deprivation, and identification with the oppressed group. All these factors can increase the likelihood that an individual will support what they perceive to be the use of defensive violence, or even engage in it themselves (Wolfowicz et al., 2020).

These types of effects have received extensive attention in the ever-growing literature on individual radicalization, which focuses on the factors that increase (or decrease) the likelihood of the development of attitudes that justify terrorism, and which could therefore underpin a turn to radical behaviors such as terrorism. In this literature, which has primarily developed only in the last 15 years, research on the role of traditional media has shifted to focusing on the role of new media, namely the Internet, and social media in particular. These medias have been successfully leveraged by radical groups and their supporters to spread their message, greatly increase their support base, and from this base generate new recruits (Scrivens & Conway, 2019).

There is little doubt that media exposure in general can and does impact a range of cognitive outcomes, from emotions to attitudes and perceptions, as well as behavioral outcomes, whether on account of cognitive changes, imitation, or other mechanisms. An extensive literature exists that demonstrates these effects with respect to a range of deviant, criminal, and violent cognitive and behavioral outcomes. However, despite what seems to be a significant amount of attention given to the topic of media effects on radicalization, which encompasses a form of violent cognition (a positive evaluation of violence) and behavioral outcomes (Wolfowicz et al., 2020), there is a worrying lack of quantitative research (Gill et al., 2017; LaFree, 2017; Whittaker, 2018).

Meta-analytic contributions to the study of media effects

While there are some exceptions (e.g., Ferguson et al., 2020), over the years there have been a slew of meta-analyses that have produced fairly consistent results showing that exposure to violent television (and movies) has a modest effect on violent cognitions and behaviors (see Comstock et al., 2014 for a review of previous systematic reviews). When it comes to the Internet and social media as mediums, most meta-analytic research point to small relationships (of about r = 0.10) with psychological outcomes—such as self-esteem and depression (Appel et al., 2020). However, when it comes to delinquent behaviors, average effect sizes of r = 0.21 have been found (Vannucci et al., 2020). This effect size is almost identical in magnitude to the effect sizes that have been found for exposure to television violence (Comstock et al., 2014). Other forms of media exposure have also been found to have pooled effect sizes of a similar magnitude on analogous outcomes, such as the effects of music on anti-social behaviors (Timmerman et al., 2008) and violent video games (e.g., Comstock et al., 2014; Ferguson et al., 2020).

The findings mirror those of other areas of investigation into risk factors for related cognitive and behavioral outcomes, (such as violent cognitions and behaviors, or suicidal thoughts and behaviors), in which effect sizes are almost always larger for the cognitive outcome—which is generally more prevalent. For factors where effect sizes are found to be larger for the behavioral outcomes, it has been suggested that this could be an indication of them serving as mediators of the continuity between the given cognitive and related behavior (Wolfowicz, Weisburd, et al., 2021a). Of all media-related factors, such a finding has been made only with respect to violent video-games (e.g., Comstock et al., 2014; Ferguson et al., 2020). This evidence points to a small but potentially important role for at least some mediums to play a role not only in the risk of development of deviant cognitions and behaviors but as a mediator, in which in increases the risk of deviant behavior for those who already possess the related deviant cognition (Ajzen & Fishbein, 1980; Bosco et al., 2015; Wolfowicz, Weisburd, et al., 2021a).

Several lessons can be learned from the collective results from the literature. First, the magnitudes of the effects between television and video games are highly similar, even though the mediums are fundamentally and qualitatively different. While television offers a one-way, passive form of exposure, videogames offer a two-way passive-active form of exposure. Secondly, the magnitudes of the effects for most media-related risk factors are considerably smaller than analogous risk factors from the offline domain, such as social learning factors relating to exposure to violence, and having deviant associations. Third, the magnitudes of the effects for media-related risk factors do differ based on the nature of content. This indicates that mere exposure to or usage of a medium is not sufficient. Rather it is the mediated exposure to specific types of content across different mediums that increases risk of deviant outcomes to varying degrees (Anderson & Dill, 2000; Anderson et al., 2007; Coyne et al., 2018; Gentile et al., 2004; Martins & Weaver, 2019).

Types of studies

The review sought to identify, collate, and synthesize observational studies, namely cross-sectional, longitudinal, and case-control designs. The outcomes from these studies were expected to be derived primarily from self-reports. However, the review also sought to include studies based on clinical reports (e.g., practitioner coded data), administrative data (e.g., from security services), and secondary databases constructed from multiple open sources (e.g., Profiles of Individual Radicalization in the United States [PIRUS]).

Experimental studies were included when the experimental manipulation involved some variation (e.g., at least two conditions) of some form of human-media interaction, such as exposure to news-media, videos. That is, the experimental condition differed in terms of the type of media that participants were exposure to, or the contents of the media that participants were exposed to assess differences in one of the eligible outcomes. This included lab-based experiments, Internet-based experiments, and experiments using vignette designs. For these types of studies to be included, the treatment or control must have included at least one condition in which the exposure was theorized to increase the likelihood of radicalization. These criteria also served to distinguish the studies and this review from research (including systematic review) on counter-narratives (e.g., Carthy et al., 2020). While these studies' outcomes were also expected to be derived primarily from self-reports, they were also open to inclusion if based on clinical reports or observations (e.g., practitioner coded data). These studies were eligible for inclusion irrespective of the method of assignment (e.g., random, random blocked).

For all study types, it was necessary for them to display variation on the dependent variable, (e.g., a proportion of the sample displayed, or didn't display the outcome of interest in the case of dichotomous measures or displayed it to varying degrees in the case of ordinal, discrete or continuous measures).

Cross-sectional and longitudinal studies were included without any limitations on the nature of the sample (e.g., gender or age composition) or sampling method (e.g., Random, Representative, Quota, Convenience, Purposive, Snowball etc.).

For case-control designs, studies were included when the “case” sample was made up of a group of individuals who displayed one or more of the outcomes of interest (see below “Types of outcome measures”) as assessed by self-reports, clinical reports, or administrative decisions (such as a clear violation of the law in line with the outcomes of interest described below). For these studies to have been included, it was necessary that the control sample would be made up of one of the following types of individuals: 1)

Individuals assessed for or who were suspected of having displayed one of the outcomes of interest but who were found not to, or who were found to display it to a lesser degree than the “case” sample or some chosen threshold.

For these studies, controls could be chosen based on having been part of a single cohort (as in the case of prospective studies) or derived from a relevant data set (such as administrative records). Studies were also eligible for inclusion if the matched controls were derived from a separate, related data set, as is often used in retrospective studies. Studies were also eligible for inclusion when the control sample was derived from the general population and included individuals considered to be at risk for displaying the outcome of interest specifically, or a distribution of individuals displaying or not displaying (or displaying to varying degrees) the outcome of interest.

For longitudinal studies, we included both prospective or retrospective studies, as well as whether they are cohort or panel studies, that measured the indicator and outcome of interest at least two different time points. We set no limits on the time between measurement but expected it to be a period of at least a few months for a study to be considered to be a true longitudinal design. For cohort studies, whether prospective or retrospective, we included studies in which the cohort had some shared characteristic, such as being from a specific age group, geographic setting, or social setting at the time of selection. For panel studies, whether prospective or retrospective, we included studies irrespective of their sampling (e.g., Random, Representative etc.).

Observational studies and experimental studies will be treated separately in the analysis.

Types of participants

Previous reviews of the literature have highlighted the dearth of research concerning the media-radicalization relationship at the individual level. It has been found that most of the research focuses on the Internet, and specifically on platforms, content, groups, and networks. Whilst important, this type of research represents a distinct line of inquiry and is better described as the analysis of “online extremism” rather than “radicalization” (Odag et al., 2019). This type of research does not provide the type of information—due to the units of analysis—that can inform us about the media effects on radicalization (Winter et al., 2020).

As such, given that previous reviews have already dealt with this distinct line of inquiry, the current review sought to address the gap in the literature concerning the individual level. The review therefore included studies in which the participants, or units of analysis were individuals. The review therefore excluded studies where the unit of analysis was some form of media content, network(s), group(s), or other units of analysis that were unable to provide information pertaining to this review's research questions and objectives.

The review placed no restrictions on inclusion based on the individual characteristics of the participants. The review included all samples regardless of the participants' age, gender, ethnicities, religion, or the type of radicalizing doctrine being investigated (e.g., right-wing, left-wing, religious etc.). Previous systematic reviews have found that such factors rarely impact the pooled estimates in meta-analyses of indicators pertaining to the same radicalization outcomes taken up by the current review (Wolfowicz et al., 2020). As such, while we combined studies examining the same indicators into a single analysis regardless of conceptual differences between the participants, where possible we explored participant population factors as possible sources of heterogeneity using meta-regression, and moderator analysis (see section on “Assessment of Heterogeneity”).

Types of factors

The review sought to include studies that examine at least one media-related factor as an independent variable that provides an indication of correlation with one of the outcomes of interest. A media-related factor was considered to be an exposure or experiential based indicator whose source is in some form of media. For example, a known risk factor for radicalization is exposure to violence (Wolfowicz et al., 2020). This review considered in-person exposure to violence to be excluded but mediated exposure to violence (through television, social media, or other forms of media) would be an eligible indicator. In line with the review's objectives and research questions, the review made no predeterminations as to what media-related factors may exist. Nevertheless, in Table 1 we provide some examples of the types of factors that were expected to be identified.

Types of outcome measures

In line with the literature and previous systematic reviews (e.g., Wolfowicz, Weisburd, et al., 2021b), this review examined two distinct, albeit interrelated outcomes, namely the cognitive and behavioral outcomes of radicalization. To ensure that the review included studies with comparable outcomes, the review limited its inclusion to studies whose dependent variables were assessed to be in line with the outcomes of McCauley and Moskalenko's (2017) Two-Pyramid Model of radicalization (TPM). Broadly speaking, the TPM was selected because rather than being a traditional process models of radicalization, it is an outcome-based typological model. The TPM differentiates between the cognitive (attitudinal) and behavioral outcomes of radicalization, and also distinguishes between legitimate, legal forms of activism that are often incorrectly conflated with radicalization, and the types of illegal, radical behaviors that are of concern to society and authorities (Moskalenko & McCauley, 2009). The TPM's typologies provide a clear set of criteria for determining if an outcome falls in to one of its categories, as well as for the categorization of the outcome (Wolfowicz et al., 2020). Moreover, when studying related cognitive and behavioral outcomes in parallel, the cognitive outcome should have a high level of specificity with reference to the behavioral outcome (Fishbein & Ajzen, 1975). This criterion is met by the TPM's typologies as described below (Figure 1).

OPEN IN VIEWER

Figure 1

Two-pyramid model (McCauley & Moskalenko, 2017).

In line with the TPM, studies were classified as examining either cognitive or behavioral radicalization outcomes, with each of these outcomes including two sub-categories. With respect to cognitive radicalization the review included studies that examined radical attitudes or intentions. Studies examining radical attitudes were included when the dependent variable assessed support for, or justification of the use of radical violence, which is the use of violence toward persons in the name of a cause. This definition strikes a good balance between specificity and sensitivity, leaving room for the inclusion of violence motivated by a range of ideologies, and excluding non-ideological forms of violence. Studies examining radical intentions were included when the dependent variable assessed intentions toward engaging in, or an expression of a willingness or desire toward the commission of radical violence. In both cases, studies were included irrespective of their use of single or multiple item outcome measures, if they included specific references to radical violence generally, or specific instances of radical violence. That is, while the review placed no specific restrictions on which measures were used, inclusion was limited to studies whose measures included such explicit reference to the use of some form of radical violence in the name of a cause.

While some reviews examining a broad, and large number of studies have previously assessed the relationship between individual factors radical attitudes and intentions separately (Wolfowicz, Weisburd, et al., 2021a), others have found it useful to combine them as a single measure of cognitive radicalization (Emmelkamp et al., 2020). Given the high degree of similarity in measurements used, and the fact that we did not expect as large a number of studies as has been found in previous reviews (e.g., Wolfowicz, Weisburd, et al., 2021a), we grouped studies examining radical attitudes and intentions together as a single outcome for cognitive radicalization. When a study reported on two separate outcomes for cognitive radicalization (e.g., radical attitudes and radical intentions), two separate analyses were conducted in which alternative effect sizes were used. Alternatively, when two or more studies contributing to a particular analysis reported on two or more outcomes, sub-group analysis was conducted. Where the analysis for any particular factor included effect sizes derived from at least two studies for each of the outcome categories (attitudes and intentions), moderator analysis was performed to assess the degree of between outcome measurement heterogeneity (see “Sub-group analyses and investigation of heterogeneity”).

With respect to behavioral radicalization, this review included studies that examined engagement in radical behaviors, which are sub-terroristic forms of illegal, radical violence (to be distinguished from legal, non-violent forms of activism), as well as studies that examined involvement in terrorism. Regarding radical violence, the review included studies that assessed self-reports of prior engagements in radical violence (e.g., Pauwels & Schils, 2016), as well as studies based on administrative data, or databases derived from both official and open sources. Regarding terrorism, the review considered terrorism to be any form of terrorism offending (planned or successful terrorism offending) as defined by the law of the country in which the offence occurred. The review also included studies whose samples were drawn from data from official and open sources (e.g., PIRUS). Studies examining both self-reported forms of radical violence and terrorism involvement were combined as a single outcome of behavioral radicalization. It was determined that in a case in which there were more than two studies from each of these categories in a single analysis, moderator analysis would be used to assess whether any between-outcome measurement heterogeneity exists (see “Sub-group analyses and investigation of heterogeneity”).

Duration of follow-up

While no limitations were placed on the duration of follow-up, it was determined that if a sufficient number of longitudinal studies would be identified, they would be grouped together by the time interval to follow up. We considered the length of time between two time points from 0 to 12 months as a short-term follow up period, 12–24 months as a midterm follow-up period, and 24> months as a long-term follow up period. It was determined that if a sufficient number of such studies were identified, a separate analysis would be conducted for each of the time point categories.

Types of settings

No limitations were placed on the settings from which the samples of studies originated. Where possible, moderator analysis was used to identify differences in effects between different countries or regional settings.

Electronic searches

Pilot searches were conducted to identify the most appropriate combination of search terms and strings that would achieve the optimal balance between sensitivity and specificity. To be able to properly address our first research question, we excluded from the search terms any reference to specific indicators/independent variables (e.g., television), so as to not to pre-determine or bias what factors would figure in the review (Shenderovich et al., 2016; Wolfowicz et al., 2020). Rather, in their place, we utilized general terms to capture all forms of media. At the recommendation of the editors from the Campbell Collaboration, who provided their input on the search strategy, we did include direct references to the Internet, as researchers are less likely to use terms like “media” to refer to the Internet. The search string also sought to ensure that the search results would be limited, as much as possible, to quantitative studies. During the revision process, searches were re-conducted using the revised search string:

Abstract/Title/Keywords = (radical* OR extrem* OR terror* OR action OR “politically motivated*” OR ideological*) AND (media OR technology OR internet OR online) AND (quantitative OR empirical OR survey OR regression OR multivariate OR correlation OR experiment* OR manipulat* OR coefficient* OR covariat* OR longitudinal OR paramet* OR predict* OR questionnaire* OR sampl* OR standard deviation* OR statistic* OR variable* OR variance)

Searches were carried out in the following databases in EbscoHost: ERIC, Criminal justice abstracts, Political Science Complete, Violence and Abuse abstract, and Open dissertations. In addition, searches were carried out in the following databases: PsychInfo (PsychNet), Worldwide Political Science Abstracts (ProQuest), Social Sciences Citation Index (Web of Science), and PubMed.

While the original protocol stated that searches would be carried out in the International Security & Counter-Terrorism Reference Center database as well as International Political Science Abstracts (IPSA), we were unable to access these databases. We opted to include the Worldwide Political Science Abstracts database, which was not included in the original protocol, given that it covers the IPSA.

Searches were carried out in August 2020 and were run again in March 2022 before publication of the review as part of the review process.

Searching other resources

Given that the topic of radicalization is known to be addressed by a range of disciplines (Wolfowicz et al., 2020), searches were carried out in several different sources. An initial search was carried out on the Campbell Collaboration and Cochrane libraries. We also carried out searches to identify existing reviews and screen their reference lists to identify possible studies that met the inclusion criteria. While the searches were being carried out, we also contacted several recognized experts (researchers and organizations) and presented them with our review topic, criteria, and initial results, with a request that they direct us to any missing studies that they may be familiar with that could meet the inclusion criteria. For included studies, which were subjected to full-text screening, we also reviewed their literature reviews and reference lists to identify whether they include any additional studies that may meet the inclusion criteria.

Selection of studies

After searches were performed, all search results were downloaded into Endnote X9 and stored in a shared library that all reviewers and research assistants had access to. A double screening process was implemented in which two reviewers screened the titles and abstracts of retrieved items to assess whether they related in any way to the topics of interest. The screening was performed directly in Endnote, where the reference window displayed the titles and abstracts. All studies considered to potentially meet the inclusion criteria were copied to a separate Endnote folder entitled “First screening.” The second stage of this process included a more thorough reading of the abstracts of selected studies to identify whether they were likely to provide quantitative information, and whether they assessed an outcome that is in line with the topic of interest. Studies selected at this stage were copied to a separate Endnote folder entitled “Second screening.” We then identified and downloaded the full text PDFs for each of the items in this folder and attached them to their respective references within Endnote. Subsequently, the full-text PDFs were accessed and assessed for inclusion based on readings of the methods sections, with particular attention given to the sections describing the measurement of the outcomes and indicators, as well as the nature of the sample and methods used. Only studies meeting all inclusion criteria were copied to a sub-folder entitled “Final inclusion.” Screening decisions were made by two reviewers. Final inclusion decisions were compared, and any discrepancies were reconciled in a joint meeting between the reviewers.

Data extraction and management

In addition to effect sizes, we also coded several study-level characteristics that are known to potentially impact heterogeneity and thereby affect the results or the way in which they should be interpreted. Where possible, we examined the effects of study level characteristics using meta-regression and moderator analysis. Since there is no consensus as to the minimum number of studies needed for conducting such analysis, we adopted a minimalist's approach in which the minimum number of studies needed for examining a continuous variable is 6 (Fu et al., 2011) and 2 studies per category for categorical variables (Marino et al., 2018). Rather than using meta-regression to justify the use of moderator analysis as originally determined in the protocol, we carried out moderator analysis for all factors for which a sufficient number of effect sizes existed. Table 2 includes the different characteristics that were coded.

All coding was performed by two coders, one of whom is the first author. Coding was carried out using Excel. At the end of the coding stage, coder inter-reliability was assessed. In all instances in which the coding of effect sizes differed between the coders, the coders re-coded the data and conducted the analysis jointly. Similarly, the coding of study-level characteristics was coded by the same two coders. Identified differences were rectified through a joint re-coding.

Assessment of risk of bias in included studies

While the review sets few limitations on inclusion based on study quality, several measures of study quality will be coded as study-level characteristics to assess risk of bias. Where possible, we will attempt to analyze how these factors may impact the results using meta-regression. Where this is not possible, the risk of bias elements of studies will be raised in the discussion of the results. For observational studies, we will carry out the assessment of risk based on the most relevant items derived from the ROBINS-E (Risk-of-bias in non-randomized studies of exposure) assessment tool. The selected risk of bias items that will be assessed are listed below in Table 3.

In the ROBINS-E guidance, overall ratings for risk of bias are determined by the highest risk of bias rating in each individual domain. This means that all domains essentially contribute equally to the overall judgment of a study. It has been noted that one of the limitations of this approach is that studies with high risk of bias in only one domain may be assessed to be of equally low quality as studies with high risk of bias in multiple domains. As such, we report the full risk of bias assessment so that users can easily see the different sources of bias that were identified in the included studies (Bero et al., 2018).

For randomized experimental studies, we used an adapted version of Cochrane's Risk of Bias tool for randomized trials (RoB2). As with the observational studies, risk of bias relates to both the study and outcome levels, but there are two additional risk domains that differentiate experimental from non-experimental studies, namely: Domain 1 (risk of bias arising from the randomization or other selection process) and Domain 2 (risk of bias due to deviations from the intended interventions). The other domains of the RoB2 tool are analogous to the domains for the observational studies, namely: Domain 3: Risk of bias due to missing outcome data; Domain 4: Risk of bias in measurement of the outcome, and Domain 5 (risk of bias in the selection of the reported result). Given the nature of the literature, certain items in the tool were not relevant and as such, a more compact version of the tool was adopted (e.g. Lum et al., 2020). The following items from RoB2 were included (Table 4).

Similar to the observational studies, experimental studies were assessed as being of low, high, or unclear risk of bias overall according to the highest assessment from any single domain. That is, if a study was assessed to have high risk of bias in one domain, we classify the entire study as being at high risk of bias.

Measures of effects

There is a lack of consensus as to whether meta-analysis based on observational studies should give preference to effect sizes derived from bivariate correlations, or correlations derived from the results of multivariate regression models that have controlled for confounders. While standardized coefficients derived from multivariate models may produce estimates closer to the “true” value, there is significant between-study variations in model specifications, and studies are often not comparable (Hanushek & Jackson, 1977; Hunter & Schmidt, 2004). For this reason, many researchers give preference to bivariate correlations, which offer a consistent and uncontaminated measure across studies (Pratt et al., 2014). The decision whether to use bivariate, multivariate, or a combination of these effect sizes also depends on a study's objectives (Aloe et al., 2016). One of our objectives was to identify the relative magnitude of the effects for different media-related factors, and bivariate correlations provide more stable estimates for developing a rank-order of estimates among multiple factors (Hedges & Olkin, 1985). As such, we follow the approach that gives preference to bivariate correlations but allows for the inclusion of effect sizes standardized from regression models as supplementary effect sizes when that is all that is available. We will use moderator analyses to assess and report the effects of this approach (Aloe et al., 2016). We believe that this approach is the one most suited to the current review. It is also in line with related works that have been conducted (e.g., Najaka et al., 2001; Wong et al., 2010), and serves to enable the inclusion of the greatest possible number of effect sizes (Borenstein et al., 2009).

All effect sizes were standardized as Fisher's Z transformed variables (Borenstein et al., 2009; Rosenthal, 1984), which was also used for reporting of results. Effect sizes were derived from bivariate correlations, primarily from correlation matrices, or calculated from descriptive data such as means and standard deviations, t tests, χ ², analysis of variances (ANOVAs), and other classical hypothesis tests. The calculation of bivariate effect sizes from descriptive data was carried out using the formulas and conventions of Lipsey and Wilson (2001), and the “Practical Meta-Analysis Effect Size Calculator” available through the Campbell Collaboration website.

For effect sizes derived from bivariate sources, standard errors were calculated based on the variance of the z transformed variable (1/n − 3), in which the standard error is calculated as being the square root of the variance.

Where a study reported only the results from linear regression models where the independent variable (IV) and dependent variable (DV) are both continuous we calculated r as

r = S D x B S D y . $r=\frac{S{D}_{x}B}{S{D}_y}.$

In situations in which standard deviations (SD) were not reported, or when the IV was dichotomous and the DV continuous, or where the IV was ordinal or continuous and the DV was dichotomous, r was calculated based on the t ratio (B/SE):

r = t / t 2 + n − 2 . $r=t/\sqrt{t^2\,+\,}n-2.$

In instances in which both IV and DV were dichotomous, and only B was reported we first calculated Cohen's d and then converted this to r as follows:

d = B 3 π , $d=B\left(\frac{\sqrt{3}}{\pi }\right),$

r = d 4 + d 2 . $r=\frac{d}{\sqrt{4+d^2}}.$

For these conversions, standard errors were calculated, whenever possible, based on a rescaling of the model-based standard error, which is calculated as

s e r = r × S E B B . ${se}_r=\frac{r\times S{E}_B}{B}.$

When the SE was not reported, it was calculated from reported confidence intervals (CIs). In the case of 95% CIs, the SE was calculated as

S E = CI upper − CI lower 1.96 . $SE=\frac{{{CI}}_{\mathrm{upper}}-{{CI}}_{\mathrm{lower}}}{1.96}.$

For experimental studies, while it is common to calculate the standard mean difference, Cohen's d, there is a tendency for overestimation of effects in small samples (Hedges, 1981). As such, we determined to use Hedge's g, which corrects for this bias. To calculate Hedge's g, we would first calculate the correction factor, j as

J = 1 − 3 4 df − 1 . $J=1-\frac{3}{4{df}-1}.$

With Cohen's d already having been calculated based on the means and standard deviations of the groups or other summary statistics, we calculate g = J × d. Subsequently we calculate the standard error as the square root of V _g , with V_g calculated as

V g = J 2 × V d . $V_g=J^2\times V_d.$

Dealing with missing data

When a study was missing information pertaining to study level characteristics or effect sizes, the following actions were taken in the order in which they appear: 1.

Search for supplementary materials.

It was originally determined that in a case in which a study reported on the effect of a particular factor as not having been statistically significant and no additional information was given or had been acquired through the above steps, an effect size of zero would be entered into the meta-analysis (Durlak & Lipsey, 1991). Additionally, it was determined that if a study presented quantitative findings that were unable to be synthesized in the meta-analysis, because the statistic presented does not enable conversion or standardization, or there is not enough information available, the above steps would be followed.

Assessment of heterogeneity

Heterogeneity was assessed using Cochran's Q and its associated p value, as well as the I ² statistics. I ² scores of >75 indicate high, >50 moderate, >25 medium heterogeneity, and >0 low heterogeneity. When I ² = 0 it indicates an absence of heterogeneity.

Assessment of reporting biases

Data synthesis

Meta-analysis was conducted for each factor for which at least two unique effect sizes were found. All analyses were carried out using Biostat's Comprehensive Meta-Analysis (CMA) software (Borenstein et al., 2009). Random effects models were used for all analyses to account for heterogeneity. CMA V3 uses the Method of Moments approach of DerSimonian and Laird (1986) as the random effects estimator for τ ². Results were arranged in rank order according to the size of the pooled estimates for the different factors analyzed.

Sub-group analysis and investigation of heterogeneity

To further investigate and understand sources of heterogeneity, and to identify ways in which estimates for different factors may fluctuate between different contexts and conditions, a number of additional analyses were planned.

As described above, meta-regression was used to assess the effects of continuous variables, and moderator analysis was used to assess the effects of categorical variables. The following variables were examined:

Region from which the sample was derived (EU, US, and Other).

We adopted a minimalist approach in which a minimum of two studies from each category is needed to perform meta-regressions for categorical variables, and a minimum of six studies is needed for continuous variables. For categorical variables, wherever the meta-regression provides a statistically significant effect (p < 0.10), moderator analyses will be used to assess the degree of between category heterogeneity and differences in the estimates.

As our approach was to identify and utilize bivariate correlations as our preferred effect size, and only utilize effect sizes derived from multivariate models when bivariate calculations were not possible, we did not assess differences in the effect sizes from different studies based on the other (both included and excluded) covariates in a study. However, for any analysis for any factor that included at least two effect sizes derived from different sources (Bivariate or standardized partial effect sizes), we assessed the impact of combining the effect sizes through moderator analysis. The moderator analysis was used to identify the degree of between category heterogeneity and differences in the estimates across the categories. As per the above, we did not carry out meta-regressions as a justification for this analysis as was originally considered in the protocol, as it was determined that this was an unnecessary step.

Sensitivity analysis

Sensitivity analysis was carried out for each analysis that included 3 or more studies by using the “leave-one-out” method. In this method the meta-analysis is iteratively repeated k times (k = the number of studies), with a different study excluded at each iteration and the estimates and heterogeneity statistics re-calculated. The results provide for the ability to examine whether a specific study is responsible for greatly influencing the results (Viechtbauer & Cheung, 2010). We report on those analyses in which the removal of a single study was found to reduce heterogeneity by at least one level, with the levels being low (I ² < 25), moderate (I ² < 50), high (I ² < 75), and very high (I ² > 75).

Results of the search

Our initial systematic searches were carried out in August 2020 and resulted in the retrieval of 36,732 unique items which were subsequently transferred to an EndNote library. Once imported to the library and fully updated, an initial screen was carried out based on their titles and abstracts. The initial screening sought to remove all studies that were clearly not related to the general topic of radicalization (or associated terms) in any way (such as studies from medicine examining “free radicals,” environmental studies examining “extreme” weather, and studies examining more closely related topics such as fear of terrorism, or the impact of terrorism on different markets). In addition, all duplicates were removed at this stage. Following this process, we were left with 569 items, which together with 4 items we received from experts, left 573 items to be reviewed. The second screening stage included a more careful reading of the abstracts to identify whether studies dealt with a topic related to the current review. In some cases, where the abstracts were not clear as to what was being examined, a scanning of the methods sections of the papers was carried out. This process led to the removal of a further 469 items, as most of these studies were found to be examining group, platform, or content level dynamics, or using proxies for radicalization that fell outside of the inclusion criteria, such as voting for populist right-wing parties in democratic elections. For the remaining set of 104 studies, we downloaded the full-text PDF for each item and attached them to their respective items in the EndNote library. From here, we reviewed the methodology sections of each item and identified whether the dependent variable(s) was in line with the inclusion criteria, and whether any of the independent variables measured a media-related factor. The final inclusion was 41 studies. In addition to these studies, we included 1 study that was sent to us by an expert (others that were received did not meet the inclusion criteria), 4 studies that we were familiar with, and which did not appear in our searches, and 1 unpublished study by the review's authors, for a total of 47 included studies.

In March 2022, following consultations with the editorial team of Campbell Collaboration's Crime and Justice Group, we re-ran the searches using a slightly modified search syntax, limiting search results to those published before August 2020 when the initial searches were carried out. These updated searches resulted in the retrieval of an additional 11,191 unique items. Following the same process as the initial screening, 87 potentially relevant items were identified in the first screening. However, upon a more careful inspection of the titles and abstracts, 20 of these items were removed. Among the remaining 67 studies, we excluded N = 43 based on their outcome not meeting the inclusion criteria, N = 4 for not having a relevant indicator variable, N = 6 for their unit of analysis (not individual level), and N = 5 for being qualitative, and N = 1 for having an invalid control group.

While N = 6 studies met the inclusion criteria, N = 1 of them was the dissertation that preceded an already included study and did not include any new information (Luchsinger, 2018), and N = 2 others were published by the same author of an already included study (Pauwels & Schils, 2016) and did not provide any new information to be synthesized.

In combining the results from the two sets of searches, the review had a final inclusion of N = 53 publications containing N = 76 samples. The PRISMA flowchart below depicts the different stages of the searches and screening process (Figure 2).

OPEN IN VIEWER

Figure 2

PRISMA flowchart.

Included studies

Most studies were published from 2016 to 2020, with only a maximum of two studies appearing in years before this, going back to two studies published in 2002. The year with the largest number of studies was 2019, followed by 2020. Most of the studies were published in journals; however, the included items also included published reports, one book chapter, three theses, and one unpublished paper. Among the 76 samples included in 53 items, most were derived from European countries (32.9%), followed by North America (26.3%), Middle Eastern and African countries (25%), and East-Asian countries (7.9%). There was an additional study which had a mixed sample of Europe and North America, as well as for North America and Asia. One study focused on a South American sample, and another on Australia. Table 5 provides a description of the included studies (more details are available in the Supporting Information Appendix).

From experimental studies, we extracted 12 effect sizes that pertained to manipulated exposure to mediated content theorized to increase radicalization. We also extracted an additional 4 effect sizes pertaining to relevant interactions between treatment exposure and trait aggression. From observational studies, we identified 23 media-related factors for which at least two effect sizes were identified, with a total of 110 effect sizes extracted for the primary analyses, and an additional 10 effect sizes extracted pertaining to sub-groups. Table 6 provides a description of these factors, which are listed in alphabetical order.

While the descriptions for many of the factors are quite straightforward, some are deserving of further elaboration, particularly what we refer to as “radical content.” We consider “radical content” to be mediated content that is produced by individuals or groups with the goal of, or potential of increasing support or justification of terrorism (the use of violence in the name of a cause or ideology), or terrorist groups. This content usually makes use of violent images or messages which seek to generate sympathy or justification for the use of violence in defense or furtherance of an ideology or cause. For example, Frissen, (2019, 2020) examines passive exposure to, and the active seeking out of Jihadist information, which is measured by different types of materials, including: Dabiq (Islamic State's magazine), Inspire (Al-Qaeda's magazine), jihadist videos, beheading videos, Salafi-jihadist fora, and Salafi-jihadist Facebook groups. Similarly, Manzoni et al's (2019) study assessed whether participants had visited websites, or viewed videos produced by well-known radical groups, such as the Neo-Nazi Blood & Honor, as well as Al-Qaeda and ISIS. Wolfowicz's (2021b) study took a slightly different approach and examined whether participants had been exposed to violent content that they felt promoted violence against civilians and security forces. In the review, we distinguish between passive exposure to and the active seeking out of this type of radical content.

Excluded studies

Outcomes

For the experimental studies, one study used a validated measure of radicalization (Shortland et al., 2020), whereas the other studies used a variety of common, yet unvalidated measures assessing justification of a variety of radical behaviors.

For observational studies, the most common measures of cognitive radicalization were single item measures of justification of suicide bombings/terrorism. While this is a widely used and accepted measure of cognitive radicalization (see Schmid, 2017 for a discussion on this), it is not a validated measure. A few other studies used adapted versions of validated instruments to measure analogous outcomes, however these adapted instruments have either not been validated or otherwise lack evidence that they are valid measures for the outcomes of interest (e.g., Eyal et al., 2006; Gvirsman et al., 2016; Kremerman et al., 2012).

Only three groups of studies used fully validated measures, including the Sympathies for Radicalization scales (SyFor; Bhui et al., 2016; Frissen, 2019, 2020) the Activism-Radicalism-Intentions Scales (ARIS; Ellis et al., 2016, 2019; Frissen, 2019, 2020), and studies comparing violent with non-violent radicals (Clemmow et al., 2020; Sahani, 2018).

Independent variables

Among the experimental studies, treatment included manipulated exposure to a piece of mediated content that was theorized to increase the risk of radicalization compared to a neutral piece of content, in line with standard practices in media-effects research (Ferguson & Savage, 2012). For two studies (Azeem et al., 2019; Kalmoe, 2014), treatment included exposure to manipulated news items. Another study included comparisons of exposure to highlight reels from different action films against a control group, as well exposure to video concerning a political protest against a control group (Jones & Paris, 2018). Only one study included a treatment of exposure to explicitly radical content in the form of an ISIS propaganda video (Shortland et al., 2020).

None of the included studies relied upon a validated measure of media-usage/exposure. Rather, they drew on self-reported measures of media usage and engagement with certain types of activities or content. Self-reports such as these have the potential for both over and underestimation of actual usage. Nevertheless, they are the most common way in which the effects of different types of media exposure are examined in observational studies examining related outcomes (even if only distantly) such as aggression and general delinquency (Ferguson & Savage, 2012). When self-reports are used to measure exposure to specific types of content, there is an added source of potential bias, as specific interest in the topic of the content (e.g., politics) may influence memory recall of exposure to it (Donohew et al., 1998; Slater, 2004). This added source of potential bias is relevant to those risk factors included in the review which measure exposure to specific types of content (e.g., political, radical, or violent).

Report type

The studies were primarily based on self-reported measures of both the dependent and independent variables. One of the studies measuring behavioral radicalization was based on a database derived from open sources that included a group of violent radicals and a control group of non-violent radicals (Sahani, 2018). Another study measuring behavioral radicalization was also based on an open-source database of convicted terrorists and compared it with a general population survey relying on self-reported measures (Clemmow et al., 2020).

While the broader radicalization literature includes some examples of clinical assessments of radicalization, these are usually made as part of risk assessments for individuals that have been referred to various authorities (e.g., Pfundmair et al., 2022). Self-reports of cognitive radicalization remain the dominant report type in the literature and are even common for measuring sub-terroristic forms of radical behaviors (see Wolfowicz, Weisburd, et al., 2021a). The studies included in this review are therefore representative of the types of studies found in the broader literature on radicalization.

Sampling strategy

Of the included studies, the most common strategy was to employ random sampling (N = 17), followed by representative sampling (N = 9) and stratified sampling (N = 7). Purposive sampling was used in N = 4 studies, Convenience sampling in N = 5 studies, snowball sampling in N = 2 studies, probability sampling in N = 2 studies, non-probability in N = 1 study, and mixed sampling strategy in another N = 1 study.

Temporal ordering

In experimental studies, it was not evident that Time 1 measures of the outcome were controlled for, nor did any of the studies report on the collection of such measures.

Regarding observational studies, the review included only two longitudinal studies. These studies did not report on the same set of factors and as such it was not possible to conduct a separate meta-analysis for longitudinal studies (Gvirsman et al., 2016; Zhu et al., 2020).

For two studies that examined radical behaviors (Clemmow et al., 2020; Sahani, 2018), exposure to the risk factors inherently preceded the outcome of radical offending behaviors.

It is not possible to determine that exposure to the independent variable preceded the onset of the dependent variable for the remainder of the studies (N = 45) as they are cross-sectional. Additionally, in the two experimental studies, pre-test measures of the outcome were not taken.

The issue of temporal ordering precludes the ability to draw causal inferences from the results, and all interpretations should be weighed against this limitation.

Statistical controls

As noted earlier, we use a risk-factor approach in which the main outcome measures are based on uncontrolled correlations. Statistical controls in the multiple regression models were appropriate, most commonly accounting for basic socio-demographic variables such as age and gender. Most studies provided correlation matrices that included other variables, however, the combination of these variables differed considerably across studies. As such, it was not feasible to conduct a separate analysis on partial effect sizes controlling for similar sets of confounding variables.

Missing data

Few of the studies noted any issues with missing data. However, this is not necessarily an indication that missing data did not exist as it is very common for publications to fail to note this issue. A few studies that noted the presence of missing data noted that it was handled either through exclusion from the analysis or using estimation techniques.

Non-significant results

There was no clear indication of non-significant result reporting bias in all but one of the studies (Calenda & Mosca, 2007), which stated that certain relationships existed but failed to report on them.

Excluded variables

There were N = 3 studies that explicitly mentioned variables that were apparently available in the data but then absent in reporting of the results. There were also several studies that relied on secondary data which are known to contain other potentially relevant variables that were not included (N = 6). There were N = 5 studies that had the ability to report on different types of media-related factors separately but only reported on them as an aggregate measure. For those studies for which such bias was not evident, we are still unable to fully discount the possibility that variables were omitted as studies were not pre-registered.

Excluded outcomes

Two studies (Frissen, 2019, 2020) relied on a combined measure of cognitive radicalization, made up of both the SyFor and Activism-Radicalism-Intentions-Scale (ARIS). While other radicalization studies not included in this review have taken a similar approach (e.g., Rousseau et al., 2019), these scales may represent distinct constructs (Wolfowicz et al., 2020) and it thus would have been informative to have had the ability to assess effect sizes as they pertain to each of the scales separately.

In one study (Calenda & Mosca, 2007) correlations between different versions of the dependent variable and independent variables are reported on. In some cases, this study reported on individual items that made up the dependent variable, whereas in other cases they report on correlations with the index dependent variable.

In another study (Mulyana, 2020), insufficient information is given about the dependent variable. However, as it is described, it appears that an additional measure of radical intentions is available but not reported on.

In Pauwels and Schils (2016) study (see also Pauwels & Hardyns, 2018; Pauwels et al., 2020), only one version of the dependent variable is reported. However, in data supplied by the authors multiple versions of the dependent variable were available.

There was no evidence of pre-published or registered protocols for any of the studies. As such, even where there was no evidence of excluded outcome bias, we are still unable to discount the possibility that they employed more than one outcome measure but only reported on one. We are also unable to discount the possibility that additional models were used in which the dependent variable was measured differently (e.g., dichotomized).

Additional items for experimental studies

Observational studies

A total of 23 factors were identified for the outcome of cognitive radicalization, consisting of 3 factors with negative associations and 20 factors with positive associations with the outcome. Only two factors were identified for the outcome of behavioral radicalization (see Table 6 for a description of these factors). Following the classification of very small, small, moderate, and large estimates, most of the factors had very small-small estimates (Hopkins, 2002).

Among the factors with negative associations were Internet Access, Newspaper reading, and Radio listening. In addition to the effect sizes being exceptionally small, they were also not statistically significant. Among factors with positive associations, 12 had very small estimates ranging from r = 0.01 to 0.08. These included, in rank order: Television consumption, Facebook usage, using the Internet to consume political news, engaging in content piracy, time spent online, trust in media, Twitter usage, posting of political content online, general media consumption, online network participation, technical skills, and being a victim of cyberbullying. The next tier included factors with small to moderate estimates (r = 0.10 to 0.26). Following the rank order, similar estimates were found for factors pertaining to general components of Internet and media usage. This included: Self-censorship online, hacking, usage of online discussion forums, attachment to online networks, active seeking of radical content online, passive consumption of radical content online, and consumption of ISIS news. An estimate that can be categorized as large was found for a single factor, namely willingness to engage in ideologically motivated cyber-attacks (r = 0.57).

The results show that certain attitudinal and experiential factors, as they pertain to the Internet, have effect sizes that are not substantively different from general Internet usage. They also show that more specific forms of online activity, as well as some attitudinal factors, in particular a willingness to engage in ideologically motivated cyber-attacks, demonstrate more robust relationships with radicalization than general forms of consumption.

As opposed to cognitive radicalization, for behavioral radicalization the included studies only provided effect sizes pertaining to two factors for which meta-analysis was possible, namely passive and active forms of consumption and engagement with radical content. The pooled estimate for active engagement with radical Internet-based content (r = 0.28 [95% CI = 0.21, 0.36], k = 5, I ² = 78.49) was slightly larger than the estimate for passive exposure to such content (r = 0.23 [95% CI = 0.12, 0.33], k = 4, I ² = 91.10; see Table 8).

Experimental studies

The experimental studies included in this review were combined in a single meta-analysis to examine the effects of 'one-off exposure to radicalizing content'. However, the nature of the exposure differed between the studies. Kalmoe (2014) used a vignette design, exposing the treatment group to violent or neutral political messages, whereas Azeem et al. (2019) manipulated exposure to either positive or negative news stories concerning political events. Jones and Paris (2018) used exposure to clips from action-drama films, as well as from broadcasts concerning political protests. Shortland et al. (2020) was the only study to expose the treatment group to an explicitly radical video, being an approximately 6-min-long propaganda video produced by the ISIS terrorist group. Two of the studies, which included a total of four samples (Kalmoe, 2014; Shortland et al., 2020), also examined interactions between exposure to the respective experimental conditions and trait aggression (Table 9).

The results of the random effects meta-analysis (k = 12, n = 3196) found a pooled estimate of g = 0.08 [95% CI = −0.03, 0.19], which presented with more than a moderate degree of heterogeneity (Q = 26.643, p = 0.003, I ² = 58.09).

OPEN IN VIEWER

Figure 3

Exposure on cognitive radicalization.

When looking at the interactions with trait aggression, exposure to the experimental conditions had a pooled estimate of g = 0.13 [95% CI = 0.01, 0.25]. Heterogeneity was found to be moderate (Q = 4.346, p = 0.226, I ²  = 30.97). The results show that in the presence of high trait aggression, simple exposure to media theorized to radicalize was associated with a small increase in risk, larger than exposure on its own (Figure 4).

OPEN IN VIEWER

Figure 4

Exposure × Trait aggression interaction.

Heterogeneity

As per Table 3, heterogeneity was found to be very high (I ² > 75) for 13/23 factors derived from observational studies for cognitive radicalization, and both the factors pertaining to behavioral radicalization. Heterogeneity was high for one additional factor, moderate for two factors, and absent for seven factors, of which four factors were made up of only two effect sizes. Heterogeneity was moderate for both analyses of experimental studies. As such, and following the protocol, we sought to investigate a range of potential sources of heterogeneity.

Single effect sizes

The systematic review identified several other media-related factors for which only a single effect size was found, and thus meta-analysis was not possible. However, it is important to consider the nature of these factors as part of the broader scope of the review.

Publication bias

Publication bias analysis was carried out for all factors that included a minimum of three effect sizes. Egger's regression was found to be marginally significant for only one factor, Network attachment, and was not statistically significant for all other factors. The Trim-and-Fill method identified missing effect sizes for nine factors for cognitive radicalization, one of which was Network attachment, and both factors pertaining to behavioral radicalization. In all such cases, between 1 and 2 missing studies were identified, and the maximum difference between the original estimate and the adjusted estimate did not exceed r = 0.09 (network attachment). Thus, most of the factors analyzed were robust against publication bias and the differences emerging from the trim-and-fill analysis do not change the interpretation of the results (see Table 15).

Dependent variables

With respect to outcome measures, most studies did not use validated measures of cognitive radicalization. However, validated measures of cognitive radicalization are relatively recent developments in the literature, and the types of measure used in the included studies are widely considered to be appropriate (Schmid, 2017). Measures of behavioral radicalization were measured dichotomously and primarily based on self-reports. While self-reports provide an interesting perspective on radical behaviors, there is considerable distance between these behaviors and "terrorism” (McCauley & Moskalenko, 2017).

Independent variables

The nature of the independent variables examined in this review are like those that are examined in the broader study of media-effects. That is, the independent variables capture self-reports pertaining to media usage and consumption. While these types of measures are quite standard in media-effects research, they are sensitive to under and over estimations of exposure time (Prior, 2009a, 2009b, 2012). Additionally, self-reports are sensitive to memory recall bias, especially when the reports pertain to topical content toward which respondents possess differential levels of interest (Donohew et al., 1998; Slater, 2004).

Sampling

The included studies employed appropriate sampling procedures for observational research. However, given a lack of nationally representative surveys, we were unable to assess differences in effect sizes based on sampling strategies.

Statistical approaches

Studies generally employed appropriate statistical techniques for identifying the strength of the relationship between measures. In this regard, another important consideration is whether the factors have a theoretically plausible relationship with the outcome (Murray et al., 2009). Indeed, most factors analyzed in this study, especially those with the largest estimates (as per the above discussion), are theoretically derived factors which have plausible relationships with radicalization outcomes. However, the studies did not always control for important confounding factors, especially those known to be especially relevant for media-effects such as self-control, trait aggression, and gender. It has previously been suggested that bivariate correlations between violent media consumption and violent outcomes may simply represent gender effects, or the effects of trait aggression, which is also highly correlated to gender (Ferguson, 2015). Additionally, these and other factors may impact differential exposure to media (Savage & Yancey, 2008). Given the nature of the data, we were unable to analyze effects adjusted for such factors and we were unable to account for this potential source of bias. As such, whilst the effects identified in this review certainly follow the theorized direction, it is also entirely possible that more radical individuals are simply more likely to engage in the types of media behaviors analyzed.

Publication bias and sensitivity of the results

Publication bias and sensitivity analysis were limited to factors for which there were more than three effect sizes. For most factors, there was no evidence of significant publication bias or sensitivity to outliers. For those factors for which these analyses indicated changes to the pooled estimates, the differences were exceptionally small and do not change the substantive findings. However, for one factor, network attachment, the results are inconclusive. The leave-one-out analysis indicated that one study significantly contributed to an inflated estimate. In removing this study, the pooled estimate was reduced from r = 0.20 to r = 0.07. On the other hand, the Trim-and-Fill analysis imputed two missing effect sizes to the right of the pooled estimate, increasing the estimate to r = 0.29. These results indicate that until additional effect sizes can be synthesized, the results for this factor should be interpreted cautiously. The only other factor for which there was significant evidence of publication bias was for exposure to radicalizing exposure in experimental studies. Here, the Trim-and-Fill analysis indicated a small reduction in the effect size.

Review process

We acknowledge that there are studies that may provide important evidence concerning the risk and protective factors for radicalization that were not included in this review on account of their outcome measures failing to meet the review's inclusion criteria. For example, some studies are known to assess “willingness to die for a cause/group.” However, as described above, such studies were excluded since a willingness to die does not necessarily indicate a willingness to use violence against others (e.g., Bélanger et al., 2014). Other studies have examined the effects of different forms of media consumption on outcomes such as racism and hate-speech (Soral et al., 2020). Yet others, such as Machackova and Šerek (2017), analyzed how online political activity predicted the acceptance of “non-conventional” activism, which included a vague reference to “run-ins with authorities.” We do not believe that the review's results are biased because of having excluded such studies. Rather it means that our results are based on a more homogenous set of outcomes and should therefore be more robust from a meta-analysis perspective. However, we can note here that the findings from these studies demonstrate a significant degree of overlap with our results.

It is also important to highlight the fact that several of the meta-analyses performed in this review are based on a very small number of effect sizes (as few as two) and some of those with a greater number of effect sizes demonstrate significant heterogeneity. As such, caution must be used in drawing inferences from our analyses and interpreting our results.

Another potential limitation of this review is language. While it is common to assume that the most important studies are at least indexed in databases in English, we cannot discount the possibility that studies in other languages exist that we were unable to identify. Indeed, as our results found a handful of studies examining samples in Asia, we suspect that additional studies from these countries may exist in local languages. Future attempts to synthesize the body of evidence may consider dedicating more significant resources to attempting to identify studies published in other languages.

Whilst we encourage replication, we acknowledge that the authors' familiarity with the literature may have impacted the number of studies that passed through the different screening stages. As such, while a replication will likely reach similar substantive findings, differences in inclusions and exclusions at different stages would be expected.