PROTOCOL: Criminal justice interventions for preventing terrorism and radicalisation: An evidence and gap map

Types of study design

Systematic reviews and randomised controlled trials (RCTs) are considered the gold standard for ascertaining intervention effectiveness. Eligible comparison conditions/groups for inclusion in the EGM will include no treatment, placebo, ‘business as usual,’ waitlist control, or an alternative treatment. While other designs are less causally robust, they may be appropriate to include due to the difficulties associated with conducting RCTs in criminal justice settings, particularly when targeting radicalisation, violent extremism, and/or terrorism. The strong quasi-experimental designs eligible for this EGM include:

Matched control group designs with or without preintervention baseline measures (propensity or statistically matched)

In line with previous reviews (Mazerolle et al., 2020; Mazerolle et al., 2021) which recognise the significant shift in counterterrorism policies post 9/11 (Gaibulloev & Sandler, 2019), only studies that are published from or report on impact evaluations conducted between January 2002 and December 2021 will be included in the EGM. We will not include qualitative research as we are exclusively interested in mapping high quality impact evaluations. Should our searches identify any ongoing studies that meet our selection criteria but for which findings have not yet been published, we will include these as ongoing studies. These will be assessed for possible inclusion when the review is updated by the authors.

Types of intervention/problem

Our EGM will capture criminal justice interventions that aim to prevent or respond to radicalisation, violent extremism, and/or terrorism.

A criminal justice intervention is defined as a strategy, technique, approach, activity, campaign, training, directive, funding, or organisational change that involves at least one criminal justice agency. Criminal justice agencies include police, courts and corrections (both custodial and community based). Interventions that involve partnerships between at least one criminal justice agency and at least one external agency/organisation will also be included. (Eggins et al., 2021; Mazerolle et al., 2018). Examples of criminal justice interventions may include:

Criminal justice system initiation, development, or leadership of the intervention.

Radicalisation is defined as the process of an individual or group adopting extreme political, social, or religious ideals (Hardy, 2018; Jensen et al., 2018). Analogous concepts are disengagement and/or deradicalisation, which are often encompassed within conceptualisations of violent extremism. Disengagement refers to reducing or ceasing physical involvement in violent or radical activities, while deradicalisation is defined as the psychological shift in attitudes or beliefs (Windisch et al., 2016).

Violent extremism is defined as ‘advocating, engaging in, preparing, or otherwise supporting ideologically motivated or justified violence to further social, economic, and political objectives’ (US Agency for International Development, 2016).

Terrorism is defined as ‘the unlawful use of force or violence against persons or property to intimidate or coerce a government, the civilian population or any segment thereof, in furtherance of political or social objectives’ (Shanahan, 2016, p. 108).

Eligible interventions may include but are not limited to:

Police training programs to improve officer recognition and responses to radicalisation and terrorism (Davis et al., 2021).

Eligible comparison conditions/groups for inclusion in the EGM will include no treatment, placebo, ‘business as usual,’ waitlist control, or an alternative treatment. If a study reports on multiple interventions whereby only a subset meets the eligibility criteria, only the eligible interventions will be included in the map.

Types of population

The following populations will be included in the EGM:

Criminal justice practitioners

Types of outcome measures

No restrictions will be placed on the types of outcomes used to evaluate criminal justice responses for preventing radicalisation, violent extremism and/or terrorism. Depending on the studies identified, we tentatively plan to group outcomes under three domains: 1.

Extremism/radicalisation indicators such as a.

Violent Extremist Risk Assessment-2 (VERA-2): a risk assessment tool used to assess an individual's risk of violent extremism (Pressman & Flockton, 2013)

Unintended or adverse outcomes will be included.

Global policing database

The search for this review will be led by the Global Policing Database (GPD) research team at the University of Queensland (Lorraine Mazerolle and Lorelei Hine), Queensland University of Technology (Angela Higginson) and Griffith University (Elizabeth Eggins). The University of Queensland is home to the GPD (see http://www.gpd.uq.edu.au), which will serve as a key search location for this review. The GPD is a web-based and searchable database designed to capture all published and unpublished experimental and quasi-experimental evaluations of policing interventions conducted since 1950. There are no restrictions on the type of policing technique, type of outcome measure or language of the research. The GPD is compiled using systematic search and screening techniques, which are reported in Higginson et al., (2015) and summarised in Supporting Information: Appendices 1 and 2. Broadly, the GPD search protocol includes an extensive range of search locations to ensure that both published and unpublished research is captured across criminology and allied disciplines.

To capture studies for this review, we will use terms related to terrorism and radicalisation to search the GPD corpus of full-text documents that have been screened as reporting on a quantitative impact evaluation of a policing intervention. Specifically, we will use the following terms to search the title and abstract fields of the corpus of documents published between 1 January 2002 and 31 December 2021:

extremis* OR ‘far#left*’ OR ‘far#right*’ OR ‘foreign#fight*’ OR ‘freedom#fight*’ OR guerrilla OR ‘homeland#security’ OR ‘ideological violence*’ OR ‘ideologically#motivat*’ OR indoctrinat* OR ‘left#wing*’ OR ‘lone#wol*’ OR militant* OR ‘national#security’ OR ‘political violence*’ OR ‘politically#motivat*’ OR radicali* OR rebel* OR ‘religious violence*’ OR ‘religiously#motivat*’ OR ‘right#wing*’ OR ‘single#issue*’ OR supremacis* OR terror* OR vigilante* OR vigilantism OR deradicali* OR ‘de-radicali*’ OR ‘counter-terror*’ OR counterterror* OR ‘counter-extremis*’ OR counterextremis* OR separatis* OR militia* OR jihad*

Other database searches

To capture interventions in the courts, corrections and multi-agency spheres not encapsulated within the GPD, we will conduct searches of key academic sources in criminology and criminal justice (see Table 1). We will search these databases on three sets of terms across title, abstract, keyword, and/or subject/indexing fields. These aim to capture literature on (1) terrorism within the (2) criminal justice system that uses a term indicative of a (3) evaluation of an intervention. Terms within each set will be combined with Boolean OR operators, and the three sets will then be combined with Boolean AND operators (see Supporting Information: Appendix 3 for an example search syntax). This search approach has been informed by Eggins et al. (2021) systematic review of criminal justice system responses to child sexual abuse material offending. We will search the following across the title, abstract, keyword, and/or subject/indexing fields (adapting the search to the database if fields are unavailable): (1)

extremis* OR ‘far#left*’ OR ‘far#right*’ OR ‘foreign#fight*’ OR ‘freedom#fight*’ OR guerrilla OR ‘homeland#security’ OR ‘ideological violence*’ OR ‘ideologically#motivat*’ OR indoctrinat* OR ‘left#wing*’ OR ‘lone#wol*’ OR militant* OR ‘national#security’ OR ‘political violence*’ OR ‘politically#motivat*’ OR radicali* OR rebel* OR ‘religious violence*’ OR ‘religiously#motivat*’ OR ‘right#wing*’ OR ‘single#issue*’ OR supremacis* OR terror* OR vigilante* OR vigilantism OR deradicali* OR ‘de-radicali*’ OR ‘counter-terror*’ OR counterterror* OR ‘counter-extremis*’ OR counterextremis* OR separatis* OR militia* OR jihad*

AND

Searching other resources

In addition to the database searches, we will use the following search strategies:

Searching government and non-government websites related to terrorism and radicalisation research (see Table 1)

Report structure

The EGM will comprise the following sections: (1) executive summary; (2) background; (3) objectives (4) methods (5) results and (6) discussion. The executive summary will provide an overview of the EGM findings, detailing the number of studies captured in the review across the criminal justice system (police, courts, corrections and multi-agency). The background section will detail the problem of radicalisation, violent extremism and terrorism, outlining why and how the criminal justice system provides a prime opportunity for intervention to reduce and prevent its occurrence.

The methods will provide information on the systematic search, including key search terms and search locations. The methods will also describe the screening and data extraction processes and software programs utilised. The results will provide an overview of the number of eligible studies captured by the systematic search, detailed in a PRISMA figure. Our results will also include our interactive EGM, providing a graphical representation of the available evidence. The discussion will reflect on both the paucity and density of research across different intervention and outcome domains. We will recommend areas for future research and synthesis, and implications for policy and practice.

Planned Tables and Figures:

Figure: PRISMA

Filters for presentation

The results of our online interactive EGM will be presented as a matrix of rows (comprising two layers: (1) intervention agency; (2) intervention type) and columns (outcomes). We will additionally employ the following filters:

Study type (toggle filter): research synthesis, research study

In maps that include systematic reviews, we will visually differentiate between systematic reviews that have included studies and those that do not by the use of colour coding (White et al., 2020).

Dependency

The unit of analysis is the primary study, not the publication or document that reports on the study. We are conscious of the risk of using documents as the unit of analysis, as a visual tool such as an EGM can appear to indicate a preponderance of evidence on an intervention where there are multiple publications drawn from the same research study.

The two most common ways in which dependency can manifest in an EGM are as many: one dependency or as one: many dependency. Firstly, there may be more than one document reporting on the same study (many: one dependency). In this situation, all documents that report on the one study will be linked, and the study is what will appear on the map. Secondly, one document may report on more than one study (one: many dependency). If a document reports on more than one study, we will link that document to both ‘parent’ studies.

Systematic reviews provide a unique challenge in terms of the one: many dependency. A systematic review may contain multiple studies, that may report on multiple interventions and outcome combinations. The challenge lies in deciding how a systematic review should be represented in an EGM. If it were to appear on the same map as its component studies, it would visually suggest more evidence than actually exists. But if it were only to be harvested for its component studies, then the EGM could not show where there has been rigorous synthesis of studies in the area, or attempts at rigorous synthesis, in the case of ‘empty reviews’. We propose a multi-armed approach to dealing with systematic reviews in the EGM: 1.

We will harvest the primary research studies that are included in any systematic review, and link those documents back to the parent study as with any document that has a one:many dependency.

A further form of dependency is where one study compares two eligible treatments head-to-head. In these cases, the study itself may be eligible to appear in two different columns of the map. In addition to linking any related documents, we will flag studies of this design so that it is clear where there is the potential for double-counting. Similarly, if a systematic review is eligible to appear in more than one cell on the map, we will flag the review to ensure that the reader is aware of the potential for double-counting.

Screening and study selection

Our screening approach will differ slightly for the different sources, but all documents will be assessed on the same final eligibility criteria. Documents from the GPD have already been screened on their full-text as reporting on a quantitative impact evaluation of a policing intervention. Therefore, we will assess the full-text of these documents for their eligibility to the review. Documents from other sources have not been verified as quantitative impact evaluations, so we plan to screen these on their title and abstract to remove documents not related to criminal justice responses to terrorism/radicalisation, and then screen eligible full-text documents on the same criteria as for the GPD data.

For all stages described below, we will develop standardised screening companions and inter-rater reliability tests to ensure consistency in decision-making across the research team. For title and abstract screening, each team member will screen the same set of 30 records before commencing screening. Similarly, at the full-text screening stages, before commencing screening, team members will screen the same set of 30 documents. All team member decisions will be compiled and compared to determine any divergence in decisions. Feedback on screening decisions will be provided with further training when necessary. We will double screen 5% of each screener's excluded documents to identify any potential false-negative decisions. In instances where a screener's decisions are considered unreliable, their exclusion screenings will be reassigned. Where disagreements about screening decisions occur, these will be mediated by a third team member.

All documents will be screened in DistillerSR—a web based, reference management software that incorporates artificial intelligence (AI) to expedite the review process (Evidence Partners, 2022). DistillerSR has an inbuilt AI function which cross-checks exclusion decisions. This helps to ensure consistency across reviewers and reduce human error. DistillerSR also draws on machine learning to reorder references based on their likelihood of inclusion, learning from the decisions made by human users (Evidence Partners, 2022). Using this function, we will screen documents up until the point that DistillerSR indicates 100% of all potentially eligible records are included. At this point, we will screen a random sample of 30 additional records. We will repeat this process until no eligible records are identified. We will use the AI function for Title and Abstract screening only.

Following Campbell Collaboration guidance (White et al., 2020), we will screen systematic reviews on their PICOS rather than on their included studies. In this way, we will retain ‘empty reviews', and allow the EGM user to see where synthesis has been attempted but was unsuccessful.

Data extraction and management

Once deemed eligible for the review, documents will be coded in EPPI-reviewer. We will use a standardised coding companion (see Supporting Information: Appendix 4) to code documents across the following domains: 1.

Participants (sample characteristics, recruitment, etc.)

All coders will code the same four eligible studies to ensure consistency across team members. These will be compiled and compared to determine any divergence in coding, and feedback will be provided before independent coding. To check for inter-rater reliability, 5% of each person's coding will be double-coded by a second person. Where disagreements arise, a third review author will be consulted. Study authors will be contacted directly if documents are missing any pertinent information.

Tools for assessing risk of bias/study quality of included studies

Each eligible systematic review in the EGM will be assessed for Risk of Bias using the AMSTAR 2 critical appraisal tool (Shea et al., 2017). Based on ratings across 16 items, the tool generates a rating of overall confidence in the systematic review's results: High (no more than one non-critical weakness, but no critical flaws); Moderate (more than one non-critical weakness, but no critical flaws); Low (one critical flaw, with or without non-critical weaknesses); or Critically Low (more than one critical flaw, with or without non-critical weaknesses). Two coders will independently code each systematic review using the AMSTAR 2 tool. Any discrepancies between the two coders will be resolved by discussion.

Due to the anticipated size of the EGM, we do not plan to assess confidence in the included primary studies, but note that our methodological thresholds exclude study designs with the highest inherent risk of bias.

Methods for mapping

We will use EPPI-reviewer to create the EGM (Eppi-Centre, 2019).