‘Led by Intelligence': A Scoping Review on the Experimental Evaluation of Intelligence-Led Policing

Statistical Powerlessness

The challenge of achieving sufficient statistical power to detect significant effects, particularly in terms of crime reduction, was frequently discussed, chiefly in the context of place-based policing studies (Ariel et al., 2016, 2020; Gerell, 2016; Hunt et al., 2014; Lum et al., 2011; Piza et al., 2015; Piza & O’Hara, 2014; Ratcliffe et al., 2020; Rosenfeld, 2014; Santos & Santos, 2016, 2021; Telep et al., 2014; Weisburd et al., 2015).

Several factors were identified through narrative analysis as influencing the level of statistical power. One factor is sample size, with larger sample sizes generally thought to increase statistical power to detect significant differences between treatment and control groups (Ratcliffe et al., 2021; Taylor & Ratcliffe, 2019). However, from an empirical point of view, this premise must be balanced against Weisburd’s paradox (Groff et al., 2015; Weisburd et al., 1993), which highlights the discrepancy between the expectation that increasing the sample size will enhance statistical power, and the reality that it often results in decreased statistical power. In this regard, it is argued that increasing the sample size gives rise to two main phenomena that may reduce the statistical power of a test. First, increasing the sample sizes can result in increased variability between the treatment and control groups, which subsequently elevates the standard deviation within the sample. This increased variability reduces the probability of detecting substantial and statistically significant effects. Second, increasing the sample size may introduce ‘watered down treatments.' It is theorized that experimental designs employing larger sample sizes may be more susceptible to implementation issues, potentially affecting the adherence to the intended administration or dosing of the treatment. In this regard, it has been found that studies with very small sample sizes may exhibit higher statistical power compared to those with very large sample sizes (Weisburd et al., 1993).

In place-based policing experiments, the size and shape of treatment areas were considered important factors influencing the statistical power of a test. A more granular size or shape could potentially increase the power to detect deterrent effects, as smaller areas allow for increased sample sizes and higher statistical power (Phillips et al., 2016). However, smaller geographical units may also affect the crime base rates within those units, which can impact the size and strength of treatment effects (Phillips et al., 2016; Ratcliffe et al., 2021).

In addition, it is argued that the issue of statistical powerlessness is often more prevalent in place-based policing (quasi-)experiments conducted in small- and medium-sized cities, where low crime base rates make it challenging to isolate the effects of interventions, and fluctuations in crime rates can hinder the detection of significant treatment effects (Ariel et al., 2021; Hinkle et al., 2013; Phillips et al., 2016).

To address the problem of statistical powerlessness in quasi-experimental designs, researchers have suggested matching treatment units to a larger number of comparison units, and conducting panel regression analyses (Phillips et al., 2016). In place-based predictive policing randomized controlled trials (RCTs), focusing on larger mission areas rather than micro-scaled intervention areas has been proposed as a more appropriate approach (Ratcliffe et al., 2021; Taylor & Ratcliffe, 2019). However, while widening the scale of treatment areas may make patrol officers’ activities less monotonous and reduce the likelihood of boundary violations, it could also lead to net-widening effects and pose challenges to police legitimacy (Ratcliffe et al., 2021; Taylor & Ratcliffe, 2020).

Spillover Effects and SUTVA

Many of the studies included in this review addressed concerns regarding the stable unit treatment value assumption (SUTVA) (e.g., Ariel et al., 2016, 2019, 2020; Ariel & Partridge, 2016; Collazos et al., 2020). SUTVA posits that the assignment of units to treatment or control groups should not influence the outcomes of other units (Braga et al., 2012, 2018). Avoiding spillover effects, which can obscure the true impact of the intervention and compromise causal inference, is crucial for addressing SUTVA. Two types of spillover effects were commonly addressed in the studies. The first involves the spillover of treatment conditions from treatment groups to control groups, known as major interference (e.g., Ariel et al., 2019; Sobel, 2006). Given the focus on place-based policing, preventing major interference often required buffer zones to be created, to ensure that control areas did not receive any part of the treatment (e.g., Ariel et al., 2020; Collazos et al., 2020). However, it is important to note that major interference can also occur at the individual level due to social contamination processes within offending and co-offending networks, as offenders living or residing in the treatment areas may be socially connected to offenders in the control areas, thus increasing the likelihood of spillover from treatment to control groups (Ariel et al., 2019). The second type of spillover was partial interference, where spillover occurs within the same group from one unit of analysis to another. This mostly required consideration of how the same interventions and conditions could be applied consistently and uniformly within all treated and untreated groups (Ariel et al., 2019; Hunt et al., 2014; Saunders et al., 2016). If not, this could affect the fidelity of the study and bias the treatment effects to some extent (Ariel et al., 2018; Sobel, 2006).

Data and Measurement Errors

Several studies emphasized the importance of evaluating data quality issues, particularly those using big data and algorithmic applications for crime prediction purposes. It is crucial to assess data quality in order to mitigate biases in the data, as biased data can lead to feedback loops and unreliable crime predictions (Hunt et al., 2014; Mohler et al., 2016; Ratcliffe et al., 2021). Official police data, which is commonly used in many studies, has inherent challenges that can introduce measurement errors and inconsistencies in the results. Reliance on official crime reports, in particular, has been highlighted as a potential source of bias and measurement issues (Callazos et al., 2020). In studies evaluating place-based policing interventions, problems may arise if the spatial and temporal components of crime data are not accurately collected or systematically registered, although the extent of this issue can vary depending on the type of crime. However, inconsistencies in data quality are even more problematic in person-based policing interventions, as errors in the data can result in wrongful convictions (Ariel et al., 2016, 2021). Therefore, it has been argued that researchers should not solely rely on official police data as the primary source for crime intelligence analysis. For example, supplementing official crime records with human intelligence gathered from crime control partners can significantly enhance the accuracy and completeness of the data (Morton et al., 2018). By leveraging additional sources of information, researchers can overcome some of the limitations and potential biases associated with relying solely on official police data.

Statistical Regression to the Mean

Statistical regression to the mean is another challenge that has been recognized by some of the included studies (e.g., Mastrubuoni, 2020; Ratcliffe et al., 2011). Regression to the mean refers to the phenomenon where initially high scores on a variable tend to move closer to the mean when measured at different points in time. This challenge is particularly relevant in ILP studies evaluating interventions that focus on prolific offenders, high-crime locations, repeat victims, and criminal groups. In such studies, treatment and control groups are often assigned based on extreme cases. Consequently, when crime rates are measured between two time periods, extreme cases may exhibit less extreme behavior during the second measurement, suggesting the intervention/program was successful when it was not (Farrington & Welsh, 2006; Galton, 1886; Ratcliffe et al., 2011; Twisk & De Vente, 2008). To address this challenge, it has been suggested that (quasi-)experimental studies should account for differences in extreme scores between the treatment and control groups, a task that is typically easier to control for in RCTs. One approach is to use a tailored analysis that includes the pre-intervention crime level as a covariate in the crime impact model (Ratcliffe et al., 2011). By considering the pre-intervention crime level, researchers can better assess the actual impact of the intervention while accounting for the regression to the mean effect.

Treatment Integrity

Several studies highlighted that ensuring the integrity of both the treatment and control conditions was a practical challenge (e.g., Corsaro et al., 2012; Novak et al., 2016; Rydberg et al., 2018). Failing to maintain the integrity of the experiment can lead to an over- or underestimation of the intervention’s effectiveness. To address this issue, researchers implemented procedural and organizational safeguards to prevent contamination or interference.

One important aspect mentioned in this context is treatment fidelity, which refers to the extent to which participants adhere to the experimental protocol. Various methods were employed to enhance monitoring and compliance. For example, technological applications such as AVL systems were utilized to track police presence (Ariel et al., 2016; Ariel & Partridge, 2016; Telep et al., 2014; Weisburd et al., 2015). More traditional approaches such as log-in systems, call-ins, and observations were also employed (Ratcliffe et al., 2011).

Some studies established deliberative structures, such as task forces, to facilitate regular meetings and briefings. These structures provided a platform for discussing tactical decisions, addressing requirements in the police intelligence infrastructure, and ensuring accountability among stakeholders (e.g., Ariel et al., 2020; Caplan et al., 2021; Ratcliffe et al., 2021; Santos & Santos, 2016). They also enabled the evaluation of qualitative insights and challenges arising from interactions with commanders and police officers, which may prove valuable for evaluating and implementing an intelligence-led policing approach. However, it is crucial to maintain confidentiality and avoid disclosing sensitive information about the experimental protocol during cooperation and communication with stakeholders, in order to prevent treatment integrity failures.

Accounting for the (Different) Objectives of Different Stakeholder Groups

As mentioned earlier, cooperation and deliberation play a crucial role in mitigating treatment integrity failures (Ariel et al., 2020; Caplan et al., 2021; Ratcliffe et al., 2021; Santos & Santos, 2016). However, it can be challenging to consider the diverse needs, objectives, capabilities, and limitations of the stakeholders involved in the experiment. One important aspect to consider is the nature of the ‘policing craft,' which encompasses the skills, knowledge, heuristics, attitudes, and acquired expertise of police officers in their daily duties (Caplan et al., 2021; Ratcliffe et al., 2011, 2021). These factors can influence officers’ perceptions and attitudes towards the experiment and specific policing approaches. Officers may feel constrained in their daily tasks when required to adhere to a protocol and meet specific criteria (Ratcliffe et al., 2011, 2021).

The novelty or level of institutionalization of the evaluated intervention within the police department, can also impact officers’ willingness to actively participate. Moreover, (quasi-)experimental studies necessitate institutional collaboration, forming coalitions between police administration and academic staff with potentially conflicting interests and aims. Limited resources (e.g., staff) available in a policing ecosystem, however, can impede scientific goals (Kennedy et al., 2022). Additionally, the aims and preferences of the participating police department may impose practical limitations, leading to methodological compromises from the outset of the experiment (Novak et al., 2016). In addition, some studies have advocated including other stakeholders, such as citizens, criminals, or victims who are directly or indirectly affected by the intervention, to enhance awareness and communication about specific policing methods (Groff et al., 2015; Ratcliffe et al., 2011). However, it is crucial to ensure that communication does not compromise treatment integrity.

Crime and Harm Reduction

Appendix 1 clearly shows that most of the studies focused on establishing crime reduction effects, whereas only one explicitly focused on measuring and reducing social harm in so-called social harm spots (Carter et al., 2021). As discussed earlier, establishing statistically significant reduction effects is inherently challenging and influenced by various methodological factors. Nonetheless, researchers often argued that observed reductions in crime or harm measures were a result of the successful deployment of police resources to targeted areas or individuals identified through crime intelligence analysis. For instance, several studies investigated the implementation of police crackdowns in hot spots, leading to initial and residual deterrence by increasing the variability of risk estimates and uncertainty about sanction risks (Ariel & Partridge, 2016; Ratcliffe et al., 2011). In some studies, it is argued that varying the frequency and intensity of police presence can be more effective in addressing criminals’ potential adaptation to the policing approaches imposed (Ariel et al., 2016; Telep et al., 2014).

Furthermore, deterrence has been recognized as a significant mechanism through which offender-focused policing interventions can yield crime reduction effects. This mainly includes communicating deterrent messages of ‘zero-tolerance' to deter potential future offenders (Ariel et al., 2019; Saunders et al., 2016). Two studies had different emphases, with one evaluating whether an ILP program resulted in increased human intelligence collection and detection (Morton et al., 2019), while the other primarily aimed to enhance police productivity through predictive policing to detect criminal activity (Mastrubuoni, 2020). Although these studies did not focus primarily on crime or harm reduction, they are relevant within the framework of ILP, as they demonstrate the positive effects of ILP-focused interventions on the internal processes of law enforcement agencies, improving their intelligence capabilities and relationships with specific communities.

Crime Displacement and the Diffusion of Crime Control Benefits

In addition to analyzing the effects on crime, most place-based policing studies also quantified crime displacement effects, including the diffusion of crime control benefits (a concept introduced by Clarke & Weisburd, 1994) to adjacent areas (e.g., Ariel et al., 2016; Caplan et al., 2021; Collazos et al., 2020; Corsaro et al., 2021; Lum et al., 2011; MacDonald et al., 2016; Rosenfeld et al., 2014; Santos & Santos, 2015; Telep et al., 2014). ⁵ Controlling for displacement effects was usually done by creating so-called geographical buffer or cushion zones (e.g., Ariel et al., 2016; Piza et al., 2015; Piza & O’Hara, 2014). Crime displacement was then measured by using specific mathematical parameters such as the weighted displacement quotient (WDQ) (e.g., Ariel et al., 2016; Groff et al., 2015; Piza et al., 2015; Piza & O’Hara, 2014; Ratcliffe et al., 2011, 2017) or the weighted displacement difference statistic (Kennedy et al., 2022), taking into account the crime rates or counts prior to and during the intervention in both treatment and displacement sites, including control sites for each. As can be seen from Appendix 1, the findings of most studies did not support crime displacement, and only a few studies reported a diffusion of crime control benefits. This is in line with the place-based policing literature, which contends that a diffusion of crime control benefits is more likely to occur than crime displacement (Braga et al., 2019).

User Experience(s)

When we refer to user experience(s) as a secondary outcome, we are focusing on the effects of ILP-related interventions on the perceptions and experiences of the various actors involved in evaluating and implementing specific interventions. Several studies employed qualitative research methods, such as field observations or interviews, to formally assess and report on the experiences of these actors during the interventions.

For instance, in the Philadelphia policing tactics experiment, Groff et al. (2015) conducted a survey and found that police officers supported the use of actionable knowledge as part of an offender-focused policing approach rather than relying solely on intuition or so-called ‘gut feelings.' Similarly, officers in the Shreveport predictive policing experiment emphasized the value of predictive hot spot maps in providing concrete plans for targeted patrol activities (Hunt et al., 2014). However, officers also acknowledged that they sometimes followed their ‘gut feelings' rather than the knowledge provided by intelligence products when the information contradicted their own common-sense wisdom. Consequently, despite the potential benefits of new technologies like LPR systems (Lum et al., 2011) or predictive algorithms (Ratcliffe et al., 2020), officers were sometimes skeptical about adopting these new tools.

In this context, Saunders et al. (2016), who conducted interviews with police officers on the use of tactical or strategic intelligence products, argued that there needs to be practical reasoning and guidance from police command staff to frontline officers on how to effectively employ intelligence in practice. This raises practical questions about the adaptability of intelligence within the context of police administration. For example, from a theoretical and methodological standpoint, patrolling smaller grid cells (e.g., 200 by 200 meters) with high crime rates may increase the risk of apprehension. However, from a practical perspective, officers may experience constraints from being confined to such small patrol areas and may struggle with determining whom to target or how to operate in these high-crime locations.

Ethical Risks

Ethical risks are inherent concerns related to the collection, analysis, and utilization of data and information for law enforcement purposes, as well as the tactics employed based on criminal intelligence. While user experiences were a focus of several of the studies, a few also considered the potential ethical risks associated with proactive policing interventions based on criminal intelligence.

Most of the identified risks were linked to the impact of policing interventions on the increased exposure of certain areas or individuals to additional police activity. For example, Fitzpatrick et al. (2020) found that when violent crime hot spots received additional police presence through a community-oriented approach, there was no evidence of over-policing of minorities or other populations. Similarly, in the Shreveport predictive policing experiment, officers reported that citizens were more willing to provide additional information when officers patrolled in predicted crime hot spots. This could potentially contribute to improving police–community relationships and enhancing the intelligence capabilities of police departments (Hunt et al., 2014).

In the Philadelphia policing tactics experiment, it was revealed that police officers exercised more discretion when implementing an offender focused ILP intervention (Groff et al., 2015). This resulted in no significant difference in the number of investigative stops conducted by police officers in high-crime areas. The argument was made that stopping and arresting the ‘right people in the right areas' could help to create a perception of procedural justice in law enforcement activities. Similar findings were acknowledged by Carter et al. (2014), who also discovered that citizens generally did not oppose the use of algorithms for decision-making processes by police agencies. However, citizens expressed a preference for some human involvement when decisions were based on intelligence derived from fully automated (big data) algorithms.

These findings suggest that while proactive policing interventions based on criminal intelligence can pose ethical risks, when they are implemented judiciously and with consideration for procedural justice, they may not exacerbate disparities or negatively impact community perceptions. The inclusion of human oversight in decision-making processes also aligns with citizens’ preferences, emphasizing the importance of balancing technological advancements with human involvement to ensure accountability and transparency in law enforcement practices.

Cost-Effectiveness

Finally, some of the studies examined the cost-effectiveness of ILP-related approaches (see Appendix 1), that is, whether the increased effectiveness of an intervention can justify the additional costs associated with its implementation.

For example, in the Shreveport predictive policing experiment, a cost-savings analysis (CSA) was conducted based on the cost estimations for labor and equipment in both the treatment and control districts. The costs were mainly related to the distribution of predictive hot spot maps (intelligence products) to commanders, the collection of extra intelligence that was added to the predictive hot spot maps, the costs associated with running the predictive model on a monthly basis, but also the costs of making tactical and strategic decisions based on the intelligence derived from the predictive hot spot maps, and of executing the strategic and tactical plans needed to address certain crime problems. According to the findings of the CSA, predictive policing in the treatment districts resulted in a 6–10% cost reduction, when compared to control districts where hot spots policing was employed (Hunt et al., 2014).

Evidence for cost-effectiveness was also found in a predictive policing experiment conducted by Mohler et al. (2014) in Los Angeles. In this study, the potential savings to society from applying predictive policing versus hot spots policing were calculated using the methodology developed by McCollister et al. (2010). More specifically, they calculated the societal costs per crime type according to the costs related to the victim, the police and court system, and the offender. These cost estimates were then used to compute the weekly savings per police division that would result from a weekly crime reduction of 4.3 crimes. Based on these calculations, it was estimated that using predictive hot spot maps for 31 minutes per day for each hot spot would save the LAPD $17,258,801 per year vis-à-vis applying traditional hot spot mapping techniques by crime analysts, which would only save half as much (Mohler et al., 2015). These findings suggest that predictive policing methods may be more cost-effective than traditional hot spots policing tactics, albeit the robustness and validity of these projections should be critically scrutinized at all times.

Nonetheless, evidence of cost-effectiveness has also been found in studies employing traditional hot spots policing approaches. For example, in the Indianapolis harm spot policing experiment, the application of hot spots policing in crime and social harm hot spots was translated into potential cost savings to society. It was calculated that per 10.4 minutes of officer proactive activity in social harm (hot) spots, the costs of social harm were reduced by $38.6, a total of $118,232 during the experimental period (Carter et al., 2014). The costs of deploying police resources were not reported, however.