PROTOCOL: Systematic Review of the Effects of “Pulling Levers” Focused Deterrence Strategies on Crime

Types of Studies

Only studies that use comparison group designs involving before and after measures will be eligible for the main analyses of this review. In most pulling levers evaluations (e.g. Braga et al. 2001; McGarrell et al. 2006), the control group experiences routine modern police responses to crime. Control areas usually experience a blend of traditional police responses (e.g., random patrol, rapid response, and ad-hoc investigations) and opportunistic community problem-solving responses. While strategic interventions developed from community policing initiatives may be present in certain control areas, none of the controls engage pulling levers policing strategies to address crime problems. As part of the review, we will also examine and document less rigorous research evaluation evidence (e.g. simple pre-post assessments without control groups). Although we have strong concerns regarding the methodological rigor of such studies, we will identify and analyze them separately from our main analyses.

Types of Areas

Pulling levers strategies attempt to influence the criminal behavior of individuals through the strategic application of enforcement and social service resources to facilitate desirable behaviors. However, existing reviews of the crime prevention value of pulling levers strategies have noted that published evaluations only report aggregated measures of underlying levels of criminal behaviors in targeted areas (Wellford et al. 2005). For example, in Boston, the pulling levers intervention targeted violent behavior among gang-involved offenders. The evaluators measured behavioral change among gang-involved offenders by examining city-wide trends in aggregated measures of serious violence (Braga et al. 2001). Since most of the existing evidence on the crime prevention value of pulling levers involves aggregated measures of changes in individual behaviors, this review will be limited to area-level analyses of aggregated crime trends. If the search strategies reveal new or unpublished studies that examine individual-level analyses of changes in individual criminal behavior, these studies will be included in our examination as additional descriptive information on the crime prevention mechanisms associated with the pulling levers approach. However, individual-level studies will not be included in the formal systematic review.

It is important to note here that the research strategy will yield a diverse set of targeted areas across the identified studies. For example, evaluations of pulling levers policing strategies in Boston and Indianapolis compared the homicide trends in the targeted cities to homicide trends in nonrandomized groups of control cities (Braga et al. 2001; McGarrell et al. 2006). In evaluations of pulling levers programs implemented in Los Angeles and Chicago, however, compared violent crime trends in treatment neighborhoods to violent crime trends in non-randomized groups of control neighborhoods (Tita et al. 2003; Papachristos et al. 2007). This heterogeneity in the units of analysis across studies could have varying and policy-relevant effects on crime prevention outcomes associated with the pulling levers strategies. As such, the review will also classify the types of areas to ensure that the review is measuring similar findings across the potentially diverse set of locations subjected to treatment.

Types of Interventions

To be eligible for this review, interventions must follow the basic steps of the pulling levers policing strategy. As described by Kennedy (2006: 156-157), pulling levers operations have tended to follow this basic framework:

Selection of a particular crime problem, such as youth homicide or street drug dealing.

Types of Outcome Measures

Eligible studies must measure the effects of police intervention on officially recorded levels of crime at places. Appropriate measures of crime could include crime incident reports, citizen emergency calls for service, or arrest data. Other outcomes measures such as survey, interview, and victimization measures used by eligible studies to measure program effectiveness will also be coded and analyzed. While all eligible studies must include a crime outcome measure, we will also collect data on community satisfaction measures such as citizen attitudes towards police, fear of crime, and other outcomes.

Particular attention was paid to studies that measured crime displacement effects and diffusion of crime control benefit effects. Kennedy (2006) describes a place-based application of pulling levers focused on a disorderly drug market operating in High Point, North Carolina. Policing strategies focused on specific locations have been criticized as resulting in displacement (see Repetto 1976). More recently, academics have observed that crime prevention programs may result in the complete opposite of displacement—that crime control benefits were greater than expected and “spill over” into places beyond the target areas (Clarke and Weisburd 1994). The quality of the methodologies used to measure displacement and diffusion effects, as well as the types of displacement (spatial, temporal, target, modus operandi) examined, was assessed.

Selection of studies

The reviewer will screen abstracts and leads to potentially eligible studies and decide which full-text reports should be acquired. Only the full-text papers of titles and abstracts indicating, or potentially indicating, an evaluation of a pulling levers strategy will be obtained. Studies that use randomized controlled designs or quasi-experimental techniques such as matching, statistical controls, comparison groups, and the like will be considered for inclusion in the review. In cases of ambiguity, the full text of the study will be obtained in order to properly determine whether an eligible study design was used. Correlation and observational studies without control groups that examine the effects of pulling levers policing on crime will be noted and appear in catalog form in an appendix to the final report. These descriptive studies will not be included in the formal analysis reporting the findings of the review.

Assessment of Methodological Quality

Studies meeting the eligibility set forth above will be coded for a range of characteristics related to methodological quality including the definition criteria used to identify the units of analysis, the statistical tests used to determine crime prevention effectiveness, the measurement of displacement, the violation of randomization procedures, case attrition from the study, and the subversion of the experiment by participants. Farrington (2003) proposes five easily understood methodological criteria to assess the methodological quality of evaluation studies. These criteria include statistical conclusion validity, internal validity, construct validity, external validity, and descriptive validity. As appropriate and possible, the role of the various methodological factors on the observed empirical results will be assessed. However, it is important to recognize that eligible studies may not detail or even mention implementation issues. Indeed, all field experiments face implementation difficulties and care will be taken not to artificially downgrade the value of certain studies simply because study provided an open account of potential process problems.

Data Management and Extraction

The reviewer, with the help of a trained research assistant, will extract information from the full text report on the characteristics of the study using a carefully designed data extraction instrument (see coding instruments included in Appendix A). A content analysis will be conducted on the full text of the report and the data extraction instrument will capture data on the relevant dimensions of this review. These dimensions include: a complete description of the treatment, methods used to define and identify targeted areas, research design and statistical techniques, threats to the research design, crime outcome measures, and alternative outcome measures. When important information is missing from available study reports, the original researchers will be contacted, if possible, to determine if they can supply that information.

Determination of Independent Findings

A single evaluation of pulling levers policing intervention may provide data on multiple outcome measures. For example, the quasi-experimental evaluation of Boston's Operation Ceasefire intervention presents an array of outcome measures including youth homicide incident data, gun assault incident data, and “shots fired” citizen calls for service data (Braga et al. 2001). Separate studies by Winship and Berrien (1999) and Stoutland (2001) reported community perceptions of the value of Operation Ceasefire in preventing gang violence. Pulling levers interventions could also be targeted at chronic offenders responsible for generating a variety of offenses such as violent crimes, property crimes, and drug crimes. The treatment could have varying effects on offending trends and patterns in different crime categories.

For cases such as this with multiple findings from the same sample, each will be examined independently to decide how to either combine the findings or to choose the one that best represents the study. It is likely that most pulling levers interventions will be designed to deal with a specific problem, but some may also target some secondary problems and report outcomes for these as well. In these cases the effect size for the primary problem will be reported. For instance, while multiple outcomes are reported, the Braga et al. (2001) evaluation clearly identifies the monthly counts of “youth homicide” (homicide victims 24 and younger) in Boston as the primary outcome measure.

However, some studies might have multiple primary outcomes. Analyzing these separately would clearly lead to problems regarding statistical dependence of outcomes. As such, we will code a maximum of three primary outcomes, with the criteria of choosing the maximum, moderate/median effect size and minimum effect size to offer flexibility in calculating an overall effect size for such studies. The same strategy will be used for any studies reporting the same outcome multiple times with different types of data (i.e. a study evaluating the impact of a pulling levers on “gun violence” may use gun homicide incidents, gun assault incidents, and “shots fired” calls for service as primary outcome measures). Finally, some studies may involve multiple sites—i.e. pulling levers program delivered by one police department/taskforce to specific problems in multiple areas within a city. Such cases will be treated as one study with sub units, and independent effect sizes for primary outcomes will again be created in the same manner as above.

Statistical Procedures and Conventions

Analysis of outcome measures across studies will be carried out in a uniform manner and, where appropriate and possible, involve quantitative analytical methods. As described above, existing reviews of the effects of pulling levers policing on crime have identified only a handful studies that use control group evaluation designs. The expanded search strategy and time period described below may identify a few additional studies. The review will consist of simple descriptive statistics reporting the proportion of studies reporting significant effects on outcome measures, the size of the effect, and the direction of the effect. At a minimum, the systematic review will rely upon “vote counting” procedures to assess the effects of the pulling levers interventions on crime. In this rudimentary approach, each study metaphorically casts a vote for or against the effectiveness of treatment.

Unfortunately, vote counting methods of synthesizing results across studies suffer from a number of limitations. These weaknesses include: failing to account for the differential precision of the studies being reviewed (e.g. larger studies, all else being equal, provide more precise estimates), failing to recognize the fundamental asymmetry of statistical significance tests (e.g. a large proportion of nonsignificant findings in the same direction provide evidence that the null hypothesis is false), ignoring the size and direction of observed program effects, and, if the statistical power of the studies in that area of concern is low, the likelihood of arriving at an incorrect conclusion increases as the number of studies on a topic increases (Wilson 2001: 73–74). Meta-analyses of program effects avoid these pitfalls by focusing on the size and direction of the effects, not whether the individual effects were statistically significant, and by weighting effect sizes based on the variance of the effect size and the study sample size (Lipsey and Wilson 2001).

Meta-analytic procedures will be used to combine data from studies. For eligible studies, with enough data present, effect sizes will be calculated using the standardized measures of effect sizes as suggested in the meta-analytic literature (e.g. see Lipsey and Wilson 2001). Mean effect sizes will be computed across studies and we will use a correction such as the inverse variance weight for computing the associated standard error. For eligible studies that contrast two groups that have a continuous underlying distribution, the standardized mean difference effect size (also known as Cohen's d; see Rosenthal 1994; Shadish et al. 2003) will be used in our meta-analysis to synthesize results. This effect size measure would be appropriate for eligible evaluations that contrast an outcome measure, such as mean homicide counts, for treatment and control areas over pre-test and post-test observation periods.

However, a priori, we know that several pulling levers policing quasi-experimental evaluations, such as the Boston Operation Ceasefire evaluation (Braga et al. 2001), use interrupted time series analyses that compare trends in serious violence in treated areas to trends in serious violence in comparison areas. Controlling for other rival factors, seasonal variations, and existing secular trends, program impacts are determined by estimating the change in outcome measures via a dummy variable indicating the absence and presence of the pulling levers program over the time series. The estimated program effect size, direction, and statistical significance level are then compared to similar coefficients estimating pre-post contrasts in outcome trends in comparison areas. In our proposed systematic review, we will qualitatively examine the assessment of the treatment coefficient relative to comparison coefficients in each eligible study to determine whether study comes to an appropriate conclusion regarding the absence or presence of an actual treatment effect. Then, for each eligible study, we will calculate standardized mean gain effect size coefficients (a measure that calculates a standardized pre-post contrast that can be compared across samples and studies; see Lipsey and Wilson 2001: 44 – 45) to be used in our meta-analysis to synthesize results.

In our proposed meta-analyses, we will examine the Q statistic to assess heterogeneity of effect sizes across studies, though it is our initial assumption that effect size is a fixed factor in our analysis. Our assumption that fixed effects comes from our view that pulling levers programs are fairly consistent one to another. Nonetheless, we recognize that there may be considerable diversity across programs that have not been so far recognized. If the Q statistic and our reading of the studies suggests this we will implement a random effects model for all analyses involving effect sizes.

We also hope to examine contextual or moderating features of pulling levers approaches. We are interested in whether the strength of the effect varies across departments or other contextual variables. To assess this we will use the analog to the ANOVA method of moderator analysis (see Lipsey and Wilson 2001) for categorical moderator variables and meta-analytic regression analysis for continuous moderator variables or analyses involving multiple moderators.

As described earlier, we will also examine and document less rigorous research evaluation evidence (e.g. simple pre-post assessments without control groups) on the crime prevention effectiveness of pulling levers strategies. Following the methods of a recent systematic review by Weisburd et al. (2010), we will report the percent change for the pre-post studies and the average percent change for all pre-post studies. When more than one primary outcome is present, we will average these outcomes to create a single outcome. To account for variation in sample size, we will also calculate a weighted average percent change by weighting each study by the inverse of its variance (assuming that crime follows a Poisson distribution). The sampling variance will allow us to construct a confidence interval around the percent change of each study and, after weighting each study by the inverse of its variance, we will recalculate the average percent change. This analysis will be separate from the main analyses described in this section and will receive far less emphasis in our conclusions on the crime prevention value of pulling levers strategies.

Treatment of Qualitative Research

Qualitative research on crime and disorder outcomes will not be formally included in this systematic review. Qualitative insights on the crime prevention value of pulling levers policing will be included as descriptive information in the review report. If the search strategies reveal a number of qualitative studies, the authors will engage a qualitative researcher to assist in future updates to this review with a synthesis of qualitative evaluation measures.