Problem-oriented policing for reducing crime and disorder: An updated systematic review and meta-analysis

Problem-oriented policing (POP) is associated with reductions in crime and disorder

POP is associated with statistically significant reductions in crime and disorder. Place-based POP programs are more likely to produce a diffusion of benefits into areas adjacent to targeted locations than to lead to crime displacement.

What is this review about?

POP is a proactive policing strategy developed by Herman Goldstein, who argued that the standard reactive model of policing was ineffective as it was overly focused on the means of policing (number of arrests, average response time, etc.) rather than the end goal of reducing crime and enhancing community safety. He suggested that police could be more effective if they were more proactive and researched root causes of crime, and developed tailor-made responses.

This review assesses the effectiveness of POP interventions—defined as those programs which generally followed the tenets of the SARA model (scanning, analysis, response, assessment) developed by Spelman and Eck—in reducing crime and disorder and fear of crime, and improving citizen perceptions of police.

What is the aim of this review?

This update of a Campbell systematic review assesses the effectiveness of problem-oriented policing in reducing crime and disorder. It summarises the evidence from 34 studies: 28 from the United States, five from the United Kingdom, and one from Canada.

The issue

In an article in Crime & Delinquency in 1979, Herman Goldstein critiqued police practices of the time by noting that they were more focused on the “means” of policing than the “ends” or goals of policing. His critique drew from a series of recently completed studies that suggested that such standard policing practices as “preventive patrol” (Kelling, Pate, Dieckman, & Brown, 1974) or “rapid patrol car response to calls for service” (Kansas City Police Department, 1977) had little impact on crime. Goldstein suggested that the research evidence was not idiosyncratic but reflected a crisis in policing. To illustrate his concern, he referred to a newspaper article in the United Kingdom that reported on bus drivers in a small city that were passing bus stops waving and smiling but failing to pick up passengers. When questioned by a reporter, a representative for the bus company responded that “it is impossible for the drivers to keep their timetable if they have to stop for passengers” (Goldstein, 1979, p. 236). Goldstein termed this the “means over ends syndrome” and noted that police were particularly susceptible to this problem. Goldstein noted that the police too had become so focused on the means of policing—such issues as the staffing and management of police—that they had begun to ignore the problems policing was meant to solve. Goldstein saw this dysfunction as at the heart of the failures of the police to be effective in addressing community problems.

Goldstein called for a paradigm shift in policing that would replace the primarily reactive, incident driven “standard model of policing” (Skogan & Frydl, 2004; Weisburd & Eck, 2004) with a model that required the police to be proactive in identifying underlying problems that could be targeted to alleviate crime and disorder at their roots. He termed this new approach “problem-oriented policing” to accentuate its call for police to focus on problems and not on the everyday management of police agencies. Goldstein also expanded the traditional mandate of policing beyond crime and law enforcement. He argued that the police should deal with an array of problems in the community, including not only crime, but also social and physical disorder.

He also called for police to expand the tools of policing much beyond the law enforcement powers that are seen as the predominant tools of the standard model of policing. In Goldstein's view, the police needed to draw upon not only the criminal law, but also civil statutes and rely on other municipal and community resources if they were to successfully ameliorate crime and disorder problems. As such, successful implementations of POP would be reliant on forming partnerships with other agencies, community organizations and community members to deliver non-law enforcement responses. This would particularly be the case when the targeted problems do not necessarily involve law violations.

John Eck and William Spelman (1987) drew upon Goldstein's idea to create a straightforward model for implementing POP, which has become widely accepted. In an application of problem solving in Newport News, in which Goldstein acted as a consultant, they developed the SARA model for problem solving. SARA is an acronym representing four steps they suggest police should follow when implementing POP, which will be outlined in Section 2.2.

A 2004 report from the National Research Council offered the following description of POP and how the SARA model works in practice:

The heart of problem-oriented policing is that this concept calls on police to analyze problems, which can include learning more about victims as well as offenders, and to consider carefully why they came together where they did. The interconnectedness of person, place, and seemingly unrelated events needs to be examined and documented. Then police are to craft responses that may go beyond traditional police practices … Finally, problem-oriented policing calls for police to assess how well they are doing. Did it work? What worked, exactly? Did the project fail because they had the wrong idea, or did they have a good idea but fail to implement it properly? (Skogan & Frydl, 2004, p. 91).

A number of studies going back to the mid-1980s demonstrate that problem solving can be utilized to address a variety of police concerns, including fear of crime (Cordner, 1986), violent and property crime (Eck & Spelman, 1987), firearm-related youth homicide (Kennedy, Braga, Piehl, & Waring., 2001), and various forms of disorder, including prostitution and drug dealing (Capowich & Roehl, 1994; Eck & Spelman, 1987; Hope, 1994). As a further example of the proliferation of POP, the Center for Problem-Oriented Policing (POP Center, https://popcenter.asu.edu/) documents a large number of case studies and evaluations of POP. For instance, there have been over 1,000 programs submitted for consideration for the Goldstein Award and more than 800 submissions to the Tilley Award. These submissions document the use of a wide array of problem-solving responses to document crime, disorder and a host of other issues police are tasked with addressing, highlighting the utility of the POP model for a wide variety of problem types (see also, Scott, 2000; Scott & Clarke, 2020). As our review is focused on impacts on crime and disorder, we limit our discussion here to those outcomes.

There are also a number of experimental and other more rigorous examinations of POP. For example, a study in Jersey City, New Jersey, public housing complexes (Mazerolle, Ready, Terrill & Waring, 2000) found that a police problem-solving model could be used to respond to violent and property crime in six housing complexes. In another example, Clarke and Goldstein (2002) report a POP project to reduce thefts of appliances from new home construction in Charlotte, North Carolina. Officers carefully analyzed this problem before working with construction firms to implement changes in building practices.

Two early experimental evaluations of applications of problem solving in crime hot spots (Braga et al., 1999; Weisburd & Green, 1995) suggested POP interventions, particular those implemented in crime hot spots, could be evaluated rigorously.

In sum, POP has emerged as one of the most widely accepted and widely used strategies in American policing (Scott, 2000; Weisburd & Majmundar, 2018). This is indicated both by the adoption of POP by major federal agencies and national policing groups, the creation of national awards for effective POP programs, and the widespread adoption of the approach in American policing and throughout the world. For example, the U.S. federal agency, the Office of Community Oriented Policing Services (COPS) adopted POP as a key strategy, initially funding the Center for Problem-Oriented Policing, which has developed over 70 problem-specific guides for police. More recently, the Bureau of Justice Assistance has also funded the creation of problem-oriented response and tool guides. The Police Executive Research Forum adopted POP as a “powerful tool in the policing arsenal,” in the 1980s and began to run a yearly national conference to promulgate and advance POP strategies (Solé Brito & Allan, 1999, p. xiii) that the POP Center still continues today. In 1993 the Herman Goldstein Award was created for “problem solving excellence.” In the United Kingdom, the Tilley Award for POP was created in 1999.

While POP has been widely adopted and assessed, it is important to note that fully implementing POP has been challenging (Cordner & Biebel, 2005; Maguire, Uchida, & Hassell, 2015), and programs are often characterized by partial implementations of the SARA model that have been termed “shallow” problem-solving (Braga & Weisburd, 2010). For instance, in his Stockholm lecture Goldstein (2018) noted that many early initiatives lacked fundamental understanding of the POP approach and that he did not adequately acknowledge the importance of having enough individuals with the requisite research and assessment skills when developing his model. However, he also noted that he has been impressed by improvements in areas such as focusing specifically on micro-problems, the engagement of rank-and-file officers in problem solving, the use of a broad range of responses, and increasing engagement of the private sector in partnerships to share responsibility for public safety problems. Thus while POP still has a long way to go to be fully embedded in police agencies, much less to become standard police practice as Goldstein hoped, there is reason to believe that the model is both spreading and improving in quality over time.

POP in practice

Since its initial proposition, the POP model has been further articulated by Eck and Spelman (1987) whose work in Newport News produced the SARA model. SARA is an acronym representing four steps they suggest police should follow when implementing POP. “Scanning” is the first step, and involves the police identifying and prioritizing potential problems in their jurisdiction that may be causing crime and disorder. After potential problems have been identified, the next step is “Analysis.” This involves the police analyzing the identified problem(s) in-depth using a variety of data sources so that appropriate responses can be developed. The third step, “Response,” has the police developing and implementing interventions tailored to what was learned in the “analysis” step and designed to solve the problem(s). The search for responses should be broad and not limited to law enforcement methods, and often should involve partnering with other agencies, community groups and/or community members depending on the type of problem and its causes. Indeed, the POP model stresses the need to shift and share responsibility for public safety, and this will require police to identify and mobilize partners (Goldstein, 2018; Scott & Goldstein, 2005). Finally, once the response has been administered, the final step is “Assessment” which involves assessing the impact of the response on the targeted problem(s).

For example, a police agency may determine that drug-related crime is on the rise in their jurisdiction, constituting a problem in need of prioritization (Scanning phase). Further examination of the nature of drug-related crime may reveal problem areas and times (Analysis phase). Based on this analysis, the agency may choose to direct increased patrol and enforcement to the specific areas deemed problematic at the specific times deemed problematic and to partner with community organizations to deliver substance abuse treatment programs (Response phase). After a period of time the agency may compare drug-related crime in the jurisdiction as a whole, as well as in the targeted areas, from before and after the response was implemented (Assessment phase).

This process in general, rather than the specific problem or response chosen, represents the core concept of POP. Thus, a diverse set of variations in problems, responses, and length of interventions are possible across an array of targets (i.e., problem places of varying size or problem people may be the focus) and virtually any unit of analysis.

How might POP work?

It is hypothesized that POP affects change in problem outcomes through an increased knowledge of, and responsiveness to, the mechanisms through which a particular problem operates. The National Academies of Sciences Committee on Proactive Policing noted in its consensus report:

Problem-oriented policing is an analytic method for developing crime reduction tactics. This strategy draws upon theories of criminal opportunity, such as rational choice and routine activities, to analyze crime problems and develop appropriate responses (Braga, 2008; Clarke, 1997; Reisig, 2010). Using a basic iterative process of problem identification, analysis, response, assessment, and adjustment of the response (often called the scanning, analysis, response, and assessment [SARA] model), this adaptable and dynamic analytic method provides a framework for uncovering the complex mechanisms at play in crime problems and for developing tailor-made interventions to address the underlying conditions that cause crime problems (Eck & Spelman, 1987; Goldstein, 1990). Depending on the nuances of particular problems, the responses that are developed—even for seemingly similar problems—can be diverse. Indeed, problem-oriented policing interventions draw upon a variety of tactics and practices, ranging from arrest of offenders and modification of the physical environment to engagement with community members (Weisburd & Majmundar, 2018, p. 53).

POP is not concerned simply with the problem outcomes themselves but rather with the underlying processes that lead to problems emerging and developing. Addressing the underlying mechanisms that cause problems should lead to long-term solutions and should lead police agencies to think and act in ways that go beyond their normal day-to-day operations. Furthermore, the assessment of results should lead to refinement and improvement in subsequent efforts.

Moreover, while the POP model does not favor any particular kind of intervention, one can still look beyond this literature and find support for the basic ideas behind the models by examining evidence for approaches commonly utilized in POP programs. For instance, evidence of the effectiveness of situational and opportunity-blocking strategies, while not necessarily police based, provides indirect support for the effectiveness of problem solving in reducing crime and disorder. Moreover, POP has been linked to routine activity theory, rational choice perspectives, and situational crime prevention (Clarke, 1992a, 1992b; Eck & Spelman, 1987). Reviews of prevention programs designed to block crime and disorder opportunities in small places find that most of the studies report reductions in target crime and disorder events (Eck, 2002; Poyner, 1981; Weisburd & Telep, 2014; Weisburd, 1997),

Why is it important to do this review?

This is an update to an earlier Campbell systematic review of the effectiveness of POP that included studies through 2006 and identified a total of 10 studies that met the Campbell criterion for inclusion—4 randomized experiments and 6 quasi-experiments (Weisburd et al., 2008, 2010). Overall, the findings of this review largely reinforced those of prior narrative reviews and more general assumptions of the effectiveness of POP. Specifically, the authors noted “[w]hether we used a more conservative mean effect size approach or examined the largest effects on crime and disorder reported, we found that POP approaches have a statistically significant effect on the outcomes examined. Importantly, the results are similar whether we look at experimental or nonexperimental studies” (Weisburd et al., 2010, p. 162).

However, the original review also noted that effect sizes were relatively modest, ranging between 0.10 and 0.20 (measured as Cohen's D) and were based upon only 10 experimental or quasi-experimental studies. As such, an updated review may help to shed further light on the ability of POP to reduce crime and disorder problems by analyzing an increased base of empirical research on POP interventions by including studies up to the end of 2018 (12 years beyond the cutoff for the original review's search). Having more complete and current evidence on POP is especially important given an increasing focus on problem-solving and other proactive policing approaches around the world (Weisburd & Majmundar, 2008).

We also add an additional approach to measuring effect sizes suggested by Wilson (in progress) that has statistical properties better suited to the nature of place-based data and provides a more easily interpretable set of estimates of program outcomes. As empirical knowledge on POP's effectiveness increases, police agencies may be able to better determine ways to identify and respond to the various problems occurring in their jurisdictions.

Criteria for considering studies for this review

Search strategy for identification of studies

The search for this updated review was led by the Global Policing Database (GPD) research team at the University of Queensland (Elizabeth Eggins and Lorraine Mazerolle) and Queensland University of Technology (Angela Higginson). The University of Queensland is home to the GPD (see http://www.gpd.uq.edu.au), which served as the main search location for this review. The GPD is a web-based and searchable database designed to capture all published and unpublished experimental and quasi-experimental evaluations of policing interventions conducted since 1950. There are no restrictions on the type of policing technique, type of outcome measure or language of the research (Higginson et al., 2015). The GPD is compiled using systematic search and screening techniques, which are reported in Higginson et al. (2015) and summarized in detail in Appendices B and C. Broadly, the GPD search protocol includes an extensive range of search locations to ensure that both published and unpublished research is captured across criminology and allied disciplines.

To capture studies, we used POP terms to search the GPD corpus of full-text documents that have been screened as reporting on a quantitative impact evaluation of a policing intervention. Specifically, we used the following terms to search the title and abstract fields of the corpus of documents published from January 2006 through to December 2018:

"problem-orient*”

Several additional strategies were also used to extend the GPD search. First, we performed forward citation searches for works that have cited seminal POP studies.

Several strategies were used to obtain full-text versions of the studies found through our search. First, we attempted to obtain full-text versions from the electronic journals available through the George Mason University, Georgia State University, and Arizona State University libraries. When electronic versions were not available, we used print versions of journals available at the library. If the journals were not available, we made use of both the GPD team and the Interlibrary Loan Office (ILL) to obtain the journal from the libraries of other area schools. When those methods did not work, we contacted the author(s) of the article and/or the agency that conducted and/or funded the research to try to get a copy of the full-text version of the study.

Data collection and analysis

Search results were given title and abstract review by Kevin Petersen, one of the authors of this review. Any studies that were not obviously eligible or ineligible were flagged. Flagged studies were reviewed by the other three authors of this review, who then discussed and voted on each study's eligibility. All inclusion/exclusion decisions were unanimous.

Statistical procedure and conventions

We completed a meta-analysis of the 34 eligible studies by calculating a standardized effect size for each included outcome and then estimating an overall random effect for the impact of POP on crime and disorder. We used Biostat's Comprehensive Meta Analysis 3.0 program for our analyses and to create the forest plots we present below.

Computation of effect sizes in the studies was not always direct. The goal was to convert all observed effects into a standardized mean difference effect size metric. None of the studies we examined calculated standardized effect sizes, and indeed, it was sometimes difficult to develop precise effect size metrics from published materials. This reflects a more general problem in crime and justice with “reporting validity” (Farrington, 2006; Lösel & Köferl, 1989), and has been documented in recent reviews of reporting validity in crime and justice studies (see Perry & Johnson, 2008; Perry, Weisburd, & Hewitt, 2010).

For many of our eligible studies, effect sizes could only be calculated using pre- and postintervention crime/disorder counts for the treatment and control group/area. A similar approach was used for some studies in our earlier review and is common in systematic reviews of policing interventions (e.g., Braga and colleagues’ [2019] recent update of their Campbell review of the effectiveness of hot spots policing).

This approach involves calculating relative incidence rate ratio (RIRR), and the variance of the log RIRR from the raw counts using the following formulae (the table provides an example of the grid of pre and post counts used for these equations):

OPEN IN VIEWER

	Pre	Post
Treatment	a	b
Control	c	d

RIRR = ( a × d ) / ( b × c ) .

V ( log RIRR ) = ( 1 / a ) + ( 1 / b ) + ( 1 / c ) + ( 1 / d ) .

The variance of the log of the RIRR (V(log RIRR)) was adjusted for over-dispersion using the approach outlined by Farrington, Gill, Waples and Argomaniz (2007).

Note we used a slightly different adjustment for over-dispersion in our original review (Weisburd et al., 2008, 2010). We opted to use the current approach for the update to be consistent with more recent Campbell reviews, including Braga and colleagues (2019) recently updated review of hot spots policing (which includes a handful of overlapping studies/outcomes in cases of POP approaches evaluated at hot spots). Additionally, David Wilson (personal communication) has recently developed a new approach for a more robust adjustment for over-dispersion when place-based studies report crime counts for multiple treatment and control areas (or multiple time periods within each group). Unfortunately, we were unable to take advantage of this with our data as too few studies reported on more than one treatment or control area (or only provided aggregated counts by area). For those that did provide disaggregated counts, it was often only two or three areas per group, which is not a sufficient sample for a reasonable adjustment for over-dispersion in our view.

This adjustment is calculated as the product of V(log RIRR) and D, with D = 0.0008 × N + 1.2. N is indexed as the mean number of incidents per case and is calculated as the total number of incidents (a + b + c + d) divided by the total number of treatment plus control areas/groups.

Finally, Cohen's D is obtained by multiplying the log of RIRR by √3/π, while its standard error is calculated by multiplying the adjusted V(log RIRR) by (3/π ²; Hasselblad & Hedges, 1995).

While the Cohen's D approach allows us to compare our findings to the prior review, Wilson (in progress) argues that the Cohen's D approach fails to produce effect sizes that are comparable across studies when based on place-based count data (the majority of studies in our review). Moreover, he has also pointed out that Cohen's Ds obtained through the above conversion are not comparable to those calculated directly through conventional means. As such, we also present meta-analysis models where the effect size is the log RIRR and its standard error (which is the square root of adjusted V(log RIRR)).

One study (Groff et al., 2015) provided the log IRR and its standard error. For three studies, the effect sizes are risk ratios rather than RIRRs. Two studies reported on probation/parole success and failures in a treatment and control group (Knoxville Police Department, 2002; Thomas, 1998) and one reported on students being attacked or not in a treatment and control school (Stokes et al., 1996). While these are calculated slightly differently, they are comparable to RIRR and interpreted in the same manner. The risk ratios were calculated as follows. Referring to the RIRR two-by-two grid above, here a and b are successes (or not being attacked) in the treatment and control group, respectively. Similarly, c and d are the failures (or being attacked) in the two groups. In the traditional notation for risk ratios (see Wilson, in progress), the treatment group successes are indicated by T and the control group successes are labeled C, while N _T and N _C are the sample sizes (successes plus failures) in the treatment and control groups, respectively. RR is then calculated as (T × N _C)/(C × N _T) and its log is the effect size in our models. The variance of log RR is calculated as (N _T − T)/T × N _T) + (N _C − C)/C × N _C) and its square root is the standard error for the log RR.

This approach also has an advantage in that the exponent of the log RIRR can be interpreted simply as the relative percent change in the treatment group compared with the control group.

Selection of studies

Results of the search

Search strategies in systematic reviews return a large number of results that must be screened for eligibility. Utilizing the GPD helped keep this number more manageable. Even though the GPD search strategy is much more comprehensive than those typically employed in searches by researchers conducting individual reviews, the studies included in the database have already been screened and confirmed to be policing evaluations that meet their methodological criterion (see above and the full details provided in Appendices B and C).

The initial steps of the review consisted of reviewing titles and abstracts to eliminate any duplicates and studies that were clearly not evaluations of POP. For any studies that could not be eliminated at this this step, we obtained the full-text of the articles, reports, theses/dissertations or books for careful review to assess whether the interventions and evaluations met the eligibility criterion.

In total, the GPD searches and other strategies used in this review yielded a total of 2,464 results to review. Reviewing titles and abstracts eliminated 1,481 studies which were clearly not evaluations of POP. This left 983 studies which received full-text review. Of these 39 met our eligibility criteria, and 24 provided the quantitative data needed to calculate effect sizes for our meta-analyses. As the original Campbell systematic review of POP (Weisburd et al., 2008, 2010) included 10 experimental or quasi-experimental studies in their main analyses we have a total of 34 studies included in our summary tables and meta-analyses.

Of the 983 publications which received full-text review, 746 studies were Goldstein or Tilley Award submissions (all available award submissions from 2006 to 2018 received full-text review), 204 studies were GPD search results, and 33 studies were identified via forward citation searches.

Figure 1 provides a visual summary of the number of eligible studies by year of publication. As the graph highlights, there was a clear increase in evaluations of POP in the years after the 2006 cutoff for our original review. This uptick was relatively evenly spread over the 12-year period, with each year other than 2014 having between one and five eligible studies. While it is possible that some of the increase is due to the use of the GPD for searches for this update, the data suggest that there has simply been an increase in experimental and quasi-experimental evaluations of POP since 2006. For example, the original review only included 2 Goldstein/Tilley award submissions, while our update found 11 new submissions which met our inclusion criteria and included the data needed for effect size calculations.

OPEN IN VIEWER

Figure 1

Number of eligible problem-oriented policing studies by year (N = 49)

Below we list the 49 studies that met our inclusion criterion. We first list the 34 studies which provided the quantitative data needed to be included in our meta-analyses. We refer to these as “included studies.” As noted above, the publication that provided data for the effect size(s) for each study is listed, and any supplemental publications that were used to complete other parts of our coding of studies are listed in sub-bullets via “see also” notes. Below this, we list the 15 studies that are not included in our analyses due to not providing sufficient data to calculate effect sizes for either of our approaches. Both lists are in chronological order.

Studies included in summary tables and meta-analyses (N = 34):

Sherman, L., Buerger, M., & Gartin, P. (1989). Repeat call address policing: The Minneapolis RECAP Experiment. Washington, DC: Crime Control Institute.

Stone, S. S. (1993). Problem-oriented policing approach to drug enforcement: Atlanta as a case study (Ph.D. dissertation). Emory University.

Weisburd, D., & Green, L. (1995). Policing drug hot spots: The Jersey City drug market analysis experiment. Justice Quarterly, 12(4), 711–735.

Stokes, R., Donahue, N., Caron, D., & Greene, J. R. (1996). Safe travel to and from school: A problem-oriented policing approach. Washington, DC: U.S. Department of Justice.

San Diego Police Department. (1998). Coordinated agency network. San Diego, CA: Herman Goldstein Award Submission.

Braga, A. A., Weisburd, D. L., Waring, E. J., Mazerolle Green, L., Spelman, W., & Gajewski, F. (1999). Problem-oriented policing in violent crime places: A randomized controlled experiment. Criminology, 37(3), 541–580.

◦

See also Braga (1997).

Mazerolle, L., Price, J. F., & Roehl, J. (2000). Civil remedies and drug control: A randomized field trial in Oakland, California. Evaluation Review, 24(2), 212–241.

Knoxville Police Department. (2002). The Knoxville public safety collaborative. Knoxville, TN: Herman Goldstein Award Submission.

Baker, T., & Wolfer, L. (2003). The crime triangle: Alcohol, drug use, and vandalism. Police Practice and Research, 4(1), 47–61.

Nunn, S., Quinet, K., Rowe, K., & Christ, D. (2006). Interdiction day: Covert surveillance operations, drugs, and serious crime in an inner-city neighborhood. Police Quarterly, 9(1), 73–99.

San Angelo Police Department. (2006). “See! It's me!” identity theft prevention program. San Angelo, TX: Herman Goldstein Award Submission.

Tuffin, R., Morris, J., & Poole, A. (2006). An evaluation of the impact of the National Reassurance Policing Programme. London, UK: Home Office Research.

Elliott, M. (2007). An evaluation of specialized police response teams on motel crime (Master's thesis). Reno, NV: University of Nevada.

◦

See also Reno Police Department (2006).

Boston Police Department. (2008). District D-14: Breaking and entering solution plan. Boston, MA: Herman Goldstein Award Submission.

Braga, A. A., & Bond, B. J. (2008). Policing crime and disorder hot spots: A randomized controlled trial. Criminology, 46(3), 577–607.

◦

See also Braga and Bond (2009).

Lancashire Constabulary. (2008). “Moppin” up dodge. Lancashire, UK: Herman Goldstein Award Submission.

Lexington Division of Police. (2009). Community law enforcement action and response program. Lexington, KY: Herman Goldstein Award Submission.

Vancouver Police Department. (2009). Reclaiming the “street of shame”: A problem oriented solution to Vancouver's entertainment district. Vancouver, BC: Herman Goldstein Award Submission.

Guseynov, N. R. (2010). Policing serious crime: A longitudinal examination of geographically focused policing activities (Master's thesis). University of Missouri-Kansas City.

London Borough of Enfield. (2011). Safe as houses- domestic burglary project. London, UK. Herman Goldstein Award Submission.

Niagara County Sheriff's Office. (2011). Operation panther pride. Lockport, NY: Herman Goldstein Award Submission.

Taylor, B., Koper, C. S., & Woods, D. J. (2011). A randomized controlled trial of different policing strategies at hot spots of violent crime. Journal of Experimental Criminology, 7(2), 149–181.

Houston Police Department. (2012). Back from the brink: Reclaiming the Antoine corridor and the development of problem oriented policing within the Houston Police Department. Houston, TX: Herman Goldstein Award Submission.

Lancashire Constabulary. (2012). The custody experience: Reducing 1st time entrants into the criminal justice system. Lancashire, UK: Herman Goldstein Award Submission.

Bichler, G., Schmerler, K., & Enriquez, J. (2013). Curbing nuisance motels: An evaluation of police as place regulators. Policing: An International Journal, 36(2), 437–462.

◦

See also Chula Vista Police Department (2009).

Bond, B. J., & Hajjar, L. M. (2013). Measuring congruence between property crime problems and response strategies: Enhancing the problem-solving process. Police Quarterly, 16(3), 323–338.

Groff, E. R., Ratcliffe, J. H., Haberman, C. P., Sorg, E. T., Joyce, N. M., & Taylor, R. B. (2015). Does what police do at hot spots matter? The Philadelphia policing tactics experiment. Criminology, 53(1), 23–53.

◦

See also Ratcliffe, Groff, Sorg, and Haberman (2015).

Hollywood Police Department. (2015). West district burglary reduction initiative. Hollywood, FL: Herman Goldstein Award Submission.

Kochel, T. R., Burruss, G., & Weisburd, D. (2015). St. Louis County Hot Spots in Residential Areas (SCHIRA) final report: Assessing the effects of hot spots policing strategies on police legitimacy, crime, and collective efficacy. Carbondale, IL: Southern Illinois University

◦

See also Kochel and Weisburd (2017, 2019).

Dario, L. M. (2016). Crime at convenience stores: Assessing an in-depth problem-oriented policing initiative (Ph.D. dissertation). Arizona State University.

◦

See also White and Katz (2013) and Glendale Police Department (2016).

Durham Constabulary. (2017). Reducing dwelling burglaries in areas which repeatedly suffer high rates in county Durham, UK. County Durham, UK: Herman Goldstein Award Submission.

Zidar, M. S., Shafer, J. G., & Eck, J. E. (2017). Reframing an obvious police problem: Discovery, analysis and response to a manufactured problem in a small city. Policing: A Journal of Policy and Practice, 12(3), 316–331.

Gill, C., Weisburd, D., Vitter, Z., Shader, C. G., Nelson-Zagar, T., & Spain, L. (2018). Collaborative problem-solving at youth crime hot posts: A pilot study. Policing: An International Journal, 41(3), 325–338.

Cooley, W., Bemiller, M., Jefferis, E., & Penix, R. (2019). Neighborhood by neighborhood: Community policing in a rust belt city. Policing: An International Journal, 42(2), 226–239.

This 2019 publication is included as the “online first” version came out in 2018 and was included in our search results.

Eligible studies which lacked effect size data (N = 15):

Hampshire Constabulary. (2006). Operation Mullion: Reducing anti-social behaviour and crime in and around Mayfield School. Portsmouth, Hampshire, UK: Herman Goldstein Award Submission.

South Yorkshire Police. (2006). Focusing on car crime: An initiative by South Yorkshire Police to tackle the problem of offenders stealing from Ford Focus cars. Barnsley, South Yorkshire, UK: Tilley Award Submission.

Charlotte-Mecklenburg Police Department. (2007). Operation safe storage. Charlotte, NC: Herman Goldstein Award Submission.

Regina Police Services. (2007). Regina auto theft strategy. Saskatchewan, Canada: Herman Goldstein Award Submission.

Northhamptonshire Police. (2008). Northampton Countywide Traveler Unit. Northhamptonshire, UK: Tilley Award Submission.

Sussex Police. (2008). Operation athlete. Sussex, UK: Tilley Award Submission.

Anaheim Police Department. (2009). Anaheim Police Department's GRIP on gangs: Gang reduction and intervention partnership: An early gang prevention problem solving strategy. Anaheim, CA: Herman Goldstein Award Submission.

Warwickshire Police. (2009). Trolley safe: A design based problem solving response to reduce purse thefts from shoppers in supermarkets. Warwickshire, UK: Herman Goldstein Award Submission.

Dayton Police Department. (2011). The urban high school disorder reduction project: Restoring safe schools and inspiring academic excellence. Dayton, OH: Herman Goldstein Award Submission.

State College Police Department. (2011). Reducing crime and disorder in rental properties: An evaluation of the state college nuisance property ordinance. State College, PA: Herman Goldstein Award Submission.

Boston Police Department. (2012). Safe street teams problem-oriented policing initiative. Boston, MA: Herman Goldstein Award Submission.

Palm Beach County Sheriff's Office. (2012). Smart Policing Initiative: Increasing police legitimacy and reducing victimization against immigrants in Lake Worth. Lake Worth, FL: Herman Goldstein Award Submission.

Wolfe, S. E., Rojek, J., Kaminski, R., & Nix, J. (2015). City of Columbia (SC) Police Department Smart Policing Initiative: Final Report. Retrieved from http://strategiesforpolicinginnovation.com/sites/default/files/2015_Wolfe%20et%20al_SPI_Final%20Report_Submission%20to%20CNA%20and%20BJA.pdf

Portland Police Bureau. (2018). Zombie houses: The Portland approach to vacant homes. Portland, OR: Herman Goldstein Award Submission.

Carson, J. V., & Wellman, A. P. (2018). Problem-oriented policing in suburban low-income housing: A quasi-experiment. Police Quarterly, 21 (2), 139–170.

Several studies that received full-text review after the initial abstract screening were excluded after determining that they did not meet our inclusion criteria. These studies are noted in Appendix F.

Characteristics of selected studies

Table 1 provides an overview of the 34 studies that are included in our meta-analyses. In terms of location, 28 studies (82.4%) were conducted in the United States, 5 (14.7%) in the United Kingdom and 1 (2.9%) in Canada. Studies were conducted in a total of 23 U.S. cities and 2 counties across 17 states. Lowell, MA, Jersey City, NJ and Philadelphia, PA all served as the jurisdiction for two studies. The U.K. studies included a total of eight jurisdictions, with Lancashire Constabulary serving as a study site in three evaluations. Vancouver was the setting of the Canadian study.

The study documents we identified were predominantly peer-reviewed journal articles (N = 13, 38.2%) and submissions to the Goldstein Award (N = 13, 38.2%) for excellence in POP (no Tilley Award submissions met our inclusion criteria). There were also 4 (11.8%) research reports, 2 (5.9%) doctoral dissertations, and 2 (5.9%) master's theses. A few scholars served as an author or coauthor on multiple included studies, including David Weisburd (four studies and a coauthor of this review), Lorraine Mazerolle (three studies), Anthony Braga (two studies), and Brenda Bond (two studies).

In terms of rigor of research design, our sample of studies includes 9 (26.5%) randomized experiments and 25 (73.5%) quasi-experiments. This is an increase from four randomized experiments and six quasi-experiments in the original review and suggests a trend toward more rigorous evaluation of POP since 2006.

Turning to the unit of analysis for the POP interventions in these studies, 26 (76.5%) programs targeted problem places, 4 (11.8%) were targeted at place managers, and 4 (11.8%) targeted individuals. Three of the individual-focused programs targeted problem offenders, while one intervention was aimed at potential victims. Eight studies of place-based POP approaches quantitatively assessed displacement and diffusion effects, and we conduct a meta-analysis on these effects.

Table 2a provides a quick overview of the type of problems targeted and the type of responses delivered, while Table 2b provides a detailed summary of the studies based on type/depth of scanning for problems and problem analyses used, the responses delivered and the research design. The latter columns in the table also highlight implementation problems and research design limitations where applicable. These are discussed in detail in the following two sections.

OPEN IN VIEWER

Table 2b

SARA model characteristics of problem-oriented policing experiments and quasi-experiments

Note: The descriptions provided are summaries and are not intended to cover every aspect of the intervention.

Goldstein Award Submission.

Report used for intervention description but not included in meta-analysis.

Finally, 15 of 34 (44.1%) studies reported significant reductions in at least one crime or disorder outcome, while another 17 (50%) studies reported raw differences favoring the treatment group for at least one crime or disorder outcome. Table 3 provides a summary of study conclusions about impacts on crime and disorder, as well as displacement and diffusion of crime control benefits where applicable.

Study implementation

While there was a relative lack of substantial complications reported, 67.6% (n = 23) of the 34 included studies did identify some degree of difficulty during implementation. Based on our coding criteria, the severity of these issues was classified as either minor (n = 15), more substantial (n = 7), or major (n = 1). There also appeared to be thematic consistencies across the studies in the types of issues reported. While such issues are in no way limited to POP interventions, this is perhaps indication that POP interventions are at greater risk of certain complications. Brief summaries of each study's implementation issues are also presented in Table 2b.

Given that POP is an iterative process, such that problems and responses are often continually changing, implementation issues may arise from problem instability. In the Philadelphia Policing Tactics experiment, Groff et al. (2015) note that nearly half of the POP intervention hot spots began focusing on nonviolent crime problems after determining that violent crime was no longer the primary concern in the area. Specifically, eight POP sites were noted to have targeted nonviolent or quality of life offenses, and at least four POP sites were noted to have focused on drug crime in addition to a violent or nonviolent crime problem. This is an issue from an evaluation standpoint as violent crime incidents were the only measured outcomes; thus it is possible that the interventions were effective in ameliorating the problems that they targeted, but this was unclear absent measurement of those outcomes. The RECAP experiment (Sherman et al., 1989) also suffered from issues related to problem instability. Specifically, call trends for many high-crime addresses were remarkably heterogeneous from year to year, subsequently reducing the experiment's statistical power. The RECAP experiment and the Philadelphia Policing Tactics experiment both suffered from additional resource constraints as well. In the Philadelphia Policing Tactics experiment, POP officers were not dedicated to the intervention full-time, but were instead drawn from patrol and expected to conduct POP activities during their free time. Similarly, in the RECAP experiment, there were likely too many addresses assigned to the experimental unit, thus spreading the unit's resources too thin and, perhaps, contributing to the lack of effectiveness in the second half of the intervention year.

The reality of resource constraints and other internal barriers to proper program implementation was not uncommon across these studies. Stokes (1996) reported that the safe travel corridor was poorly staffed during the afternoon hours, despite violence being more prevalent during this time. It was subsequently determined that this incongruence was due to officer shift changes and high numbers of outside calls during the afternoon hours. These factors created a gap in coverage and limited police resources toward the intervention (though the authors note that officer presence was adequate due to Temple University Police presence). It was also revealed that very few students were aware of the safety corridor, though it is not clear whether this was resource related, as school administration reported that the corridor was advertised over school announcements and letters that were distributed to students and parents. Stone (1993) also reported organizational and resource-related constraints during the Atlanta public housing POP project. There was a relative lack of interest regarding the intervention within the department, little administrative support, and police training was minimal. These issues were compounded by the fact that the city of Atlanta had hosted the Democratic National Convention prior to the intervention, which forced officers to delay vacation during this time. Thus, when the POP project started, many officers opted to take time off and the project was chronically understaffed. In their evaluation of the Lowell (MA) Smart Policing Initiative (SPI, now referred to as Strategies for Police Innovation), Bond and Hajjar (2013) also noted that a common complaint from police captains was a lack of resources. It was suggested that such constraints prevented an increased level of proactivity during the intervention, though these issues appeared to be minor as the intervention was still considered to be effective. Minor issues with internal resistance were also noted by the Knoxville Police Department (2002) during their Public Safety Collaborative.

Police subversion concerns created barriers for both the Jersey City violent places POP study (Braga, 1997; Braga et al., 1999) and the Drug Market Analysis Program (Weisburd & Green, 1995). Partly as a result of officer resistance, the Drug Market Analysis Program achieved limited implementation in the first 9 months, with only nine hot spots receiving all program elements. This forced Weisburd and Green (1995) to increase the length of the intervention and develop a more detailed implementation schedule, and ultimately the program was fully implemented for the last 5 months of the intervention period. Braga (1997) noted similar resistance among officers in the violent places POP project, as well as a disconnect between middle management and department headquarters that threatened the integrity of the program and slowed progress during the first 8 months. Ultimately the intervention unit was placed under new leadership and protocols were established to document instances of subversion. Braga also noted significant organizational changes, such as an influx of retirement and scheduled vacations which strained resources and reduced the sample size of the experiment.

Another frequent implementation barrier was resistance from stakeholders that were intended to be involved in the intervention. In the Glendale SPI study, White and Katz (2013) indicated that in phase I of the response, the SPI team was largely unsuccessful in working with Circle K management to change the physical structure and operating policies of the stores. They note that, at this stage, communication between Circe K representatives and the SPI team suffered, and the intervention was forced in a different direction. Partly as a result of this, Dario (2016) expressed concern over treatment dosage, noting that treatment quality likely varied by store location due, in part, to differing levels of responsiveness. In their attempted intervention with Walmart management, Zidar et al. (2017) also reported resistance toward environmental and policy-oriented intervention measures. This resistance dictated the future direction of the program, as the initial plan to partner with Walmart seemed futile. In response, Zidar et al. forced responsibility on Walmart leadership by forcing them to handle petit shoplifting incidents without police assistance; however, even after doing so they noted instances of Walmart loss-prevention misrepresenting case facts to illicit police response.

The San Angelo Police Department (2006) cited heavy opposition from business owners to the implementation of an identification checking program, largely over concern that the program would inconvenience customers. This resistance forced the department to shift responsibility for the intervention toward the customer, though despite this, few businesses ever became willing to implement the program. The Motel Interdiction Team (MIT) program in Reno experienced similar complications (Elliott, 2007; Reno Police Department, 2006). Motel owners were concerned about the economic ramifications that would result from the eviction of criminal tenants. The Reno Police Department (2006) noted that it became difficult to educate these owners about recognition of criminal behavior and eviction processes, ultimately slowing the intervention's progress. In response to uncooperative property owners, the Houston Police Department (2012) increased code enforcement in their targeted intervention of the Antoine corridor. However, this response temporarily led to a backlog in the court system, and prosecutors began dropping charges (though this issue was subsequently resolved by use of specialized prosecutors). Lancashire Constabulary (2008) noted a similar delay in court processing based on their enforcement responses.

Several studies reported resistance from other outside stakeholders such as neighborhood residents, community organizations, and local government. Cooley et al. (2019) described an attempted POP replication in Canton, OH. However, while the initial intervention was successful in establishing partnerships with community residents and neighborhood groups, the attempted replication was not. Cooley and colleagues noted that there was a lack of community organizations available to partner with and that local residents were distrusting of and unwilling to cooperate with police. Local community meetings were unsuccessful at bridging the gap between law enforcement and neighborhood residents, officer morale was low, and ultimately the intervention showed limited effectiveness. Issues were encountered forming resident partnerships in the St. Louis County Hot Spots in Residential Areas experiment as well (Kochel & Weisburd, 2017). Specifically, program evaluation revealed that resident partnerships were less frequently established than had been originally intended. While residents appeared to be cooperative in the early stages of the Brightwood Interdiction project, the same residents were subsequently caught tipping off offenders to police surveillance during the intervention (Nunn et al., 2006). This slowed the evidence gathering process, but police were ultimately able to generate enough evidence to execute the planned warrant sweep. Lastly, as a result of varying levels of difficulty partnering with the community, Tuffin et al. (2006) reported that only two of six intended sites receiving full implementation, though the sites that did achieve full implementation showed strong results.

Gill et al. (2018) noted several aspects of the collaborative problem-solving intervention in Seattle (WA) that were halted by local government resistance. In one intervention area, officers sought to implement a smoking ban, but were unable to do so largely due to a lack of political support. In another target area, officers were unable to implement environmental changes due to a city redevelopment plan that was operating on a different timeline. Gill et al. also noted that, despite the project's intention to be fully non-enforcement, officers in one of the intervention areas felt that enforcement measures were necessary to stabilize the area.

Studies that targeted residential burglary incidents reported minor issues with environmental changes to the target area. London Borough of Enfield (2011) referenced issues with the installation of alley gates. Installation required 100% approval from area residents; however, several properties were rentals with out of town owners. The police were subsequently able to adjust the approval rate from 100% to 98% to circumvent this issue. Durham Constabulary (2017) noted similar issues installing security measures in private housing areas; however, they were eventually able to gain permission to do so. Lancashire Constabulary (2008) also documented issues implementing situational responses that altered the physical environment, and the Hollywood Police Department (2015) determined that closing access to alleyways was not financially feasible (however, they promoted the use of see-through fencing instead).

Finally, there were some problems unique to certain studies. Sherman et al. (1989) encountered issues with hot spot selection, discovering that up to 15% of calls were “mirrors,” or duplicates created as a result of multiple 911 calls for the same incident. Additionally, several addresses that were originally believed to be independent were subsequently determined to correspond to the same building. This led to the inclusion of some buildings in both treatment and control groups, and required a series of pairwise deletions to modify the initial assignments. Bichler et al. (2013) expressed some concern over the possibility of unknown history effects, noting that at least one motel location was disqualified from their study after being the target of another policing intervention. Ultimately, however, there was no reported evidence to suggest that similar issues occurred at other intervention motels.

It is also worth noting that these studies may vary in their level of reporting validity. All interventions are likely to encounter obstacles; however, it is not necessarily the case that all such obstacles are accurately documented. At times there may be incentive to represent an intervention in the best possible light, perhaps at the expense of complete transparency. While this is certainly true of any form of research, it bears reminding that our determinations are limited to what was reported in study publications.

Risk of bias in included studies

Five main measures from our coding instrument (see Appendix E) were used to assess potential sources of bias in our included studies. These items included: (a) Were any sources of nonequivalence or bias reported or implied in the application of the intervention or its analysis (i.e., threats to internal validity)? (b) If yes, what sources of nonequivalence or bias were identified? (c) Did the researcher(s) express any concerns over the quality of the data? (d) If yes, explain. (e) If a quasi-experiment, how was matching of groups achieved? The studies that reported issues along these dimensions and/or compared treatment groups to the rest of a jurisdiction or population are presented in Table 4. The remainder of our included studies reported no such issues and/or employed higher quality matching procedures.

Overall, only 14.7% (n = 5) of studies reported internal validity concerns and 5.9% (n = 2) of studies reported concerns over data quality. However, the validity of the matching techniques used in our sample of studies does need to be considered, as 40.0% (10 of 25) of the identified quasi-experiments used the rest of a jurisdiction or population not receiving treatment as the comparison unit. Additionally, the remaining quasi-experimental evaluations exclusively matched comparison units based on descriptive and demographic characteristics, or simple statistical tests of such characteristics. None of the included quasi-experimental evaluations reported propensity or regression-based matching techniques.

Of the randomized experiments (n = 9) included in this review, very few reported concerns over randomization procedures or other issues related to internal validity and bias. All experimental studies included some form of blocking or pair-matching technique in addition to randomization. However, specific concerns were noted in a few of these experiments. In the RECAP experiment (Sherman et al., 1989, p. 17), there was possible contamination (or “spillover” effects) as treatment and control addresses were, at times, under shared ownership. Moreover, the instability in the call frequencies of particular addresses created additional variability between treatment and control groups. However, ultimately the groups were reported to be roughly equivalent. Despite the use of matched-pair techniques in the Collaborative Problem-Solving at Youth Crime Hot Spots study, the treatment and control locations were unable to be optimally matched, and Gill et al. (2018) noted some concern over the equivalence of these matched pairs.

The risks of bias in the quasi-experimental evaluations are undoubtedly greater. However, of the 15 studies that matched treatment to comparison locations based on simple analysis of descriptive, social, or demographic characteristics, no major concerns were reported. It should be noted, however, that few of these studies (n = 4) were identified as providing a visual comparison of descriptive statistics between treatment and comparison areas. The remaining studies (n = 11) described the rationale and/or process for selection of comparison units, but did not provide further evidence that equivalence was attained.

The most notable concerns among our included studies were related to the use of particularly nonequivalent comparison groups. There were six studies that compared small geographic treatment areas (or collections of small geographic treatment areas) to city, district, or other population-wide trends (Boston Police Department, 2008; Guseynov, 2010; Houston Police Department, 2012; Lexington Division of Police, 2009; London Borough of Enfield, 2011; White & Katz, 2013). Of note, four of these studies are Goldstein Award submissions, and while they do not report substantial concern over the comparability of the units, there are clear threats to internal validity caused by the use of such comparisons. There were another five studies that compared treatment units to the remainder of a population not receiving treatment, but where the size discrepancy between the groups was not as large (Baker & Wolfer, 2003; Elliott, 2007; San Angelo Police Department, 2006; Zidar et al., 2017).

In addition to the inherent threat to internal validity, several of the weaker quasi-experimental studies reported unique issues. Comparison of descriptive statistics for the treatment and control areas in the CSTAR intervention (Guseynov, 2010) indicated statistically significant differences on measures of race, unemployment, poverty, single parent households, and population mobility. Both Guseynov (2010) and Elliott (2007) also reported data-related concerns. Guseynov specifically notes that the Kansas City Police Department switched reporting systems during the beginning of the postintervention period. This switch resulted in the omission of 17 weeks of data, possibly leading to bias in the analysis. Zidar et al. (2017) suggested that the significant decline in reported shoplifting incidents indicated by their evaluation may have been the result of under-reporting rather than true changes in the outcome. The intervention involved the implementation of a new reporting system for target Walmarts only. Thus, they imply that it is likely the observed outcome was the result of this measurement change rather than changes in the incident rate. In the Glendale SPI (Dario, 2016; White & Katz, 2013), the intervention locations were selected based on high pre-existing crime rates. Dario (2016, p. 99) noted the inherent potential of regression to the mean when treatment units are selected in such a way, noting the possibility of bias in treatment selection.

Meta-analysis of the effects of POP on crime and disorder

Our first meta-analytic model presents the overall mean effects for 70 outcomes across the 34 included studies. As noted, above, many studies reported on multiple crime/disorder outcomes and for many the authors did not specify any one outcome as the primary target of their intervention. In our data, 13 of the 34 (38.2%) studies fit into this category. To avoid any “creaming” of results, we include all relevant outcomes from such studies. For studies with multiple outcomes, a mean effect size is used in this model; thus each study is only counted once in the analysis. This is the same approach used in our original review, as well as other recent Campbell reviews of policing strategies (e.g., Braga, Turchan, et al., 2019).

The results from the first model are presented in Figures 2a (Cohen's D model) and 2b (RIRR model). The forest plots show the standardized mean differences and log RIRRs, respectively, between the treatment and control groups, with the lines on either side representing the 95% confidence interval (CI). Effects to the right of 0 are supportive of reductions in crime/disorder, while effects to the left would suggest backfire effects where problems increased in the treatment areas relative to the controls. A random effects model was estimated based on an a priori assumption of a heterogeneous distribution of effect sizes (and the Q statistics for our models confirm this assumption).

OPEN IN VIEWER Figure 2 OPEN IN VIEWER

The overall mean effect size for Cohen's D approach is 0.183 (p < .001). The largest effects were found in the studies conducted by the London Borough Enfield (0.841), Thomas (0.771), the Knoxville Police Department (0.664) and the San Angelo Police Department (0.654)—all four were submissions for the Goldstein Award. Three studies reported negative overall effects, Stone (−0.001), Taylor et al. (−0.012), and Stokes et al. (−0.203).

The overall mean effect is considered a small effect by conventional standards developed by Cohen (1988). However, Lipsey (1990) describes effects in this range as small but meaningful impacts that could “easily be of practical significance” (Lipsey, 1990, p. 109). It is also important to note here that the studies clustered immediately around the mean effect size are randomized experimental evaluations of place-based versions of POP in which the authors reported notable reductions in a variety of crime and disorder outcomes. (e.g., Weisburd & Green, 1995; ES = 0.147, Sherman et al., 1989; ES = 0.192; Braga & Bond, 2008; ES = 0.206; Braga et al., 1999; ES = 0.233). In this regard, Wilson (in progress) has raised strong concerns in interpreting place-based effect sizes similarly to person-based effect sizes. While we follow standard practice here in reporting effect sizes, we think that caution should be used in interpretation of what magnitudes mean.

For example, the largest impact in the study by Nunn et al. (2006) was an effect size of 0.200 for robbery calls for service (see Figure 3). This is an effect at the criterion of 0.20 set by Cohen (1988) for a small effect. Looking at the raw changes in the data, we see that the relative actual reduction in the proportion of crime in the treatment condition was 30.4% percent, while the control area saw no change in robbery calls. Similarly, in Gill et al.'s (2018) study the largest impact was on calls for service in the retail treatment site. Our effect size for this outcome of 0.187 is a bit below the 0.20 threshold, yet the raw change shows a 10.3% decrease in calls compared with a 25.9% increase in the comparison areas. Our point is that standard small effect sizes may translate to very meaningful crime prevention outcomes at places.

OPEN IN VIEWER Figure 3 OPEN IN VIEWER

As noted above, David Wilson (in progress) has argued that Cohen's D fails to produce effect sizes that are comparable across studies when based on place-based count data and has also shown that the process of converting RIRRs to Cohen's D is problematic. As such, we also present meta-analyses for all of our models using the Log RIRR as the effect sizes. These analyses are of 33 of 34 included studies (and 69/70 outcomes). The study by Stone (1993) is not included in the RIRR models as the data and methods used did not allow us to calculate an RIRR. This is not a major concern as this study is a near zero effect in Cohen's D model (D = −0.001, p = .992). The overall RIRR model is shown in Figure 2b.

The results of the RIRR show an overall effect of 0.291. This can be interpreted as a relative present change by taking the exponent of the effect (which is a log RIRR) and then subtracting 1 from that value and multiplying by 100. For the overall model this shows that there was a 33.8% reduction in crime/disorder in the POP treatment areas/groups relative to the controls. Some caution is needed in interpreting this effect size as analyses presented below show smaller (though still positive and statistically significant) effects in randomized experiments and after accounting for publication bias. Nonetheless, this finding supports our illustration above of how Cohen's D often understates the magnitude of effects for place-based studies (the majority of our sample) and provides further evidence that Wilson (in progress) is correct that Cohen's D is not the most appropriate effect size for these types of studies. We continue to present Cohen's D models throughout to allow for comparison to our prior review.

Regarding the meaning of effect sizes, we think it important to also note here that scholars have argued that approaches like POP that alter the characteristics of high-crime places may reduce more crime in the long run than approaches such as temporarily increased police presence or crackdowns (Braga, Turchan, et al., 2019; Braga & Weisburd, 2010). This is the case as lasting changes to places made through identifying and solving problems may reduce crime over the long term through reducing opportunities for crime or other mechanisms (see above). Thus finding a relative reduction of 33.8% may be suggestive of even larger impacts on crime/disorder in targeted areas in the long run.

Moving beyond the overall effect size, perhaps the most striking finding is that the overall trend of mean effect sizes per study skews very heavily toward studies that produced findings in the direction of POP being effective. Specifically, 31 out of 34 studies (91.2%) in Cohen's D model have positive effect sizes, with 13 of them (38.2%) being statistically significant effects in favor of the treatment group. In the RIRR model, 30/33 studies (90.9%) reported positive impacts and 12 (36.4%) were statistically significant. Only one study (Stokes et al., 1996) had a statistically significant backfire effect, and that was a project that was plagued by implementation and research design limitations as reviewed in Section 5.3. Excluding this study from our overall model slightly increases the mean effect size from 0.183 to 0.195 in Cohen's D model and increased the relative reduction in crime from 33.8% to 36.6% in the RIRR model.

Before moving on, it is also important to recall that we had 15 eligible studies that are not included in any of our analyses as they lacked data needed to calculate effect sizes. Goldstein and Tilley Award submissions accounted for 13 of these 15 studies. As one would expect for programs submitted for award consideration, all 13 of these studies discuss findings favorable to POP's effectiveness. The other two studies (Carson & Wellman, 2018; Wolfe et al., 2015) failed to find evidence of positive impacts, but also did not note any backfire effects and pointed to challenges with fully implementing the interventions. As such, there is no reason to believe that the absence of these studies from our meta-analyses would alter any of our conclusions.

Given that the overall models present mean effect sizes for studies reporting on multiple outcomes without a clear primary outcome specified, we felt it important to also estimate models including only the largest and smallest effect sizes for each study. For studies with a single outcome, or a clearly stated primary outcome, their effect sizes remain the same in all models. This approach provides an upper and lower bounds for the overall standardized mean effect. Figure 3a,b presents the largest effects models and Figures 4a,b the smallest effects models.

OPEN IN VIEWER Figure OPEN IN VIEWER

Following the logic of inclusion, the overall random effect is larger when only including the outcome with the largest effect size for each study. While the mean effects Cohen's D model had an overall effect of 0.183, the largest effects model has an overall standardized mean effect of 0.271 (p < .001)—an increase of 0.088 (48.1%). Similarly, for the RIRR model the mean effects approach had an overall effect of 0.291 (33.7% relative reduction), the largest effects model had an overall effect of 0.357. This corresponds to a 42.9% relative reduction of crime/disorder in the treatment group.

Turning to the smallest effects, we see a lower bound for the standardized mean effect of 0.135 (p < .001)—a decrease of 0.048 (−26.2%) from the overall mean effect model. For the RIRR model we see a lower bound of the overall effect of 0.223, corresponding to a 25.0% relative reduction in crime/disorder. We think that this approach gives a sense of the range of effects that can be expected in POP programs. Specifically, using the RIRR results, the overall effect ranges from a 25% relative reduction when using the smallest effects to 42.9% relative reduction when using the largest effects. Importantly, our overall conclusion about the effectiveness of POP remains consistent. Regardless of the type of effect size and whether we examine the overall mean effect or look at the largest or smallest effect size, the results suggest that POP has a significant meaningful effect (Lipsey, 1990) in reducing crime/disorder.

Finally, we felt it important to examine variation in effect sizes across crime types. Table 5 summarizes the mean effects sizes for violent crime, property crime and disorder offenses for both our Cohen's D model and the RIRR approach. Studies that reported on aggregated crime counts that included more than one of these categories are not included in these analyses, nor are other types of outcomes that did not fit into those groupings (and lacked enough cases to perform a meaningful meta-analysis).

The results show that while POP had significant impacts on property crime (31.0% relative reduction) and disorder offenses (18.9% relative reduction), the overall effect for violent crime did not reach statistical significance (p = .156 in the RIRR model, p = .218 in Cohen's D model). However, the effect is still in the positive direction (9.5% relative reduction) and 13 of the 18 violent crime outcomes were positive. Future research should further explore the potential reasons for the heterogeneous impact of POP across crime types.

Moderator analyses

We also conducted moderator analyses to examine heterogeneity in effect sizes across three dimensions—(a) experiments versus quasi-experiments, (b) studies with nonequivalent groups versus all others, and, (c) the type of publication (scholarly publications vs. Goldstein Award submissions).

Study design is important to assess as it is well known that more rigorous designs are more likely to produce null findings (Rossi, 1987). Figure 5a,b shows the moderator results for randomized experiments versus quasi-experiments.

OPEN IN VIEWER Figure OPEN IN VIEWER

The moderator analysis of the impact of study design shows that quasi-experiments have a larger overall effect size than randomized experiments. For Cohen's D model (see Figure 5a) the quasi-experimental studies have an effect size of 0.212 (p < .001), while the randomized experiments have an effect size of 0.107 (p < .001). The difference between groups was statistically significant (Q = 4.914, df = 1, p = .027) and the moderated effect size is 0.147 (p < .001). Turning to the RIRR models (see Figure 5b) we see the same pattern. For quasi-experiments, the effect of 0.377 (a relative reduction of 45.8%) was larger than the effect of 0.089 (a relative reduction of 9.3%) for randomized experiments. The difference between groups was again statistically significant (Q = 14.171, df = 1, p < .001) and the moderated effect size was 0.183—a relative reduction of 20.1%.

These results are consistent with Weisburd, Lum, and Yang's (2003) proposal that experimental designs more generally show smaller impacts in crime and justice research (see also, Welsh, 2016). The results show that while there is a bias toward finding stronger effects in studies with weaker research designs, the overall finding of a significant meaningful effect for POP is supported across study types, as well as by the moderated effect size.

Our second methodological moderator analysis examined the impact of the nonequivalent research designs highlighted in Table 4. Specifically, we compared the studies that are listed in Table 4 as having nonequivalent control groups to studies with better matching methods. The results here suggest that studies with nonequivalent control groups did not bias our conclusions. Indeed, effect sizes were actually slightly larger in the studies with better matching methods. For Cohen's D models, the 11 studies with nonequivalent control groups had an overall effect size of 0.178 (p < .001), while the effect for the 23 studies with more rigorous matching approaches was 0.190 (p < .001). The difference between groups was not statistically significant (Q = 0.034, df = 1, p = 0.854) and the moderated effect size was 0.184 (p < .001). Similarly, for the RIRR models, the nonequivalent control groups had an overall effect size of 0.263 (p < .001; a 30.1% relative reduction), while the studies which used better matching methods had an effect of 0.309 (p < .001; a relative reduction of 36.2%). The between-groups difference was again not statistically significant (Q = 0.231, df = 1, p = .631) and the moderated effect size was 0.289 (p < .001; a 33.5% relative reduction).

Next, we examined the impact of the type of publication. This is important as the award submissions are inherently biased toward successful outcomes and likely also toward larger effects. The reasoning here is simple—police departments are not going to submit a program that did not work for consideration for an award and are probably most likely to submit when a project has a larger impact. As such, our moderator analysis here compares the mean effects for the award submissions to those for scholarly publications (journal articles, research reports, theses and dissertations in our current sample). These results are shown in Figure 6a,b.

OPEN IN VIEWER Figure OPEN IN VIEWER

As expected, the award submissions have a larger overall mean effect size than scholarly publications. Examining Cohen's D model (Figure 6a) we see that the award submissions had an overall effect of 0.362 (p < .001) while the effect for scholarly publications was 0.101 (p = .001). The difference between groups was statistically significant (Q = 13.702, df = 1, p < .001) and the moderated effect size is 0.149 (p < .001). The results from the RIRR model (Figure 6b) are very similar, with the effect for award submissions being 0.580 (a 78.6% relative reduction) compared with an effect of .166 (an 18.1% relative reduction) for the scholarly publications. The between-groups difference was again statistically significant (Q = 12.329, df = 1, p < .001) and the moderated effect size was 0.228 (a 25.6% relative reduction).

These results raise possible concerns regarding the exclusion of police-initiated POP programs that were evaluated in some way but were not submitted for award nominations. We conduct analyses regarding publication bias below, and note the biases there. Nonetheless, because our analyses without the award submissions remain statistically significant, this finding does not alter our overall conclusion of a significant crime prevention outcome for POP. In turn, these award studies do provide additional information about successful interventions and a sense of the upper range of POP impacts that are to be expected.

As a final summary and sensitivity analysis, Tables 6a and 6b below summarize the main effect sizes outlined above, and also present results from models using the smallest and largest effects in each applicable study for randomized experiments, quasi-experiments, award submissions and scholarly publications that were not presented above to save space. The range of effects support our conclusion of a meaningful effect of POP in reducing crime in all but two of the models. The overall effect is not significant in the two most conservative models—smallest effect outcomes for randomized experiments and scholarly publications—for both Cohen's D and RIRR models. All the other models produced statistically significant overall effects. For Cohen's D models (see Table 6a), those effects ranged from 0.101 (mean effect for randomized experiments) to 0.415 (largest effect outcomes for award submissions). The mean effects shown in the “Combined ES” column provide the overall summary of effects across all outcomes for each model and show that across all models the average overall effect of POP on crime/disorder ranges from 0.101 to 0.362. Similarly, looking at the RIRR models summarized in Table 6b (again excluding the smallest effects for the randomized experiments and scholarly publications which are not significant) shows that crime in the POP group relative to the controls was reduced between 9.3% (mean effect for randomized experiments) and 81.5% (largest effect for award submissions). The mean effect ranged from the 9.3% relative reduction in randomized experiments to 78.6% for award submissions.

OPEN IN VIEWER

Table 6a

Summary of effect sizes ranges across models (Cohen's D)

Model	Smallest ES	Combined ES	Largest ES
All studies	0.135***	0.183***	0.271***
Randomized experiments	0.005	0.107***	0.229***
Quasi-experiments	0.189***	0.212***	0.276***
Scholarly publications	0.039	0.101***	0.167***
Award submissions	0.350***	0.362***	0.415***

p ≤ .001.

OPEN IN VIEWER

Table 6b

Summary of effect sizes ranges across models (Log RIRR)

Model	Smallest ES	Combined ES	Largest ES
All studies	0.223 (25.0%)***	0.291 (33.8%)***	0.357 (42.9%)***
Randomized experiments	−0.003 (−0.3%)	0.089 (9.3%)*	0.241 (27.3%)***
Quasi-experiments	0.337 (40.1%)***	0.377 (45.8%)***	0.419 (52.0%)***
Scholarly publications	0.080 (8.3%)	0.166 (18.1%)***	0.256 (29.2%)***
Award submissions	0.566 (76.1%)***	0.580 (78.6%)***	0.596 (81.5%)***

p ≤ .05.

p ≤ .001.

As such, our review provides strong and consistent evidence that POP is an effective approach to reducing crime and disorder. However, there is a great deal of heterogeneity in the magnitude of effect sizes across factors such as study type, study rigor, and crime type. This will be discussed more in the discussion and conclusion sections of this report.

Meta-analysis of displacement and diffusion effects

Many of our studies were place-based approaches to POP that may have had potential to either displace crime/disorder to surrounding areas or to see the benefits of the intervention diffuse to areas that were not targeted (see Weisburd et al., 2006). Eight of our studies provided data for a total of 19 outcomes that allowed us to create effect sizes and conduct a meta-analysis of displacement and diffusion effects.

These effect sizes were calculated using pre- and postintervention counts for target, control, and buffer areas. This was done following the approach used by Telep, Weisburd, Gill, Vitter, and Teichman (2014) in their meta-analysis of displacement and diffusion effects of interventions in large-scale geographic areas (see also Bowers, Johnson, Guerette, Summers, & Poynton, 2011; Braga, Turchan, et al., 2019). Effect sizes were calculated using the relative incidence risk ratio approach described above. Pre- and postintervention counts from the treatment buffer area(s) are compared with pre- and postintervention counts either from a control buffer area or to the control area itself in studies that did not have a catchment area for the control site. Figure 7 shows the mean overall effect for displacement/diffusion. Effects to the right of zero indicate evidence of diffusion of crime control benefits, while effects to the left suggest displacement effects.

OPEN IN VIEWER

Figure 7

Displacement and diffusion-combined effect size for study outcomes (a) Cohen's D (random effects model: Q = 30.069, df = 7, p < .001, I ² = 76.720) and (b) Log RIRR (random effects model: Q = 30.547, df = 7, p < .001, I ² = 77.085). CI, confidence interval

Looking at Cohen's D model (see Figure 7a), the overall model provides no evidence of displacement. Seven of the eight studies have positive effect sizes, while the sole negative effect for Taylor et al. (2011) was very small and not statistically significant (−0.050, p = .765). Moreover, the overall random effect for the model of 0.089 (p = .023) is suggestive of diffusion of benefits across these eight studies, though caution is needed given the small effect size and limited number of studies. We also estimated displacement/diffusion models using the largest and smallest effects to provide a range for our overall estimate. This showed that when using the smallest effects, there is no evidence of either displacement or diffusion (0.029, p = .345), while when only including largest effects the evidence in favor of diffusion of benefits is moderately stronger than in the mean effects model (0.154, p = .006). The results from the RIRR model lead to the same conclusions. The overall model (see Figure 7b) shows a relative reduction of 17.5% (p = .005) in favor of diffusion effects and the largest effects model shows a larger relative decline of 34.6% (p = .004). The smallest effects model shows no statistical evidence of displacement or diffusion, with a nonsignificant relative difference of 4.6% (p = .327) in favor of diffusion.

Publication bias

Publication bias presents a strong challenge to any review of evaluation studies (Rothstein, 2008). Campbell reviews, such as ours, take a number of steps to reduce publication bias, as represented by the fact that 21 of the 34 (61.8%) included studies in our main analyses came from unpublished sources (13 Goldstein Award Submissions, 4 research reports, 2 doctoral dissertations, and 2 Master's theses). Wilson has argued that there is often little difference in methodological quality between published and unpublished studies, suggesting the importance of searching the “gray literature” (Wilson, 2009). For our review, there may also be a bias in unpublished studies that are nevertheless available for review, since 13 studies were identified through the Goldstein Award competition. As noted earlier, award submissions are inherently biased toward successful programs. This was evidenced by our moderator analyses, which showed effect sizes were significantly larger for award submissions versus the other publication types.

Here we focus on an overall comparison of the 13 studies published in scholarly journals versus the other 21 studies (20 for the RIRR approach) through use of moderator analysis. For Cohen's D model, the mean overall effect size for studies published in scholarly journals is 0.156 (p = .002; Q = 46.482, df = 12, p < .001) and for unpublished studies the average effect is 0.199 (p < .001; Q = 111.367, df = 20, p < .001). Moreover, the moderated effect size is 0.184 (p < .001) and the between models heterogeneity test was not statistically significant (Q = 0.468, df = 1, p = .494). Similar findings are seen in the RIRR model. The mean effect for the scholarly journal studies shows a relative decline of 29.7% (p = .005; Q = 63.518, df = 12, p < .001) while the unpublished studies show a relative decline of 35.1% (p < .001; Q = 141.113, df = 19, p < .001). The moderated effect size shows a relative decline of 33.8% and the between models differences test was again not statistically significant (Q = 0.145, df = 1, p = .703). The lack of significance for the between-model tests, along with the similarity between the mean overall effect sizes, suggests that publication bias may not have major impact on the outcomes of this review.

To more formally assess publication bias we generated a funnel plot to examine for possible selection bias in our results. This is shown in Figure 8a,b. A visual inspection indicates some asymmetry with more studies with a large effect and a large standard error to the right of the mean than the left of the mean. We used the trim-and-fill procedure developed by Duval and Tweedie (2000) to examine how our estimates would change in the absence of this asymmetry. The trim-and-fill procedure determined that nine studies should be added to create symmetry.

OPEN IN VIEWER Figure OPEN IN VIEWER

For Cohen's D model, these additional nine imputed studies slightly reduced the mean effect size from 0.183 (95% CI = 0.0124–0.241) to 0.106 (95% CI = 0.043–0.170). For the RIRR approach, 11 studies were imputed and this reduced the overall effect from 0.291 (95% CI = 0.202–0.379) to 0.132 (95% CI = 0.040–0.223). Put into relative risk reduction terms, the imputed studies decreased the overall relative reduction from 33.8% to 14.1%.

Along with the moderator analysis above, this suggests that while there is some potential for publication bias in our sample, especially given the nature of including police award submissions, it does not alter our overall conclusion of POP having an overall significant, meaningful impact on crime and disorder.

Summary of main results

The results of this updated review provide strong evidence that POP is effective in reducing crime. Across a large array of analyses, we find statistically significant impacts of POP. Overall we find that there was a 33.8% reduction in crime/disorder in the POP treatment areas/groups relative to the controls. At the same time, the effect sizes we observe are strongly heterogeneous and the overall effect is likely overstated as smaller effects were found when looking only at the randomized experimental evaluations and after accounting for publication bias. Nonetheless, the findings of those models still show that POP is associated with meaningful and statistically significant declines in crime/disorder in treatment groups relative to controls. Such heterogeneity across models is very common in meta-analyses in criminology (Lösel, 2018), but points to the importance of going beyond what works in POP to what are the most effective strategies for specific problems.

Overall, our findings show that police following the tenets of the SARA model to identify specific problems, conduct analyses to examine underlying causes, and develop and deliver tailor-made responses is an evidence-based approach to crime prevention. This is especially true for property crime and disorder offenses based on our results. POP is also an approach that fits well with hot spots policing, another tactic that has been found effective in reducing crime through Campbell reviews (Braga, Turchan, et al., 2019). Some of our studies overlap with those in that review as they involve applying problem-solving tactics at small hot spots of crime/disorder, and the authors of the hot spots review note that their strongest effects were associated with programs that involved problem-solving efforts rather than just increased police presence/activity. The positive findings of both reviews suggest that combining the two approaches is likely a fruitful endeavor for crime-control efforts.

As a number studies involved place-based POP programs, it was important to also examine the potential for spatial displacement and diffusion of crime control benefits. Eight of our 26 place-based studies reported data that allowed us to calculate effect sizes for displacement/diffusion. There is no evidence of spatial displacement of crime/disorder in these studies. Indeed, there is evidence of a small diffusion effect when looking at the mean effect across outcomes. This finding, along with a similar finding of a small diffusion effect in the hot spots policing review (Braga, Turchan, et al., 2019), suggests that place-based policing efforts do not simply cause crime to “move around the corner” (see Weisburd et al., 2006).

There was some evidence that research design moderated the magnitude of the impact of POP on crime/disorder. The effect size for quasi-experiments was somewhat larger than that for randomized experiments. However, the mean overall effect is significant for both models (as well as the moderated effect size). The same was true when examining the award submissions, which are inherently biased toward success and larger effects, as those had larger effects than the other types of studies. Nonetheless the overall effect for the non-award submission studies was statistically significant. Adding to this the fact that nearly all of the studies were weighted to the prevention side of the distribution across analyses, we have additional confidence in our overall conclusion.

Finally, as noted above, if underlying causes at problem places are successfully addressed, the crime-reduction benefits at targeted locations may be longer lasting and more meaningful in terms of overall crimes prevented compared with a similar effect size for a temporary police crackdown on an area or an intervention delivered to individual offenders. In plain language, there may be more “bang for our buck” when lasting changes are made at places, versus crackdowns that see deterrent effects erode when the program ends or person-based programs that have to be continually delivered to different individuals over time. Unfortunately, existing studies do not examine crime prevention benefits in the long run, and are generally limited to follow-up within a year or less (see also Weisburd & Majmundar, 2018 for a similar critique of short-term follow-up periods for POP studies). Future evaluations should include longer follow-up periods so that later updates to this review can quantitatively assess the potential lasting impacts of POP.

Overall completeness and applicability of evidence

The findings of this review have widespread applicability to policing and crime prevention given the consistency of the conclusions drawn across all of our models. This review also represents a large increase in the available body of evidence in the time since the original review, which included only 10 studies (4 randomized experiments and 6 quasi-experiments) and 16 outcomes. The current review includes an additional 24 studies (5 randomized experiments and 19 quasi-experiments) published through December 2018. Our overall model now provides a meta-analysis on the impact of POP on a total of 70 crime and disorder outcomes across 34 studies. The inclusion of these additional studies reaffirms the conclusion of the original review about POP's effectiveness in reducing crime and disorder with support from a much larger number of tests. While most studies (82.4%) were conducted in the United States, the fact that 5 were conducted in the United Kingdom and 1 in Canada offer initial support that POP is applicable in different contexts. However, this is still limited and caution may be needed when trying to generalize these findings to contexts outside of North America and the United Kingdom. We also were unable to perform a meaningful meta-analysis on noncrime outcomes. With the current data we could only provide a preliminary, narrative summary of study findings due to the small number of studies that report on the same noncrime outcomes captured through similar measures. The same was true for cost-benefit analysis.

Quality of evidence

The overall quality of evidence has improved from the original review with the inclusion of 5 new randomized experiments and 19 quasi-experiments. However, as we discussed above and assessed with our moderator analyses, POP remains an area that needs more rigorous evaluations. The majority of studies are still quasi-experiments, and several are weaker designs with unmatched control groups, comparisons of a target area to the rest of a jurisdiction and so forth. While we have confidence in our conclusions as the main finding of a significant effect for POP holds when looking only at the most rigorous studies, caution is needed in individually interpreting the larger effects from the weaker studies—especially the award submissions which are inherently biased toward positive outcomes.

We note that while more randomized experiments, and quasi-experiments with matched control groups/areas, would improve the quality of evidence on POP, the existing body of evidence is largely a function of the nature of the POP model. The approach calls for identifying specific problems to be researched and addressed and it is often not going to be possible to create a well-matched comparison area in a study jurisdiction (Eck, 2006a, 2006b). Similarly, the POP model calls on police to identify, analyze and respond to problems, and to then assess whether their efforts are successful. In this sense, the award submissions are evidence of the SARA model in action and are important to include in reviews such as this.

Moreover, the fact that 11 additional award submissions were eligible for this updated review (only two were included in the meta-analysis in the original review) is indicative of an increase in rigor as more agencies are using control groups, even if unmatched, instead of simple pre–post case study designs. As such, these studies are important evidence and simply require caution in interpreting individual effect sizes and acknowledgment that there is a “file drawer” problem here as agencies are not going to submit (or even write up research reports) for programs that failed. On that front, we retain confidence in our findings as our analyses above suggest that publication bias was not a major concern in our study.

Limitations and potential biases in the review process

In general, there are no specific limitations or biases in the review process used in this study beyond those inherent to the systematic review process. Namely, all reviews will have a “file drawer” problem to some extent, and the threat may be slightly higher for POP than other approaches due to the assessment step of the SARA model asking police to test whether their efforts were effective. As discussed above, these findings are unlikely to be written up (much less submitted for award consideration) when efforts fail. Moreover, the use of the Global Policing Database is a huge asset to the current review. The traditional and gray literature sources searched to compile that database are far more exhaustive than those used in individual reviews (see Appendices B and C), including the original version of this review. Lastly, there were 15 (13 of which were award submissions) potentially eligible studies of POP that we could not include as we could not calculate standardized effect sizes due to insufficient or inadequate information being presented (see Appendix F). As noted above, we do not feel the absence of these studies biased our conclusions as the 13 award submissions discussed positive impacts of POP and the other two studies reported null effects, but no backfire effects.

Agreements and disagreements with other studies or reviews

The findings of this review reaffirm those of the earlier review (Weisburd et al., 2008, 2010) and existing narrative reviews that conclude that current evidence suggests that POP is one of the most promising police approaches to preventing crime (Skogan & Frydl, 2004; Weisburd and Majmundar, 2018). Moreover, the limited evidence on displacement and diffusion confirms the findings of other studies of place-based crime prevention efforts by showing evidence of diffusion of benefits (Bowers et al., 2011; Braga, Turchan, et al., 2019; Weisburd & Majmundar, 2018; Weisburd et al., 2006). This finding is contrary to arguments made by others that displacement is the likely outcome of place-based interventions (Blattman, Green, Ortega, & Tobón, 2017; Reppetto, 1976).

Implications for practice and policy

Evidence from this review suggests that POP is an effective crime prevention approach. While there is a great deal of heterogeneity in effect sizes across studies and outcomes, 31 out of 34 studies (91.2%) have effect sizes in favor of a treatment effect. Moreover, the overall effect is positive and significant in all of our mean effect size models. In short, the findings suggest that POP is a promising approach to reducing a variety of types of crime and disorder in a variety of contexts (the 34 included studies included 29 unique jurisdictions in 3 countries).

Our findings suggest that it is important for police departments to be fully behind POP efforts if they are to succeed. For instance, the lone backfire effect in the study (Stokes et al., 1996) involved a program that was barely implemented as two-thirds of students at the target school were unaware of the existence of the school safety corridor and the corridor was poorly staffed in the after school hours due to the timing of police shift changes and limited police resources. Similarly, a null effect was reported in the study by Stone (1993), who noted that the police department did not seem entirely interested in implementing POP and that study officers did not view problem solving as “real” police work. This and other factors led to the program being chronically understaffed.

However, our results also highlight the fact that POP efforts can be successful even if the SARA approach is not delivered in its ideal version. This is important as studies have noted that it is difficult to fully implement the ideal model (Braga, Turchan, et al., 2019; Weisburd & Braga, 2006; Cordner & Biebel, 2005; Eck, 2006b; Maguire et al., 2015). Our findings show that even though the SARA model is often loosely followed, with the problem analysis often being shallow rather than in-depth (see Table 2b), the approach is still found to lead to crime reductions when compared with control areas that received standard police services. This adds support to arguments that even “shallow” problem-solving efforts can be lead to significant reductions in crime (Braga & Weisburd, 2010).

POP can also be fruitfully combined with other police tactics that have been found effective in recent Campbell reviews. In particularly, the POP approach has been shown to be effective when combined with hot spots policing. Braga, Weisburd, et al. (2019) found larger effect sizes for POP approaches at hot spots than approaches which simply increased police presence/activity at target areas. Elements of POP often also underlie the focused deterrence approach, which has been found effective and could perhaps be enhanced with more in-depth problem solving in future programs rather than largely replicating the existing “pulling levers” model (Braga, Weisburd, et al., 2019).

There is also potential for combining POP with the popular Compstat model. Indeed, “innovative problem solving” is one of the key elements in the ideal form of Compstat. While evaluations of Compstat in the United States suggest that the emphasis on holding commanders accountable through Compstat meetings has limited innovative problem solving in the field (Weisburd, Willis, Mastrofski, & Greenspan, 2019), a recent Israeli program suggests that innovative problem solving can be encouraged in a Compstat-like reform (see Weisburd et al., Unpublished manuscript). In that study, evidence-based policing practices were strongly encouraged in the context of a national POP reform program, and the message of the program was communicated more directly to the rank and file. Robust prevention benefits were identified in moderate and large size police agencies in quasi-experimental analyses of property crimes.

Implications for research

Our study identified 70 tests of POP in 34 included studies. Our meta-analyses suggest that overall there is a significant effect of the approach in reducing crime and disorder. Our moderator analyses showed that effects are larger for the quasi-experiments compared with the randomized experiments. Nonetheless, this does highlight the need for more rigorous evaluations of POP in order for a future update of this review to provide a more robust estimate of overall effect size.

This is not in any way meant to discourage quasi-experimental evaluations of POP, or even pre–post case studies. As discussed earlier, the nature of the POP model means there may sometimes only be a single area with the problem being treated, and even if there are multiple locations it will often be difficult, and sometimes impossible, to identify suitable control areas—much less to identify enough sites to create a randomized experiment with sufficient statistical power. As such, knowledge from less rigorous research remains important. That said, given the similarity of our current findings to those of our original review, we view it as unlikely that future updates will shed new light on our knowledge of POP's effectiveness in the absence of more rigorous evaluations.

In our view, the question at this point is not whether POP works. The vast majority of the studies we reviewed show prevention benefits. This conclusion is also supported by past narrative reviews (Skogan & Frydl, 2004; Weisburd & Majmundar, 2018) and the summary of simple pre–post case studies provided in our original review (Weisburd et al., 2008, 2010). The key remaining questions surround what characteristics are associated with larger impacts of POP on crime. To truly assess this, future evaluations need to not only be more rigorous, but also must capture and report more data about the problems targeted, the level of problem analysis applied, the specific responses actually delivered and report outcomes more often by crime type than aggregate categories. The current sample of studies have a combination of lack of detailed information on many of these factors and tremendous heterogeneity in what is reported, which makes it difficult to draw strong conclusions about what makes POP most effective.

But irrespective of reporting practices, to build an evidence base that will be useful for practitioners in the field it is important for there to be a robust evidence base that is related to specific problems and specific interventions. This is a limitation more generally in the crime and justice field, but is particularly important to address when we have strong evidence of crime prevention effectiveness of a strategy, as we do here. Practitioners want to know what works in what situations, and which practices are most cost-effective. Building such an evidence base would take a major federal or foundation effort to advance the practice of POP, and literally hundreds of studies testing practices in regard to specific types of problems. In turn, we have limited cost-benefit analysis data from the studies we reviewed. The studies that did examine cost savings generally used limited data to estimate both costs and benefits and rarely did any systematic analysis. For practice it is not simply whether something works, it is equally important to provide a sense of what cost for what benefit. Answering this question should be a major focus of future studies.

The authors of this review plan to do a deeper dive and coding of the eligible studies to see if more light can be shed on these matters in a follow-up publication. The prospects of succeeding in this effort with existing data are unclear, and was beyond the scope and timeline for this funded review.

On this front, it is important for more future studies to evaluate the impact of POP on outcomes beyond the standard crime and disorder outcomes examined in the studies included in this review. These studies tested impacts on aggregate crime/disorder, violence, property crime, disorder, drug sales/use and related outcomes such as probation/parole success or failure. POP was proposed as a flexible model that can be applied to a wide array of problems and our understanding of the model's potential would be enhanced through studies that assess its impact on issues such as cyber-crime, human trafficking and other issues increasingly of interest to criminologists and criminal justice practitioners. Additionally, more future studies should be designed to examine its applicability in reducing resident fear of crime, improving citizens’ opinions of the police, and bolstering collective efficacy. Too few existing evaluations report on such outcomes to allow for a meaningful meta-analysis, but our narrative review of the existing evidence suggests mixed and inconsistent findings across studies. This suggests the need for further research, particularly on POP interventions that include close partnership with and involvement of the community, which might be expected to have the greatest impacts on these perception-based outcomes.

Lastly, this updated review also added the approach of performing meta-analyses using log RIRRs as the effect sizes. This was done based upon in-progress work by David Wilson which argues both that Cohen's D fails to produce effect sizes that are comparable across studies when based on place-based count data and that the conversion of RIRR to Cohen's D is problematic. We still reported Cohen's D, including a majority (27 out of 34) of effects that were converted from RIRR, as we wanted to be consistent with the approach used in our original review (Weisburd et al., 2008, 2010) and other recent Campbell Reviews such as the updated hot spots policing review (Braga, Turchan, et al., 2019). This also allowed us to compare the two approaches.

Our results show that while similar conclusions would be reached about POP's effectiveness using either approach, it does appear based on our comparisons and examples discussed above in Section 5.5 that the Cohen's D approach may understate the impact of place-based interventions, and that the RIRR approach appears to generate effect sizes that are more in line with actual changes reported in the studies themselves. Moreover, being able to convert the log RIRR to relative change in the treatment group versus the control group makes effect sizes more intuitive for researchers and practitioners alike. For instance, for our Cohen's D model our mean overall effect of 0.183 is not going to immediately tell a police leader much about the effectiveness of POP. However, using the RIRR approach allows us to more simply state that the relative reduction in the POP groups versus the controls was 33.8%. This is much more easily interpretable to practitioners, which is an important aim of Campbell Systematic Reviews.

Given that the RIRR approach is both more informative for practitioners and, based upon David Wilson's (in progress) work, more appropriate for place-based studies (while also avoiding the problematic conversion of RIRR to Cohen's D) we encourage future meta-analyses of place-based interventions to adopt this method.