The Effects of Problem-Oriented Policing on Crime and Disorder 1

3.1 Criteria for inclusion and exclusion of studies in the review

The scope of this review is experimental and quasi-experimental studies that include comparison groups. The preliminary eligibility criteria were as follows:

The study must be an evaluation of a problem-oriented policing intervention. For this it is necessary to develop an operational definition of problem-oriented policing. For this review only police interventions following the basic tenets of the SARA model outlined above will be eligible for inclusion. This is to say that such interventions must involve the identification of a problem believed to be related to crime and/or disorder outcomes, the development and administration of a response specifically tailored to this problem and an assessment of the effects of the response on a crime or disorder outcome.

While the main focus of our review follows these criteria, a number of problem-oriented policing experts who were contacted in the study identification stage of our research (see below) suggested that a review which ignores simple pre-post studies without control groups would miss a large number of problem-oriented policing evaluations. Though as we note below, we have strong concerns regarding the methodological rigor of such studies, we did identify such studies and analyze them separately from our main analysis.

3.2 Search strategy for identification of relevant studies

Several strategies were used to perform an exhaustive search for literature fitting the eligibility criteria. First, a keyword search was performed on an array of online abstract databases (see lists of keywords and databases below). Second, we reviewed the bibliographies of past reviews of problem-oriented policing. Third, we performed forward searches for works that have cited seminal problem-oriented policing studies. ³ Fourth, we performed hand searches of leading journals in the field. ⁴ Fifth, we searched the publications of several research and professional agencies (see list below). Our searches were all completed during the fall of 2006. Thus, our review only covers studies published in 2006 and earlier. Sixth, after finishing the above searches and reviewing the studies as described later, we e-mailed the list of studies meeting our eligibility criteria in June 2007 to leading policing scholars knowledgeable in the area of problem-oriented policing. These scholars were defined as those who authored at least one study which appeared on our inclusion list as well as anyone on the list of affiliates of the POP Center (http://popcenter.org/aboutCPOP.html), anyone involved with the National Academy of Sciences review of police research (NRC, 2004), and other leading policing scholars identified by the authors. This helped us identify studies the above searches left out as these experts were able to refer us to studies we missed, particularly unpublished pieces such as dissertations. Finally, we consulted with an information specialist at the outset of our review and at points along the way in order to ensure that we used appropriate search strategies.

The following databases were searched:

Criminal Justice Periodical Index

The following keywords were used to search the databases listed above (in all cases where police is listed we would also use policing and “law enforcement”):

“Problem-oriented policing”

The publications of the following groups were searched:

Center for Problem-Oriented Policing (Tilley Award and Herman Goldstein Award submissions, Problem-Specific Guides for Police)

The following agencies’ publications were searched and the agencies were contacted if necessary:

Home Office (United Kingdom)

The bibliographies of the following sources on problem-oriented policing were reviewed:

Braga (2002). Problem-oriented policing and crime prevention.

Several strategies were used to obtain full-text versions of the studies found through searches of the various abstract databases listed above. First, we attempted to obtain full-text versions from the electronic journals available through the University of Maryland library research port. When electronic versions are not available, we used print versions of journals available at the library. When the journals or articles were not available at the University of Maryland library, we made use of the Interlibrary Loan Office (ILL) to try to obtain the journal from the libraries of other area schools. When these methods did not work, we contacted the author(s) of the article and/or the agency that funded the research to try to get a copy of the full-text version of the study. We were able to identify report documents for all studies that were deemed relevant for full text review.

3.3 Details of study coding categories

All eligible studies were coded (see coding protocol attached in Appendix A) on a variety of criteria including:

Reference information (title, authors, publication etc.)

Joshua Hinkle and Cody Telep (authors of the review) independently coded each eligible study. Where there were discrepancies, either Dr. Eck or Dr. Weisburd reviewed the study and determined the final coding decision. The coding of the pre-post studies was checked by Noah Miller (a graduate student at the University of Maryland). Our coding database, which includes the quantitative data used to calculate the effect sizes we describe below, is available on the Campbell Collaboration website.

4.1 Selection of Studies

We used a broad search strategy to ensure that we identified all relevant publications that met our inclusion criteria. As a result, our initial search produced a large number of hits in the databases searched (i.e. citations). We identified 5282 studies using our set of keywords on the 12 online databases. We narrowed this list by reviewing titles and abstracts and removing any studies not related to policing, any studies not in English ⁵ , any duplicates, and book reviews. This left us with a total of 1964 citations. We then removed any studies that were not related to problem-oriented policing, leaving us with 628 studies. Finally, we removed studies that we were certain did not meet our methodological criteria (e.g. non-evaluation studies that just describe what problem-oriented policing is), leaving us with 124 citations. We reviewed the full text of these 124 studies to make final eligibility determinations. After reviewing these studies, we found four that met our inclusion criteria. We identified an additional 282 studies with our search of agency and research group publications. After more closely reviewing abstracts and full-text of these studies, we found two Goldstein Award submissions that met our inclusion criteria. Our forward search using seminal articles did not identify any additional citations we had not located with our prior searches. The hand search of leading journals in the field also did not identify any additional eligible studies. Thus, after our initial eligibility review, we found six studies that met all of our inclusion criteria.

We emailed 62 policing scholars and practitioners (see list in Appendix B) for feedback on our list of studies and asked for their help in identifying additional studies. With the assistance of these problem-oriented policing experts, we identified three additional eligible studies. In turn, we identified one new study from additional hand searches of bibliographies. Thus, we found a total of 10 studies that met all of our eligibility criteria.

While it is not uncommon in Campbell reviews to find only a small number of studies regarding a specific practice, the absence of a wide body of evidence in the area of problem-oriented policing is particularly concerning. POP represents a broad array of strategies applied to a broad array of problems. The development of systematic knowledge for policing accordingly requires that there be an equally broad array of studies that would allow us to assess what types of strategies are effective in what types of circumstances and for what types of crime. Additionally, this omission of systematic study using rigorous research methods is particularly troubling given the wide spread adoption of problem-oriented policing in the U.S. and elsewhere.

One explanation for the relatively small number of studies that met the methodological criteria of our review may be that much evaluation of problem-oriented policing has used weaker research designs. In our communications with POP scholars, some argued that it was particularly difficult to identify comparison groups for POP programs because problems by their nature were often unique. Accordingly, many problem-oriented policing programs are evaluated using simple before and after research designs. Such designs are generally excluded from Campbell reviews because the absence of a control group makes it difficult to differentiate between general trends in crime and trends produced by the intervention. For example, a decline over a period of time may reflect a general crime trend in a city rather than the direct impact of treatment. While we recognize the difficulties of identifying control groups for POP programs, our review itself documents that such approaches are possible in evaluating POP programs. Moreover, given the wide spread adoption of problem-oriented policing across the U.S. and elsewhere, the lack of a larger body of high quality research evidence is certainly an important finding of our review.

As noted earlier, based on reactions of problem-oriented policing scholars we also identified problem-oriented policing studies that were evaluations using the SARA design, but did not meet our methodological criteria of being a randomized experiment or a quasi-experiment with a comparison group. During our initial database search and search of agency publications, we found 24 studies that met our inclusion criteria, 11 of which were Goldstein and Tilley Award submissions. (We examine the potential biases of the award submissions later.) After further review, we eliminated three of these studies as ineligible after determining they did not have proper data for inclusion or did not fully follow the SARA model. Our consultation with leading policing scholars helped us identify many additional pre/post studies. After a re-reviewing of Goldstein and Tilley submissions to look for eligible pre-post studies, and reviewing additional sources provided by policing experts, we found 24 additional eligible pre/post studies, 22 of which were Goldstein and Tilley submissions, giving us a total of 45 before/after studies. These studies will be analyzed separately and discussed in greater detail in section 4.6.

For the main analysis of this systematic review, the 10 eligible studies identified and discussed below are:

POP in a suburban Pennsylvania park (Baker & Wolfer, 2003)

We do not include any evaluations of “pulling levers policing” in our main analysis, as none of the existing studies include control conditions that met our study requirements (see section 4.6). ⁶ We should also note we did not include Hope's (1994) problem-oriented policing in St. Louis project and the Beenleigh Calls for Service Project (Criminal Justice Commission, 1998). Although both of these studies report on problem-oriented policing interventions with a comparison group, neither includes sufficient data to calculate an effect size. In Hope (1994), there is not exact pre and post crime count data provided and the standardized residual change scores cannot be used for the calculation of an effect size for meta-analysis. The Beenleigh Calls for Service Project (1998) has pre and post crime counts for certain case studies, but does not have the exact data available for the entire project. In particular, there was very limited data comparing the Beenleigh Police Division and the Browns Plains Police Division (the comparison group identified by the authors). We make reference to the overall findings of these studies in our narrative description below, but based on data limitations, we did not include these two POP evaluations in our final statistical analyses.

4.2 Characteristics of studies

The 10 eligible studies come from eight different U.S. cities (Jersey City was the site for two studies) and six wards in the United Kingdom. Lorraine Green Mazerolle was lead author or co-author on three of the studies and David Weisburd was an author or co-author in two studies.

Four of the eligible studies were randomized experiments and six were quasi-experiments with a comparison group. The randomized experiments were all place-based interventions as were four of the six quasi-experiments. The two person-based interventions focused on probationers and parolees in Knoxville and San Diego.

The interventions covered a variety of problems, demonstrating the wide applicability of problem-oriented policing. Two interventions dealt with reducing probationer/parolee recidivism, two targeted drug markets, one responded to vandalism and drinking in a park, one combated crime in hot spots of violence, one addressed school victimization, two tackled problem addresses, and one targeted overall crime. These interventions also used a variety of approaches to address crime and disorder.

We briefly give some background information below on each of the eligible studies. We provide characteristics of the eligible studies in Table 1. More detailed information comparing the studies on the problems addressed, use of the SARA technique, responses, and evaluation design is provided in Table 2.

Baker & Wolfer, 2003

Baker and Wolfer describe a POP intervention in a small Pennsylvania town aimed at targeting vandalism and substance use in a local park. During the scanning and analysis process, officers noted that the park was full of litter and had overgrown brush, allowing offenders to hide from police. Using crime-prevention surveys and crime mapping, they determined that the problem was isolated in the small area in and around the park. To respond, officers target hardened by removing overgrown shrubs. They used other methods of situational crime prevention by installing cameras, repairing fences, improving lighting, locking the park at night, limiting access, and posting rules and regulations. In addition, the police used proactive patrol and increased enforcement of the curfew law to target juvenile offenders. Officers worked with residents to establish a Neighborhood Watch to coordinate cooperation between the police and area residents. To assess the project, researchers used a quasi-experimental design with a comparison group. Volunteers administered 29-question surveys both before and after the project to random samples of residents in the immediate area of the park and a comparison group of residents who lived in the same town, but not adjacent to the park.

Braga et al., 1999

Braga and colleagues document a POP project in Jersey City, NJ designed to address hot spots of violent crime. These hot spots were defined using computerized mapping and then officers worked to determine what problems existed at each hot spot. After initially choosing 28 pairs of violent crime places, the randomized experiment was narrowed to 12 pairs- 12 hot spots received problem-oriented policing and 12 received traditional patrol. In the 12 treatment pairs, officers were required to complete an analysis report assessing the specific problems in the particular hot spot. They were encouraged to use official data and meetings with, or surveys of, community members. Although all the hot spots were chosen because of high rates of violence (typically street fights, drug market violence, and/or robbery), officers also identified widespread disorder problems that included public drinking and loitering. Officers designed a response to specifically target the problems they uncovered in the analysis stage. Thus, the exact response varied by hot spot, but the responses all included some aspect of aggressive order maintenance and most included efforts to make physical improvements to the area (e.g. removing trash, improving lighting) and drug enforcement. To assess the project, the researchers used calls for service data, as well of pre and post observations of physical and social disorder.

Knoxville Police Department, 2002

The Knoxville Police Department describes a program designed in response to citizen complaints about repeat offenders. These repeat offenders tended to be parolees or probationers that received limited supervision and services in the community. Working with the Tennessee Board of Probation and Parole, officers reviewed parolee records and citizen complaints, determining that past efforts such as increased patrol (more arrests) and reduced workloads had been largely unsuccessful. They recognized that these offenders re-entering the community frequently had dysfunctional families and substance abuse and mental health problems. The two agencies created the Knoxville Public Safety Collaborative as a response, combining the resources of the police and probation services and collaborating with 25 human service providers to bring much needed services to parolees. The response involved coordinated and proactive treatment in which the parolee and parole officer developed a release plan, followed by a multi-division staff meeting to discuss treatment options, and then the parolee supervision by a team including police officers, probation officers, and community service providers. The 265 parolees in the program were compared to a historical comparison group of 261 parolees who would have been eligible for the program. This quasi-experimental evaluation was completed by the University of Tennessee School of Social Work.

Mazerolle, Price, & Roehl, 2000b

Mazerolle and colleagues describe a randomized experiment testing the impact of the Beat Health problem-oriented policing program in Oakland, CA. The Beat Health program was designed to address drugs and disorder at problem addresses/street blocks in the city. Sites were referred to the Beat Health team through hotlines, community meetings, and reviews of calls for service. Half of the sites (50) referred were randomly selected to receive the Beat Health treatment; the other half (50) received normal patrol. The analysis used a blocked design to compare residential and commercial addresses separately. The Beat Health intervention involved a team of one police officer and one police service technician visiting a site to identify and analyze the problem and to make contact with the property owner or place manager to try to address the problems. The police attempted to build a close working relationship with individuals who had a stake in improving the property and tried to provide guidance on crime prevention. The intervention typically involved pressuring third parties (usually the landlord of a problem apartment building or property owner) to make changes to improve property conditions. The Beat Health team could also use the SMART (Specialized Multi-Agency Response) Team, made up of city inspectors, to enforce local housing, fire, and safety codes. The team could also instigate legal action against landlords and property owners through civil law. This project used a problem-oriented approach to third party policing: Beat Health teams met with property owners and closely examined problem sites to determine the best course of action to target problems. Calls for service data were used for the assessment.

Sherman, Buerger, & Gartin, 1989

Sherman and associates describe the Minneapolis, MN Repeat Call Policing (RECAP) program designed to respond to commercial and residential addresses with a high number of calls for service. Using calls for service data, the top 500 addresses with the most calls were examined. Schools, city hall, hospitals, police stations, parks, check-cashing locations, and intersections were all removed because police felt these locations were inappropriate for the intervention. The remaining sites were blocked into half commercial (250) and half residential sites (250). These sites were then randomized in rank-ordered pairs with half of the sites assigned to receive a problem-oriented policing treatment and half to receive standard patrol. After some data cleaning issues, a total of 119 residential sites and 107 commercial sites received the treatment. The treatment team was four officers and a sergeant who were assigned to visit each site and use as many sources as possible to diagnose the problem. These sources included analysis of call data and incident reports, on-site interviews of residents, and interviews of place managers. Officers were then supposed to design and implement an intervention plan that needed to be approved by the sergeant. The actual treatment varied greatly across addresses. Officers spent a lot of time helping landlords with problem tenants and providing letters to repeat domestic violence victims informing them of their rights and available services. Commercial responses were even more heterogeneous than residential responses. The time spent at each site also varied considerably with officers visiting some addresses only once and others weekly throughout the yearlong intervention period. The program was assessed using a comparison of calls for service data.

Stokes et al., 1996

Stokes and associates document a problem-oriented policing project designed to reduce student victimization on the way to and from middle school in Philadelphia, PA. Officers recognized that school violence was an issue, and they worked to understand the underlying problems. Using focus groups, victimization surveys, and analysis of police and school data, the police, along with representatives from the Center for Public Policy at Temple University and vice-principals from Philadelphia middle schools all came to better understand the dynamics of students being attacked on their way to or from school. They used crime mapping to visually display unsafe locations identified by students and the student victimization survey provided data on the level of victimization, how often this victimization was reported, and how dangerous students perceived their trip to and from school to be. Using this data, the Philadelphia Police Department decided to create a police-secured safe corridor for students to travel on foot safely to one middle school. Using officers from the Philadelphia PD, the Temple University PD, and the Philadelphia Housing Authority, the police used crime maps to create a corridor 10 blocks long and three blocks wide where police patrols were increased from 8-9am and 2:30-4pm. During these time periods, two foot patrol officers, a patrol car, and a bike patrolled the corridor. A pre and post student victimization survey was used for the assessment. Student responses in the target middle school were compared to responses from students in three similar middle schools.

Stone, 1993

Stone describes a problem-oriented policing project in Atlanta, GA designed to address drug selling and use in public housing projects. Two housing projects were chosen as intervention sites and two were used as comparisons in this quasi-experiment. To analyze the drug problems, a management team was created with representatives from the Atlanta Police Department and the housing authority. The management team conducted resident victimization surveys to determine the extent of problems and understand resident perceptions of crime problems. The research team, along with the police, conducted extensive research to document the drug problem in the area by examining data from the police, drug treatment facilities, schools, courts, social service agencies, and corrections agencies. The management team focused on five problem areas in the response: poor lighting, abandoned cars, abundant litter, poor playgrounds, and improperly strung clotheslines. These five problems were identified by residents, officers, and supervisors, and the management team thought focusing on these problems would help address some of the underlying issues leading to drug problems. There was also an effort to get uniformed officers to work more closely with undercover narcotics detectives and to have all officers work more cooperatively with the Atlanta Housing Authority. The team did successfully work with Georgia Power to implement weekly lighting checks, abandoned cars were quickly removed, resident clean up days reduced the litter problem, and dangerously strung clotheslines that could get in the way of officers were quickly repaired. The program was assessed using pre and post victimization data on whether residents in the target and comparison housing projects had been asked to buy or sell drugs.

Thomas, 1998

Thomas describes the Coordinated Agency Network (C.A.N.) designed to reduce juvenile probationer recidivism in San Diego. The San Diego Police and the San Diego County Probation Department Juvenile Division both recognized that juveniles were frequently being re-arrested after release on probation. In San Diego, low-risk juvenile offenders were typically “banked,” meaning they only had to contact their probation officer by mail. They were largely unsupervised and frequently failed to abide by the conditions of their probation. An analysis of the area revealed that many of these juveniles needed greater supervision because of unstable family lives, and because of their close geographic proximity to major drug ports, gang activity, and a large prison. The police and probation division formed C.A.N. to increase supervision and monitoring of juvenile probationers. Fifteen officers volunteered to help monitor the juveniles and to refer them and their families to community-based support programs. After an initial assessment by a senior probation officer, police officers assigned to each juvenile would make bi-weekly visits to be supervisors and mentors. The program included a graduated model of sanctions and rewards based on the juvenile's compliance with probation along with their performance at school. For the assessment, recidivism rates for a group of 80 C.A.N. participants were compared to a group of 80 similar “banked” juveniles who did not participate.

Tuffin, Morris, & Poole, 2006

Tuffin and colleagues report on the National Reassurance Policing Programme implemented in six wards (neighborhoods) in the United Kingdom. The program was designed to address the “reassurance gap,” the idea that residents are fearful of increasing crime rates even when crime is actually decreasing. This gap has been explained in part by the signal crimes perspective, which argues that certain crimes, particularly certain types of disorder, signal to the community that crime is out of control. Thus, the rates of these signal crimes are more important in generating resident perceptions that actual overall crime rates. The program had three main focuses: having accessible and visible police officers, community involvement in identifying priorities for police, and using targeted police activity and problem solving. A seven-stage model was used to implement the program: Research- officers had to find out about the neighborhood and how to engage residents; Engage- police needed to create conditions for dialogue; Public preferences-officers used surveys, questionnaires, neighborhood meetings, and visual audits to better understand problems facing the community; Investigation and analysis-police used meetings and focus groups to give a deeper analysis to identified problems; Public choices- the police presented the findings of their analysis to residents, so the community could choose priorities; Plan and action-officers developed and implemented a plan with local partners; Review- police completed an assessment of the problem. The specific problems targeted varied by ward, but all included some type of anti-social behavior, and typically involved drug problems. The researchers used total recorded crime as a method of assessment, comparing each target site to a similar comparison ward before and after the implementation of the program.

Weisburd & Green, 1995

Weisburd and Green evaluate the Jersey City, NJ Drug Market Analysis Program. The program identified 56 hot spots of high-activity drug dealing. These hot spots were identified using narcotics sales arrests, drug-related calls for service, narcotics tip-line information, and the assessments of narcotics detectives. Half of these hot spots were randomly assigned to a problem-oriented policing treatment and half received routine enforcement that relied primarily on arrest. The cases were randomized in four statistical blocks, based on volume of drug activity. The program recognized from the outset the need to assign specific officers to specific hot spots to increase accountability, and the need to allow for a diversity of responses to address the problems at a specific hot spot. The program included a step-wise process similar to the SARA model. In the planning stage, officers collected data on the physical, social, and criminal characteristics of each area. In the implementation stage, officers coordinated efforts to conduct a crackdown at the hot spot and use other relevant responses to address underlying problems at the hot spot. Finally, in the maintenance stage, officers attempted to maintain the positive impact of the crackdown. To implement the experiment, squads of narcotics officers were randomly assigned to the treatment or control hot spots. The assessment used calls for service data.

4.3 Narrative review of the impact of problem-oriented policing on crime and disorder

Of the ten eligible studies, eight reported findings in favor of problem-oriented policing, though those effects (as we will see in the next section) vary widely. In Table 3, we provide a summary of results for each eligible study and we provide a narrative analysis of the results here before turning to meta-analytic techniques in the next section.

All of the randomized experiments reported findings suggesting the effectiveness of problem-oriented policing as compared to the control conditions. These experimental studies all employed, at least to some extent, a hot spots approach to using POP (Weisburd & Braga, 2006), which suggests that problem-oriented policing may be particularly effective when used in concert with hot spots policing. In the Jersey City POP in violent crime places experiment (Braga et al., 1999), there was a statistically significant decline in total calls for service and total crime incidents when comparing six months before and after the intervention. For specific call types, there were significant decreases in calls for street fighting, property crime, and narcotics at treatment sites relative to control areas after intervention and significant decreases in incidents for robbery and property crimes. Change in calls for robbery and disorder and changes in incidents for disorder, narcotics, and non-domestic assault incidents were statistically nonsignificant between groups. Social and physical observation data showed improvement in visible disorder in 10 of the 11 treatment areas compared to the control sites after the intervention.

In the Oakland Beat Health study (Mazerolle et al., 2000b), there was a significant decrease in drug calls for service in the experimental sites compared to the control sites using data from 12 months before and after the intervention. Examining only residential sites, experimental drug calls decreased by 13.2 percent while control drug calls increased by 14.4 percent. There was no significant difference between the two groups for disorder, violence, and property calls for service, although only drugs and disorder were primary outcomes. Disorder at commercial experimental sites declined more significantly than disorder at residential experimental sites.

In the Minneapolis RECAP study (Sherman et al., 1989), there was a slightly larger decline in calls for service at target residential sites compared to control sites, but little or no difference in commercial sites when comparing 1986 to 1987 data. For residential sites, calls declined 6 percent in RECAP sites, but increased 0.10 percent in control sites. For commercial sites, calls declined in both sites with a very slightly larger decline in the RECAP sites (10.96 percent vs. 10.70 percent). The residential call decline was more dramatic in the first six months of the experiment when RECAP sites had a 6.96 percent drop in calls compared to an 8.07 percent increase in control sites. Significant findings were reported only for residential addresses included in the study.

While these studies tested problem-solving approaches, it is important to note that focused police attention was brought only to the experimental locations. Accordingly, it is difficult to distinguish between the effects of bringing focused attention to hot spots and that of such focused efforts being developed using a problem-oriented approach. The Jersey City Drug Market Analysis Experiment (Weisburd & Green, 1995) provides a more direct test of the application of problem-solving approaches because experimental and treatment conditions received similar levels of police attention (but a SARA approach was used only in the treatment hot spots). The experimental sites had significantly smaller increases in disorder calls compared to the control sites using seven months of before and after data. In particular, the project had a positive impact on calls related to public morals, suspicious persons, and assistance. The experiment had no significant impact on property crime or violent crime calls for service. Drug related calls for service were not analyzed both because the experimental treatment likely impacted drug-related calls for service (i.e. residents were encouraged to report drug activity to police), and statistical analyses were made difficult by distributional issues in the data (see Weisburd and Green 1995: 727, note 15).

Both of the probationer/parolee quasi-experiments reported significant findings in favor of the problem-oriented policing protocols. In the San Diego Coordinated Agency Network project (Thomas, 1998), the recidivism rate for program participants was only six percent. A random group of similar juveniles not chosen for the program had a 22 percent recidivism rate. In the Knoxville project (Knoxville Police Department, 2002), 29 percent of program participants successfully completed the terms of their parole, while only 11 percent of those in a historical comparison group did not have their parole revoked.

In the Baker and Wolfer (2003) study, the residents living near the park were significantly more likely than comparison group residents to report being the victims of vandalism or seeing public drinking. However, after the intervention, the victimization rates for the target area had declined to the point where there was not a statistically significant difference between the two groups. The authors conclude that the program helped decrease crime in the park.

The Tuffin et al. (2006) report on reassurance policing produced results favoring problem-oriented policing, although these were largely driven by major crime declines in two of the sites. Overall, crime dropped by 4 percent more in the target sites than the comparison sites. But in three of the sites declines were similar to control sites, and in one site the target group had a crime increase while the comparison had a crime decrease. Victimization rates also declined about five percent more in the target sites than the comparison sites. Thus, there was an overall positive finding related to POP and crime control effectiveness, but the impact varied greatly across the sites.

The two studies that did not report findings in favor of problem-oriented policing results were Stone (1993) and Stokes et al. (1996). In the Stone (1993) study, the rate of being asked to buy or sell drugs measured on a resident victimization survey increased in both the treatment and comparison housing projects, but the increase was substantially higher in the treatment area. Violent crime did decrease in the intervention area, but total crime and property crime increased at a rate greater than the comparison sites. In the Stokes et al. (1996) study, the safety corridor proved to be largely unsuccessful. The rate of student victimization actually increased in the target school, while decreasing significantly in the three comparison schools, indicating a backfire effect of the problem-oriented policing intervention. The victimization question was not a fluke; results for the perception of danger question are almost identical with an increase in students perceiving their trip to school as dangerous in the target school and a decrease in the comparison schools.

As we noted above we did not include Hope's (1994) POP study in St. Louis or the Beenleigh Calls for Service Project (Criminal Justice Commission, 1998) as eligible studies, because of a lack of available data for computing effect sizes. Still, these studies did employ a comparison group and were evaluations of POP, so we briefly note their findings here. Hope (1994) found that POP was successful in reducing calls for service in three drug market locations. Declines in surrounding blocks were nonexistent or less substantial. In the Beenleigh project, when Beenleigh was compared to a similar police division that did not have a POP intervention (Browns Plains), there are no significant differences between the two locations. Certain case studies were successful in reducing crime, but there was no major project impact at the aggregate level.

A note on possible displacement and diffusion impacts in place based studies

One concern in studies that examine targeted place based interventions is that crime prevented at targeted sites may “displace” to other areas (Reppetto, 1976; Weisburd et al., 2006). Though we do not examine displacement in our review it has been a focus of a previous Campbell review on Hot Spots Policing (Braga, 2007). That review concluded drawing upon five studies (two of which are included in our review: Weisburd & Green, 1995 and Braga et al., 1999) that spatial displacement was not a significant threat to hot spots policing initiatives, and indeed that the evidence was stronger that there was a “diffusion of crime prevention benefits” (Clarke & Weisburd, 1994) to areas close by than displacement of crime. Our review examines only the effects of problem-oriented policing on targeted problems and not potential displacement or diffusion of crime control benefits either spatially or in terms of methods, crime types or offenders.

4.4 Meta-analysis of the impact of problem-oriented policing on crime and disorder

We completed a meta-analysis of the 10 eligible studies to examine the standardized effect size for each study and to calculate an overall random effect for the impact of problem-oriented policing on crime and disorder. We used Biostat's Comprehensive Meta Analysis program for our analyses and to create the forest plots we present below.

Computation of effect sizes in the studies was not always direct. The goal was to convert all observed effects into a standardized mean difference effect size metric. None of the studies we examined calculated standardized effect sizes, and indeed, it was sometimes difficult to develop precise effect size metrics from published materials. This reflects a more general problem in crime and justice with “reporting validity” (Farrington, 2006; Lösel & Köferl, 1989), and has been documented in recent reviews of reporting validity in crime and justice studies (see Perry & Johnson, 2008; Perry et al., in progress). For the two probation studies (Knoxville Police Department, 2002; Thomas, 1998) and the Stokes et al. (1996) study, we used the proportion of successes (or failures) to calculate an effect size. These calculations all used the odds ratio method. For the Stone (1993) study, we used the difference in pre to post mean change between the treatment and comparison sites ⁷ , sample size, and the t-statistic value from a paired group t-test examining factor scores on a victimization survey. In the case of Weisburd and Green (1995) we calculated effect sizes from exact p-values from the F tests used in the two-way analysis of variance calculations for calls for service data. For Sherman et al. (1989), we used the chi square values comparing the difference in calls for service at RECAP and control targets before and after the intervention. We could find no satisfactory method for conversion of data from Braga, et al. (1999), and therefore converted the estimates to an odds ratio following the method outlined in the Appendix of Farrington et al. (2007) ⁸ . We also used the odds ratio method for the Baker and Wolfer (2003) study, the Mazerolle et al. (2000) article, and the Tuffin et al. (2006) report. We think it important to note that most of the studies reviewed are place based, with only two studies (Knoxville Police Department, 2002; Thomas 1998), both quasi experiments, using person based outcomes. While the very specific components of these two studies make it difficult to distinguish design effects from project components, it is clearly important in the future when larger numbers of studies are available to examine this question.

One problem in conducting meta-analyses in crime and justice is that investigators often did not prioritize outcomes examined. This is common in studies in the social sciences in which authors view good practice as demanding that all relevant outcomes be reported. However, the lack of prioritization of outcomes in a study raises the question of how to derive an overall effect of treatment. For example, the reporting of one significant result may reflect a type of “creaming” in which the authors focus on one significant finding and ignore the less positive results of other outcomes. But authors commonly view the presentation of multiple findings as a method for identifying the specific contexts in which the treatment is effective. When the number of such comparisons is small and therefore unlikely to affect the error rates for specific comparisons such an approach is often valid.

A primary outcome is defined in our review as one that was a major focus of the problem-oriented policing intervention. The police needed to be specifically targeting the crime or call type in an outcome for us to identify an outcome as primary. For example, in the Mazerolle et al. (2000: 220) study, the authors note that the Beat Health program “uses a variety of tactics to resolve drug and disorder issues.” The authors present data on calls for service for disorder, drug crime, property crime, and violent crime. Because of this description of the intervention, we chose to include only drug and disorder calls as primary outcomes, and these were the outcomes we used for our mean effect size discussed below.

Where a number of studies use similar outcome measures, it is possible to make comparisons across studies of outcomes for specific measures (e.g. specific types of crimes). In our review such an approach is not possible, because the types of interventions and types of crimes vary widely as noted earlier. Accordingly we analyze the studies using three approaches. The first is conservative in the sense that it combines all primary outcomes reported into an overall average effect size statistic. The second represents the largest effect reported in the studies and gives an upper bound to our findings. It is important to note that in some of the studies with more than one outcome reported, the largest outcome reflected what authors thought would be the most direct program effect. This was true for the Jersey City Drug Market Analysis Experiment, which examined violent and property crimes, but assumed that the largest program effects given the intervention would be found in the case of calls for disorder (Weisburd & Green 1995). Finally, we present the smallest effect size for each study. This approach is the most conservative and likely underestimates the effect of POP on crime. We use it here primarily to provide a lower bound to our findings.

In Figure 1, we present the mean effect sizes for all eligible studies. ⁹ Five of the studies had just one outcome so the mean effect size will be the same as the largest effect size (discussed below). For the Thomas (1998) and Knoxville Police Department (2002) studies, the outcome is probation/parole success (recidivism rate). For Tuffin et al. (2006), total crime incidents were reported as the primary outcome. In Stone (1993), a victimization survey question was reported that asked residents whether they had been asked to buy or sell drugs and in Stokes et al. (1996) a victimization survey question which asked students whether they had been attacked or bothered on the way to or from school was reported. For the other five studies, we combined multiple primary outcomes. In Baker and Wolfer (2003), we took the mean effect for reports of seeing vandalism and drinking. For Braga et al., (1999), we combined total crime calls and total crime incidents. In Mazerolle et al., (2000b) we averaged calls for service for drugs and for disorder. In Sherman et al. (1989), the two coded outcomes were commercial calls for service and residential calls for service, and for Weisburd and Green (1995), property, violence, and disorder calls for service were all combined. In Appendix C, we provide effect sizes for each outcome for the 10 eligible studies.

Positive effect sizes indicate an effect in favor of problem-oriented policing leading to a reduction in crime and disorder. The forest plots in Figure 1 show the standardized difference in means between the treatment and control or comparison group (effect size) with a 95 percent confidence interval plotted around them for all eligible studies. Points plotted to the right of 0 indicate a treatment effect; in this case, the study showed a reduction in crime or disorder. Points to the left of 0 indicate a backfire effect where crime or disorder actually increased after a POP intervention. We used a random effects model, because as noted earlier, problem-oriented policing interventions are a heterogeneous treatment that can vary considerably between studies. The common factor is the process used by the police. Heterogeneity is also found in the types of problems addressed and outcomes examined. Our assumption regarding the large degree of heterogeneity in our review is confirmed when we examine the Q statistic which was significant at the p < .05 level (Q = 58.240, df = 9). We further examine the issue of heterogeneity of our eligible studies at the end of this section.

Using the mean effect criterion for all eligible studies, we find a strongly significant effect in favor of problem-oriented policing strategies. The size of the effect is relatively modest however, with a standardized mean difference (Cohen's d) of .126. This means that on average the POP intervention led to a .13 standard deviation unit decline in the outcome measures examined. This magnitude of effect is defined by Lipsey (1990) as small but meaningful and could “easily be of practical significance” (Lipsey, 2000: 109). Cohen (1988) however, defines a small effect as having a d value of .20. Importantly, if we had used a simple “vote counting” approach to these data, relying only on statistically significant studies (p<.05) we would have concluded that POP was not effective. This is the case because only 4 of the ten studies met the traditional significance criterion.

In examining the individual effect sizes for specific studies, the two person-based studies have the largest overall effects. Both the probationer/parolee studies have a moderate to large positive impact on probation success. The Baker and Wolfer (2003) and Sherman et al. (1989) studies both have a modest impact on crime, but both fail to reach statistical significance because of large standard errors. Braga et al. (1999), Mazerolle et al. (2000b), and Weisburd and Green (1995) all also show a modest impact on crime and disorder. The Weisburd and Green (1995) study is highly statistically significant and the Braga et al. (1999) and Mazerolle et al. (2000b) studies are statistically significant at the p < .10 level. ¹⁰ The other three studies all failed to show a positive impact of POP on crime and disorder. In the Tuffin et al. (2006) and Stone (1993) studies, there was essentially no impact of POP on crime. The Stokes et al. (1996) study had a highly significant backfire effect; the POP intervention seemed to actual lead to increased student victimization. We discuss limitations of these studies that may have led to these null and negative findings in the next section.

Given the important distinction in methodological quality between quasi-experimental and randomized experimental studies, we also report the results separately by method. In Figure 2, we examine the mean effect sizes for only the four randomized experiments. The overall random effect becomes slightly larger (0.147) and remains highly statistically significant (p<.001). In Figure 3, we look at only the quasi-experiments. The random effect is larger than the overall average (0.158) primarily because of the very large effects in the two probationer/parolee studies, but the random effect estimate across the studies fails to reach statistical significance (p = .108). Thus, size of the effect does not vary greatly based on type of study.

In Figure 4, we present the meta-analytic results for the largest effect size for each study. As we noted above, this can be viewed as an upper limit for the effects of problem-oriented policing based on existing studies. This can also be seen as where problem-oriented policing programs that examined multiple outcomes can be most effective. For studies with a single outcome, this finding is identical to Figure 1. As one would expect, the overall random effect is substantially larger (0.297) than the mean combined effect size and this effect remains statistically significant (p = .0397). ¹¹ Among the five studies with more than one coded outcome, several of the largest effect sizes were substantially larger than the mean. For the Jersey City Drug Market Analysis Program, (Weisburd and Green, 1995), the largest effect (disorder calls for service) was more than four times the size of the mean effect (0.696 vs. 0.147) For RECAP (Sherman et al., 1989) the largest effect (residential calls for service) of 0.369 was nearly double the mean effect and was highly statistically significant. The largest effect for the Beat Health Project (Mazerolle et al. 2000b) (drugs calls for service) was more than double the mean effect. In the Jersey City POP in violent places study (Braga et al., 1999), the largest effect (total incidents) was not substantially larger than the mean, but it did reach statistical significance in this analysis. The public drinking effect for Baker and Wolfer (20003) was about .10 larger than the mean effect, but it still failed to reach statistical significance.

We show the largest effects for just the randomized experiments in Figure 5. As noted earlier, all four randomized studies reach statistical significant when examining just the largest effect, and the overall random effect of 0.394 (p value = .011) indicates a moderate impact of problem-oriented policing on crime and disorder. In Figure 6, we present the largest effect sizes for quasi-experiments. The random effect of 0.167 is substantially smaller than that for randomized experiments and fails to reach statistical significance at the p < .05 level.

In Figure 7, we present the smallest effect size for each study. As expected, the mean random effect decreases substantially to 0.058, but the effect is still positive. We present these results to help bound the findings above on mean and largest effect size. These effects are downwardly biased because, as we noted above, some studies included multiple primary outcomes but assumed the program would have the largest impact in one area. Still, even when POP performs at its “worst,” we still find an overall slight positive impact of problem-oriented policing on crime and disorder.

As we noted above, our ten eligible studies exhibited great heterogeneity, as we anticipated when evaluating an approach like problem-oriented policing that is designed to be applied to a broad array of police problems. In addition, some of the heterogeneity exhibited across studies may be due to complications in computing comparable effect sizes across studies. As we discussed above, the effect size calculation process was not always easy. As a result of the heterogeneity across our eligible studies, we urge caution in interpreting our overall findings.

Publication bias

Publication bias presents a strong challenge to any review of evaluation studies (Rothstein, 2008). Campbell reviews, such as ours, take a number of steps to reduce publication bias, as represented by the fact that six of the 10 eligible studies in our review came from unpublished sources (one dissertation, two government reports, and three unpublished reports or award submissions). Wilson has argued moreover that there is often little difference in methodological quality between published and unpublished studies suggesting the importance of searching the “grey literature” (Wilson, in progress). For our review, there may also be a bias in unpublished studies that are never the less available for review, since two studies were identified through the Goldstein Award competition. The San Diego C.A.N. project (Thomas, 1998) and the Knoxville Public Safety Collaborative (Knoxville Police Department, 2002) were both Goldstein Award submissions. These two studies also reported the largest overall effect sizes, both of which were highly statistically significant. Although these studies were both submitted for an award, and so are biased towards success (because, as we discuss further below, we would not expect police departments to submit unsuccessful interventions to a POP competition), both studies made strong efforts to identify reasonable and statistically valid comparison groups.

We compared mean effect sizes for unpublished vs. published studies. The mean effect size for published studies is 0.147 (p = .00) and for unpublished studies, the average effect is 0.153 (p = .10). The similarity between the mean effect sizes between the published and unpublished literature suggests that publication bias may not have major impact on the outcomes of this review.

We generated a funnel plot to examine for possible selection bias in our results. A visual inspection indicates some asymmetry with more studies with a large effect and a large standard error to the right of the mean than the left of the mean. We used the trim-and-fill procedure developed by Duval and Tweedie (Duval and Tweedie, 2000) to examine how our estimates would change in the absence of this asymmetry. The trim-and-fill procedure determined that three studies should be added to create symmetry. The funnel plot with imputed studies is presented in Figure 8. These additional studies dramatically alter the mean effect size estimate. The mean random effect decreased from 0.126 (95% CI = 0.033, 0.219) to 0.060 (95% CI = -0.042, 0.162). In our analysis, however, the trim-and-fill results may be somewhat misleading. These findings do not necessarily indicate a publication bias; indeed, two of the studies causing the asymmetry were unpublished. As Rothstein (2008) points out, this method assumes publication bias when there is asymmetry towards the bottom of the funnel plot. These studies towards the bottom are smaller studies (since they have a larger standard error) and have a larger effect size. However, it is possible that smaller studies genuinely produce larger effects. Particularly in the case of problem-oriented policing, evaluations tend to show more successful results when the project uses a smaller, more manageable caseload for officers. As we review below, when problem-oriented policing projects endeavor to tackle too much at one time, they often face serious implementation issues. A second issue with trim-and-fit pointed out by Rothstein (2008) is an assumption of a relatively homogenous population of studies. As we noted above, these studies are not at all homogeneous. Using simulations, Terrin and associates (2003) found that the trim-and-fit method can spuriously correct for non-existent publication bias. Indeed, when we examine a more homogeneous subset of our data, the randomized experiments, which all focused to some extent on micro places, the trim-and-fill method finds no need for additional studies. Thus, while the trim-and-fill method led to a decreased random effect estimate, we believe the trim-and-fill method may be overestimating the extent of publication bias.

4.5 Study implementation

Overall, most of the studies report at least a moderate level of success in implementing treatment. Nonetheless, there were specific implementation problems in some of the studies, and this provides a context for understanding differences in impacts across the programs. We review these implementation problems in Table 4.

Of the experimental studies, only Mazerolle et al., 2000b reported full implementation without any significant problems. The Braga et al. (1999) study originally intended for officers to focus on 56 problem hot spots (in 28 matched pairs), but due to organizational changes in the Jersey City Police Department caused by massive retirements and extensive non-POP work, the final project included only 12 hot spots and only limited progress was made in the first eight months of the intervention (Braga, 1997). After limited progress in the first nine months of the experiment, Weisburd and Green (1995) extended the intervention period to achieve fuller implementation. The experiment achieved full implementation during the last five months of the intervention.

The Sherman et al. (1989) RECAP study presented more serious intervention problems (see Buerger, 1993). There were multiple issues with the selection of hot spots for the intervention. Even after extensive efforts to remove duplicate calls from the computer logs, the researchers estimated that up to 15 percent of calls were “mirrors”–duplicates created as a result of multiple people calling 911 for the same incident. In addition, certain high call addresses showed remarkable instability in examining year-to-year call trends, affecting the precision of estimates. Certain addresses that were reviewed by police and thought to correspond to separate places were actually found to be different entrances for the same location, leading to problems when initially one location could be both in the treatment and control group. In implementing the project, the team of five officers assigned to the intervention was overwhelmed by the number of hot spot locations. In turn, the 226 addresses with a multitude of different problems were difficult to adequately respond to in a year. The absence of calls for service reductions in the second half of the experiment may be a result of officer fatigue with the intervention and an inability of officers to stay motivated during the entire year. In addition, the sergeant in charge of the RECAP team changed midway through the experiment, which may have altered the course of the treatment.

The most “successful” quasi-experiments, the two programs to reduce probationer/parolee recidivism, faced no major implementation difficulties. In turn, though these studies could not rely on the strong assumptions of a randomized experiment, they put significant effort in trying to identify valid comparison conditions. The Knoxville Police Department study (2002) made a particular effort to choose a comparable historical sample of parolees and the University of Tennessee assisted with statistical analyses to offer evidence of compatibility. The San Diego C.A.N. project (Thomas, 1998) also took strides to use a well-chosen comparison group by comparing the 80 project participants to a random sample of 80 juveniles who were on probation, but not chosen for program participation.

The Baker and Wolfer (2003) study did not evidence significant implementation failures, but the evaluation method was potentially problematic. The comparison group of borough residents not living near the park could still have included residents that used the park and were aware of the police intervention. The survey sample sizes were also fairly small, which helps explain the large standard error for the effect size estimates.

The other three quasi-experiments had more substantial problems, which may explain the study outcomes observed. Stone (1993) reported that the Atlanta Police Department did not seem entirely interested in properly implementing the POP project. Many officers did not view problem solving as “real” police work, so effort was often limited. There was a lack of administrative support from top officials in the department and the POP training was poorly delivered and limited. In addition, Atlanta hosted the Democratic National Convention prior to the intervention forcing officers to delay vacations because of high staffing demands. Finally, as the intervention began in the summer, officers frequently took time off, leaving the POP program chronically understaffed.

Stokes et al. (1996), which produced the only backfire effect in our review, also evidenced implementation difficulties, in this case with their school safety corridor. The largest problem seemed to be that despite an awareness campaign, two-thirds of students at the target school reported they were unaware of the existence of the corridor. In addition, even though violence was more likely in the post-school afternoon hours, the corridor was more poorly staffed during this time period, due to police shift changes and more limited police resources. Also, the victimization survey used by the researchers was not ideal for a middle school population, and many students had difficulty answering the questions.

Tuffin et al. (2006) reported a number of problems with full implementation of Reassurance Policing. Their process evaluation found that only two of the six target sites fully implemented the program. The other four sites had difficulties in effectively partnering with the community and using targeted problem solving. The sites that fully implemented the response showed the strongest results in favor of problem-oriented policing.

As a final note, some scholars have recently questioned whether problem-oriented policing as practiced in the field meets Goldstein's (1979; 1990) original criterion of POP (Cordner and Biebel, 2005; Braga and Weisburd, 2006). It is clear that in some of these evaluations, officers did not complete a thorough analysis of problems prior to developing a response. We do not have detailed process evaluations on all of the studies, but “shallow” problem-solving likely occurred in many of these studies. Thus, although we have focused on response implementation in this section, scholars argue that most POP interventions fail to fully follow through on the principles of POP, regardless of the success of the response.

4.6 Pre/post studies

As noted earlier, we also collected pre/post studies that did not have a control or comparison condition. These studies are weaker methodologically, but are more numerous in the problem-oriented policing literature. We found a total of 45 pre/post or before/after design studies that met our new inclusion criteria for pre-post studies. Typically, these studies examined official crime data before and after a problem-oriented policing intervention to determine how the POP project affected crime. These studies rarely took statistical steps to account for “history,” the idea that crime rates may be rising or falling independent of the specific problem-oriented policing project.

We should note that these studies vary somewhat in methodological quality and not all can be categorized as “simple pre-post.” Braga and colleagues (2001) evaluation of the Boston Gun Project, for example, used a time series analysis and a comparison to similar sized cities to assess the impact of Operation Ceasefire on youth homicide rates. ¹² This is certainly more methodologically rigorous than just an assessment of pre and post crime counts. We chose to include the Braga et al. (2001) study in this section (rather than in the main analysis), because we found this comparison to other cities insufficient for meeting our inclusion criteria. Cities chosen as comparisons for Boston were matched only on population (i.e. the largest cities in the U.S. were used) or geographic proximity (i.e. the largest cities in New England were used). We found these matching techniques to be inadequate to show that these other cities could be validly compared to Boston.

In Table 5, we briefly summarize each study by providing a description of the problem, the response, and the findings. These studies covered a wide variety of problems ranging from neighborhood disorder to homicide. As with our eligible studies, responses also varied greatly, but frequently included a combination of increased community involvement, targeted enforcement, and situational/environmental improvements.

Thirty-two of the 45 studies come from Goldstein or Tilley Award submission. Both of these awards are given to police departments for outstanding problem-oriented policing projects that are innovative, use effective problem solving, and show success in reducing crime. The Goldstein Awards began in 1993 and are given by the Center for Problem-Oriented Policing. Most submissions come from American departments, although departments from the United Kingdom, Canada, and Australia have also submitted entries. The Tilley Awards are given to departments only in the United Kingdom and have been administered by the Home Office since their inception in 1999. The Center for Problem-Oriented Policing website includes a database of every submission for both awards, providing a large resource of POP case studies.

Since many of our pre/post studies were submissions for an award, they almost exclusively report on successful problem-oriented policing interventions. This makes sense, as it would be illogical for departments to attempt to win a POP award with a project that was not effective, (though most submissions are not accompanied by a systematic evaluation). Thus, over half of our pre/post studies would appear to be biased towards success. This leads to a potential publication bias (Rothstein, 2008), or in this case, a non-publication bias. In our case these non-published award submissions may actually be more positive than the published literature. We address this issue below.

In Figure 9, we use a bar graph to display the percent change in crime and disorder reported in each study. When more than one primary outcome was present in a study, we averaged to create a single outcome for every study. These outcomes correspond to the findings described in Table 5. The results overwhelmingly are in favor of problem-oriented policing effectiveness. Of our 45 pre/post studies, 43 report a decline in crime or disorder after the problem-oriented policing intervention. Thus, even though 32 of our studies were award submissions and 31 of these showed a positive impact, 12 of our 13 other studies also reported a beneficial impact of problem-oriented policing. Only one study (Maguire & Nettleton, 2003) reported an increase in crime after using POP. The average percent change in crime over all studies was a sizeable 44.45 percent decrease in crime.

To account for variation in sample size (i.e. crime incidents or calls for service) between studies, we calculated a weighted average percent change by weighting each study by the inverse of its variance and assuming crime follows a Poisson distribution. An approximation to the variance based on the Poisson distribution is:

Variance = ( pre count + post count ) / [ pre count 2 * ( pre count / post count ) ]

With this sampling variance, we constructed a confidence interval around the percentage change for each study. A plot of proportion change with confidence intervals is presented in Figure 10. After weighting each study by the inverse of its variance, we recalculated the average percent change. Even with weighting, the average decrease in crime is still 32.49 percent. This represents the fixed effect mean estimate for the impact of problem-oriented policing on crime and disorder in the pre-post studies. Even though these before and after studies do not employ the methodological rigor of a randomized experiment, they do consistently show a substantial impact of problem-oriented policing on crime and disorder, both in the award submissions and published journal articles.

To address our publication bias concern, we compared the percent change for all studies and then for published and unpublished studies separately. We present these findings in Table 6. As noted above, the overall percent change for all studies was a 44.45 percent decrease in crime and disorder. When, we examine only award submissions, there is a larger percent decrease of 47.79 percent. For the non-award submissions, the percent decrease is smaller, but still substantial (35.55 percent). For the six published studies, the average percent decrease is very similar to the award submissions (47.42 percent). Thus, although there is variation across publication type, the results are not substantively different. Across publication medium, problem-oriented policing is associated with a sizable crime decline in before/after studies that do not employ control groups.