Abstract
1. Background for the Review
Legitimacy in policing influences the capacity of police to foster compliance and cooperation. It reflects a “social value orientation toward authority and institutions” (Hinds & Murphy, 2007, p.27) and is central to our understanding of social institutions, civil order and the derivation of compliance, cooperation and obedience (Tyler, 2004). Evidence shows that it is a person's belief in the legitimacy of the authority or institution issuing a command that “leads people to feel that the authority or institution is entitled to be deferred to and obeyed” (Sunshine & Tyler, 2003, p.514).
Scholarly interest in police-citizen interactions, perceptions of police treatment and the legitimate authority of the public police is not new (e.g. Bayley & Mendelsohn, 1968; Bellman, 1935; Decker, 1981; Parratt, 1938; Reiss, 1971; Walker, Richardson, Williams, Denyer & McGaughey, 1972; Winfree & Griffiths, 1971). But it is the contemporary stream of research by Tom Tyler and his colleagues, starting in the late 1980s, that spearheaded renewed interest in police legitimacy that continues to shape the field today (see Tyler, 1988, 1990, 1994, 1997, 1998, 2001, 2003, 2004; Tyler & Huo, 2002; Tyler & Fagan, 2008). This body of research finds that citizens are more likely to comply with police directives when they view police as legitimate (Tyler, 1990, 1997). Moreover, Tyler and his colleagues show that police legitimacy encourages law abiding behavior not just during a police-citizen encounter, but also outside of encounters (i.e. in everyday life, such as abiding by traffic rules) (Sunshine & Tyler, 2003; Tyler & Huo, 2002). As Tyler (2004, p.85) suggests, unless the police are “widely obeyed” by the public, the capacity of police to maintain order is compromised (see also Tyler, 1990).
Police legitimacy also influences cooperation with the police. Indeed, police legitimacy influences the likelihood that citizens will assist the police in crime control and prevention by, for example, engaging in formal and informal crime prevention activities and reporting crime or “suspicious activities” to the police (Sunshine & Tyler, 2003, p.541; Murphy, Hinds & Fleming, 2008; Tyler, 2004; Tyler & Fagan, 2008). Thus, by improving or increasing perceptions of police legitimacy, police may increase citizen compliance with the law (i.e. in the absence of police directives), in addition to encouraging cooperation and compliance with police.
Overall, research shows that when people perceive the police as legitimate they are more likely to report higher levels of satisfaction and confidence in the police (both for individual officers and the institution), perceive the police as effective in their crime control efforts, be more willing to assist police, as well as be more likely to accept the manifest outcomes of an interaction with police (Tyler, 2004). Arguably the most important contribution of Tyler's research over the last twenty years is the deepening of our understanding as to how the efficiency and effectiveness of the order maintenance role of police is dependent upon the public's support for, and perceived legitimacy of, the public police.
Considering the importance of police legitimacy in shaping effective and efficient order maintenance, scholars have sought to explicate the processes that influence legitimacy, identifying procedural justice as perhaps the most important antecedent of police legitimacy (Hinds & Murphy, 2007; Tyler, 2004). While instrumental factors (e.g. police performance, distributive justice) contribute to police legitimacy, Tyler (2004, p.91) argues that “the legitimacy of authorities and institutions is rooted in public views about the appropriateness of the manner in which the police exercise their authority.” Broadly speaking, therefore, procedural justice refers to the use of fair procedures in the decision-making process (Hinds & Murphy, 2007; Murphy et al, 2008; Thibaut & Walker, 1975).
Thibaut and Walker (1975) first used the term ‘procedural justice’ to refer to one's perception of treatment during decision-making processes. Since this time, scholars across many different contexts and disciplines (e.g. taxation compliance and organizational behavior) have studied the impact of a person's treatment in a range of different decision-making forums. In policing, interest in procedural justice emerged in response to a general dissatisfaction with police behavior and the treatment of citizens, prefaced by accounts of pervasive police corruption and police misconduct (e.g. racial profiling, excessive force) (Reiner, 2000, 1985). The procedural justice perspective also came at a time when policy makers expressed concerns about police inadequacies in dealing with upsurges in crime (Maher & Dixon, 1999; Weisburd & Braga, 2006), leading to a general loss of confidence in traditional police responses to crime (Weisburd & Eck, 2004). These concerns created fertile ground for the genesis of justice based on fair and respectful processes and procedures.
Tyler (2004) identifies three factors that define ‘procedural justice’ in interactions between police and the public: citizen participation in the proceedings prior to an authority reaching a decision, perceived neutrality of the authority in his/her decision, and whether or not the authority showed dignity and respect throughout the interaction (Tyler, 2004). Mastrofski (2009) adds ‘trustworthy motives’ as an important component of procedural justice. It is the interplay across these four core factors that shape police encounters with citizens and subsequently influence perceptions of police legitimacy.
But to describe procedural justice as a holistic antecedent of legitimacy over-simplifies what is, in fact, a complex process. Hawdon (2008) for example, offers an emerging perspective that highlights this complexity by hypothesizing that the relationship between procedural justice and legitimacy is not linear, but rather circular and reciprocal. That is, according to Hawdon (2008) perceptions of institutional legitimacy lead to a belief about procedural justice, which then shapes legitimacy, and so on. While this supposition has not been tested, other authors have also found evidence for this self-perpetuating cycle (e.g. Brandl, Frank, Worden, & Bynum, 1999; Reisig & Chandek, 2001; Tyler, 2004). Despite its apparent theoretical complexity, a number of studies explore and show the importance of the relationship between legitimacy and procedural justice (Hinds & Murphy, 2007; Jackson & Sunshine, 2007; Paternoster et al, 1997; Sunshine & Tyler, 2003; Tyler, 1990).
The importance of the procedural justice process reaches beyond simply improving legitimacy. The power of improving legitimacy in the police lies in what perceptions of legitimacy can offer the police, both individually and institutionally, and what this can offer the community. Of benefit to the police is, as noted above, increased compliance: police are more able to encourage compliance with both police directives and the law when they are perceived to be a legitimate authority (Sunshine & Tyler, 2003). Non-compliance, on the other hand, poses a danger for both the police and citizens, as it may precede violence towards police officers, which may increase the risk of harm to both the police and citizens at the encounter (Reiss, 1971). Furthermore, Tyler and Huo (2002) and Sunshine and Tyler (2003, p. 519) suggest that the public are more likely to allow more “intrusive police tactics” when the police are perceived to be legitimate, thus allowing police more operational flexibility in their efforts to control crime.
Research finds that police-citizens contacts involving the use of procedural justice have a direct effect on enhancing the quality of police-citizen interactions, leading citizens to be more satisfied with the interaction (Mastrofski, Snipes & Supina, 1996; McCluskey, 2003; Reiss, 1971; Tyler & Fagan, 2008; Wells, 2007). Tyler and Wakslak (2004) also identify a number of other positive outcomes from procedurally just encounters: for example, people who had recent contact with courts or police were less likely to believe they were racially profiled, and more likely to accept the decision made by authorities if they were treated in a procedurally just manner (Tyler & Wakslak, 2004). Similarly, research in Australia (Hinds & Murphy, 2007) and the UK (Jackson & Sunshine, 2007) shows that individual perceptions of procedural justice are associated with global perceptions of police legitimacy and satisfaction with police.
Research also identifies the potential wider community benefits of procedurally just encounters. Murphy and colleagues (2008), for example, examined the impact of a community policing intervention on perceptions of police legitimacy and cooperation with police, finding that procedurally just encounters influenced changes in general perceptions of police legitimacy over time, and that perceptions of police legitimacy subsequently influenced cooperation with police. Furthermore, Tyler and Fagan (2008) found that legitimacy was related to both the willingness of citizens to report crime to police, as well as the willingness to work with other community members to control and prevent crime.
Community attitudes and perceptions of legitimacy are also likely to be affected through police encounters with third-parties (e.g. Mazerolle & Ransley, 2006), such as local business owners, school principals, and parents. One example of this is the Ten Point Coalition Collaboration in Boston (Berrien & Winship, 2002). Here Berrien and Winship (2002) argue that the involvement of a third party – in this case a group of ministers (i.e. the Ten Point Coalition) – fostered the legitimization of ‘police activities’ within an inner-city community. By providing an “umbrella of legitimacy for police efforts to prevent and control crime” Berrien and Winship (2002, p.203) believe that the involvement of the Ten Point Coalition in Boston contributed to a reduction in youth violence. The Boston example is illustrative of a way in which police may develop legitimacy through third-parties.
In summary, the research literature suggests that police legitimacy engenders compliance, fosters cooperation, improves citizen satisfaction with police and thus facilitates the capacity of police to maintain order and control crime. Moreover, research finds that police adherence to the principles of procedural justice – including citizen participation, neutrality of decisions, demonstrations of dignity and respect and trustworthy motives throughout interactions – cultivates perceptions of police legitimacy. As such, procedurally-just interactions between police and community members are found to be important not only in micro-level, one-on-one encounters between police and individual citizens, but also in the more macro context of police engagement with third parties (e.g. place managers and other regulators) as well as with broader communities of interest.
1.1 Purpose and contribution
The proliferation and growing importance of research in the area of legitimacy, procedural justice, and order maintenance over the last twenty years demands careful review of empirical evidence to help police and policy makers understand more about the direct and indirect crime control benefits of this approach. Further, a systematic synthesis of the available evidence will assist in clarifying the nature of empirical work in this area. Currently, we lack a comprehensive summary of the literature on police legitimacy and we lack synthesis and analysis of the empirical evidence of the policing methods that encourage legitimacy in policing. Therefore, this systematic review will investigate what is known about police interventions designed to facilitate legitimacy (either explicitly or implicitly by using elements of procedural justice) in policing at the micro level (i.e. during police-individual encounters such as with offenders, victims, witnesses and other citizens) and at the macro level (i.e. during police encounters with groups and communities such as during community events, at schools, and in business communities). Results from this systematic review will provide evidence for policy-makers and policing agencies to (1) implement modes of police delivery that advance citizen perceptions of legitimacy (2) provide police with evidence-based models to assist them in performing their duties, and (3) hopefully improve citizen compliance and enhance the public's perceptions with respect to the police.
2. Objectives of the Review
This review will synthesize the existing published and unpublished empirical evidence on the effects of legitimacy in policing. Studies that evaluate “legitimacy interventions” will be included if they are led by public police from any level of government (i.e. local, state and federal law enforcement officers). We have carefully chosen the word “led by public police” to eliminate those interventions that are led by regulatory entities within non-police agencies (e.g. Taxation Departments) and which involve police, but are not organized, led or administered by publicly-funded police officers.
We will provide a systematic review of the direct and indirect benefits of approaches that foster legitimacy in policing. We will include policing approaches that include at least two of the principles of procedural justice (participation, neutrality, dignity/respect, trustworthy motives). That is, our review will include any type of public police intervention (e.g. routine patrol, traffic stops, problem-oriented policing, conferencing) where there is a clear statement that the intervention involved some training, directive or organizational innovation that sought to increase legitimacy or where the intervention involved some training, directive or organizational innovation provided to or by the police which used at least two core ingredients of procedural justice such as police encouraging citizen participation, remaining neutral in their decision-making, and demonstrating dignity and respect throughout interactions and include a direct outcome consistent with improving legitimacy. The direct benefits of fostering legitimacy in policing are expected to include increased compliance, cooperation and citizen satisfaction with police. The indirect benefits are expected to include reductions in crime and disorder. We will examine micro-level interventions between police and individual citizens that aim to foster legitimacy as well as those macro-level interventions between police and communities that also aim to foster legitimacy in policing more generally.
Specifically, we will provide: A summary of empirical evidence aimed at encouraging legitimacy in policing, An inventory of interventions identified in the literature that have, to date, been used for the purpose of promoting legitimacy in policing, A summary of mean effect sizes (i.e. Hedges g, odds ratio) for the interventions, A summary of the direct (e.g. compliance, cooperation, satisfaction) outcomes of interventions that seek to enhance legitimacy in policing, as well as indirect (e.g. crime and disorder) outcomes where applicable, Moderating variables that may influence the effectiveness of interventions designed to improve police legitimacy either explicitly or implicitly through the employment of at least two procedural justice ingredients and a direct procedural justice outcome (e.g. individual level demographic variables including race, gender, age of the encountered citizens; group and community-level variables including socio-economic status, racial composition and other community characteristics in which the intervention is delivered; and the context in which the policing interventions occur such as routine patrols, traffic stops, problem oriented policing).
3. Methods
This review will synthesize existing published and non-published empirical evidence on the effects of interventions designed to improve legitimacy (either explicitly or implicitly through the employment of at least two procedural justice ingredients and a direct procedural justice outcome) in policing and provide a systematic assessment of the direct and/or indirect benefits of these interventions. The stages of this review and the criteria used to select eligible studies are described below.
3.1 Criteria for inclusion and exclusion of studies in the review
While there is a considerable amount of research into legitimacy in policing and procedurally just approaches in policing, the overwhelming majority of studies do not use experimental designs. This was highlighted recently by Weisburd, Mastrofski & Telep (2009), who stated,
[existing studies provide important insights into our understanding of legitimacy and procedural justice in policing, but there has not been an experimental field study of testing key propositions set forth by proponents of legitimacy policing (p.1).
The legitimacy literature is typically general in nature and is not restricted to particular participants (e.g. young people, prolific offenders, or minority groups) or discrete crimes (e.g. drug offences) or with community problems in mind (e.g. communities with a history of police-citizen hostility). As such, our inclusion criteria will include public police interventions that focus on specific types of individuals (e.g. young people or drug dealers or community members) or on a collection of different groups of people (or both).
Sections 3.1.1 to 3.1.6 provide detail on the period of time to be covered by the review, as well as the types of studies, participants, interventions and outcomes. In addition, a typology for categorizing eligible research is proposed in section 3.1.7. A flow diagram reflecting the inclusion/exclusion criteria is provided and incorporated in Appendix 1: Coding Sheet for Legitimacy in Policing Review.
3.1.1 Period of time to be covered by systematic review
A preliminary exploration of published and unpublished literature focusing on authors who have given significant consideration to the ‘design’ of procedural justice and legitimacy (e.g. Tyler, Murphy, Hinds, Kane and Hawdon) was conducted to determine the period of time that should be covered in this review. Procedural justice and legitimacy as criminal justice concepts have developed significantly from 1990 when Tyler's influential book Why People Obey the Law was first published; however, our preliminary examination revealed some seminal pieces were published in the 1980s that would be relevant to the review (e.g. Tyler and Lind's (1986) Procedural processes and legal institutions, Roehl's (1988) Measuring perceptions of procedural justice, etc.). Consequently, the research team decided to include literature from 1980 in order to be inclusive of significant material on procedural justice and legitimacy in policing. Results from a systematic search conducted on behalf of the UK National Policing Improvement Agency (NPIA) systematic search (Bennett, Denning, Mazerolle & Stocks, 2009) identified 794 documents on procedural justice, 781 of which were published from 1980 to April 2009. Graph 1 illustrates the increase in the literature on legitimacy since 1980.
3.1.2 Types of study designs
To be included in the systematic review, studies must have used one of the following research designs: (a) an experimental (randomized) design involving at least two conditions with one condition representing a public police intervention designed to increase (either explicitly or implicitly through the employment of at least two procedural justice ingredients and a direct procedural justice outcome) police legitimacy (see types of intervention defined in section 3.1.5) and another representing a control condition (eligible comparison/control conditions that could be the same public police intervention as the experimental intervention or any other type of criminal justice intervention, but is absent a directive, training or organizational innovation that is designed to increase (either explicitly or implicitly through the employment of at least two procedural justice ingredients and a direct procedural justice outcome) police legitimacy); (b) a quasi-experimental (non-randomized) design involving at least two conditions with one condition representing an intervention designed to increase (either explicitly or implicitly through the employment of at least two procedural justice ingredients and a direct procedural justice outcome) police legitimacy and another representing a comparison condition (eligible comparison conditions are the same as above (these designs may have a pre-test but this is not an essential feature for inclusion; and (c) a quasi-experimental interrupted time-series design, i.e., a design that involves an aggregate measure, such as crime rate, measured at numerous time points both prior to and following the initiation of the police intervention that involves a directive or training to increase police legitimacy. The unit-of-analysis for eligible designs may be individuals or higher levels of aggregation, such as communities or third parties. Note that the quasi-experimental comparison group designs may be cross-sectional surveys that include a variable allowing for the categorization of observations (i.e., individuals, communities, etc.) into an intervention and comparison group.
3.1.3 Types of statistical data
To be eligible for inclusion in the analysis, studies must be reported in such a manner that effect sizes can be identified and/or calculated (see section 3.5 for detail on statistical procedures). Where it is clear that effect sizes could be drawn but missing content makes this impossible, the study will be included if missing data can be provided by corresponding authors by May 15th, 2010. Studies from which the effect size cannot be calculated will be included in the systematic review so that a comprehensive inventory of interventions designed to (either explicitly or implicitly through the employment of at least two procedural justice ingredients and a direct procedural justice outcome) encourage legitimacy in policing can be provided.
3.1.4 Types of participants/units of analysis
This review is interested in the significance of the interaction between police and the public and the impact that such interactions have at either a micro level (i.e. individuals) or macro level (i.e. group, community and/or third parties). Consequently, the review will include studies which focus on the way in which the police interact with: The individual (citizen, victim, offender etc.), and/or The group (community, group gathering etc.), and/or Third parties (religious leaders, community advisors etc.).
As such, studies collecting data on individuals, groups and or third parties as the units of analysis will be eligible for inclusion. These different units will be synthesized separately.
3.1.5 Types of interventions
Eligible interventions will be limited to those that define, in the intervention description, that there was some type of training, directive or organizational innovation provided to or by the police to encourage, foster or facilitate legitimacy in policing. We will also include studies which have a direct outcome listed in section 3.1.6 and define, in the intervention description, that there was some type of training, directive or organizational innovation provided to or by the police which used the following (at least two) core ingredients of legitimate policing: Citizen participation Perceived neutrality of the authority Dignity and respect Trustworthy motives
The intervention would involve the police in activities (both pre-arrest and post-arrest) that depict routine policing, traffic stops, investigations, warrant execution, problem-oriented policing, conferences, school-based programs, crackdowns and other types of police approach where there is a clear training program, directive or organizational innovation to approach contact with the public for the purpose of fostering legitimacy or where the elements consistent with legitimate policing have been used. The context of the intervention will, of course, be coded (see Appendix 1). We would also include police interventions that are aimed at involving police officers in community events (Murphy et al, 2007) and/or with third parties (Berrien & Winship, 2002).
Studies that focus on how other regulatory agencies (e.g. taxation departments, local governments, child safety departments etc) interact with the individuals, groups and third parties will not be included. Additionally, we will not, include those interventions that relate to within-police agency management. These types of studies aim to increase legitimacy within organizations in order to improve/encourage (for example) job satisfaction for police officers.
The control intervention will be those encounters undertaken by police or any other criminal justice institution, that do NOT entail a directive, training and/or organizational innovation that is aimed at encouraging legitimacy in policing or does not utilize at least two ingredients of legitimate policing (e.g. citizen participation, perceived neutrality, demonstrating dignity and respect, establishing trustworthy intentions).
3.1.6 Types of outcome measures
Studies will be eligible if they measure the effects of interventions aimed at encouraging legitimacy in policing (or use at least two ingredients of legitimate policing) and report at least one of the following direct or indirect outcomes::
May include measures of at least one of the following outcomes (the named outcome
Obligation to obey police Moral obligation to obey police Obey the police with good will Police legitimacy Respect for the police Confidence in police Obligation to obey the law Moral obligation to obey the law Obeying the law is the right thing to do
Fairness Police try to be fair when making decisions Opportunity to express views before decisions are made Listen to people before making decisions Neutrality Police make decisions based on fact, not personal biases or opinions Treat people as if they can be trusted Treat people as if they will only do the right thing when forced to Respect Police treat people with dignity and respect Politeness of police
Cooperation with police Would call police to report a crime Provide information to police Report dangerous/suspicious activities Willingly assist police if asked
Trust police Confidence in police Satisfied with the way police do their job
Belief that reoffending will create problems in social relationships How much of a problem would it be if you were arrested again How much of a problem would it create for your life if your family and friends found out you were arrested How much of a problem would it create for your life if the public knew you were arrested (e.g. name and offence printed in the newspaper) If caught again, how tough would your punishment be
Intention to comply with police in future Behavioral compliance
Police effectiveness How good a job are they doing How satisfied are you with the way they solve problems Fairness of outcomes How satisfied are you with the fairness of the outcomes people receive Fairness of procedures How satisfied are you with the fairness of the way that people are treated
May include measures of one or more of the following: Reduction in reoffending Reduction in crime Reduction in social disorder
3.1.7 Language and geographic origin
There are no exclusions on the geographic location of the studies; however, to be included in this review, the study must be written in English, French or German as there is ready and reliable capacity to translate such documents within the research team.
3.1.8 Publication status
Both published and unpublished studies will be eligible for this systematic review.
3.2 Search strategy for identification of relevant studies
The research team proposes to utilize a database of documents identified during the NPIA systematic search of procedural justice and legitimacy literature (Bennett et al., 2009). The search strategy included published and unpublished literature that was available from January 1 1980 to April 1 2009. Of the 20,600 ‘hits’ reviewed, 2,526 records were identified and coded as relevant to procedural justice and legitimacy and of these 794 were unique records/references. Researchers recorded search information (date of search, database and search terms used), research information (design, method(s), agency, outcome, population) and reference information in a Microsoft Excel spreadsheet so that further interrogation of the data could be conducted at a future date. The sections below detail the search strategy used for the systematic search.
3.2.1 Keyword formulation
The research team conducted a series of pilots on single and combined terms before deciding on a final list of keywords which are presented in Table 1.
Keywords for systematic literature search
Compound terms (e.g., procedural justice; criminal justice) were considered as a single term and entered into searches in quotes (i.e., “procedural justice”). This strategy ensured that the database searched for the entire term rather than “procedural” AND “justice”, which would clearly produce very different results. In addition, search terms with multiple iterations from a base word stem (e.g. fair, fairness, fairly) were typed in as word* (e.g. fair*). This approach enabled the researcher to capture relevant literature with fewer searches, thereby saving time.
The research team determined that the Tiers searched independently generated a vast number of hits. For example, searching on the term ‘police’ in isolation resulted in 59,869 records using Cambridge Scientific Abstracts (CSA), whilst ‘procedural justice’ on its own produced 849 hits, ‘compliance’ produced 10,005 and ‘study’ produced 309,253.
Results from a series of pilots suggested that the most effective searches (material retrieved relative to material found) combined Tier 1 and 2 search terms which were focused on criminal justice agents and procedural justice plus associated terms. Additionally, combining criminal justice agencies (Tier 1) with outcomes (Tier 3) drew out literature on methods/factors that would have an impact on effects such as compliance, cooperation and confidence but might not necessarily have been derived from procedurally just procedures (e.g. risk of being caught). These results produced less favorable results but did provide some important material that was not captured using other search terms combinations. Evidentiary terms (Tier 4) were added to Tier 1 and Tier 3 terms to help increase the inclusion rate (the number of relevant documents retrieved relative to the number of relevant documents found).
In summary, there were two search iterations conducted resulting in 104 searches per database/datasource: Tier 1 + Tier 2 (8 × 7 keywords = 56 searches) and Tier 1 + Tier 3 + Tier 4 (8 × 6 × 1 keywords = 48 searches)
A database with each search string was developed so that researchers could ‘cut and paste’ the keywords into select databases to reduce errors (incorrect spelling, missing keywords or mixing keyword combinations).
3.2.2 Search Field
Where this functionality exists in a database, the ‘search field’ option allows researchers to limit the keyword search to title, abstract, reference list, whole document or a combination of fields. Results from a series of pilots indicated that the search ‘anywhere’ in the document option produced more hits with a lower inclusion percentage than searches conducted on the abstract only or title, abstract and descriptors. For example, police and “procedural justice” generated 136 records when the search field ‘anywhere’ was used in CSA with a 60 percent inclusion. When the ‘abstract’ only field was used with the same terms in CSA, 61 records were returned and the inclusion rate jumped to 90 percent. Consequently, the research team decided to search on the abstract when this option was available.
3.2.3 Database selection
A fundamental objective was to develop a search strategy that could be replicated by other researchers in the future. Consequently, the focus was to utilize electronic databases/resources that could be generally accessed (e.g. not restricted material through an organization's intranet). Additionally, it was considered important to locate ‘grey’ literature or material that is not formally published, such as working papers, unpublished dissertations, and reports (e.g. government, non-government, technical reports etc.). After a review of subscription content to examine areas of content overlap between databases, and functionality (e.g. to explore capacity to search on multiple terms, restrict searches to abstract or similar, download citations to EndNote etc.), the research team decided on eight datasources comprising six electronic databases/resources databases (CSA, Informit, Ingenta Connect, Ovid, Proquest and Web of Knowledge) and two library catalogues (National Police Library and the Cambridge University Library and dependent libraries). The databases are listed below with their corresponding weblink and the sub-databases used in the search strategy. Many of the databases used in our search subscribe to an extensive array of government sites and journal providers – these secondary databases are not all listed here. Below, however, we highlight specific key sites (e.g. NIJ and NCJRS) that are searched within the eight electronic data sources.
CSA
http://www.csa.com: Criminal Justice Abstracts Sociological Abstracts Education Resources Information Centre (ERIC) CSA Social Services Abstracts SAGE Criminology SAGE Sociology SAGE Political Science Informit
http://www.informit.com.au/databases: Australian Federal Police Digest Australian Criminological Database (CINCH) Criminology Ingenta Connect
http://www.ingentaconnect.com/: Informaworld (Taylor and Francis journals Academic Press Elsevier Wiley Interscience (Blackwell Publishing) Proquest
http://www.proquest.com: ProQuest – Dissertations and Theses ProQuest – Psychological Journals ProQuest – Social Science Journals ProQuest – Legal Module Ovid
http://gateway.ovid.com/autologin.html: PsycEXTRA National Criminal Justice Reference Service (NCJRS) National Institute of Justice (NIJ) Home Office Publications PsycINFO Web of Knowledge
www.isiknowledge.com/: Web of Science – Arts and Humanities Citation List Science Citation Index Web of Science – Social Sciences Citation List National Police Library via the National Policing Improvement Agency
www.npia.police.uk/en/8495.htm. Cambridge University Library and Dependent Libraries Catalogue
http://ul-newton.lib.cam.ac.uk/
There was an emphasis on electronic datasources for retrieving information. However, in addition to the databases listed above, the research team also reviewed biographies and/or references from authors who have written influentially on the topic of procedural justice and police legitimacy. Specifically, publication lists and biographies of the following authors were reviewed: Tom Tyler, Kristina Murphy, Lyn Hinds, Stephen Mastrofski, James Hawdon and Justice Tankebe. This method of searching publication lists was used primarily to add additional references that were not retrieved in the general search strategy.
3.2.4 Additional Searches
The research team will check the references of each eligible study to be included in the review to determine if there are other studies of interest that were not retrieved in the original search. Any new literature of interest will be obtained and assessed for eligibility.
3.3 Extracting and coding research for the review
Two research assistants (RAs) will be responsible for interrogating the results of the systematic search in order to locate literature relevant and eligible for the systematic review. Preliminary eligibility characteristics are presented in section 3.1.
The following procedure will apply for extracting and coding data for the review.
Identify potentially relevant material for meta-analysis. There are 794 records identified from the systematic literature search which will provide the starting point for the review. RAs will use filters incorporated in the Excel database to produce an initial list of potentially relevant material. Many documents, for example, were discussion pieces of legitimacy in policing or procedural justice; such material will be helpful for the literature review, but will not be useful for the meta-analysis. Code literature. Two RAs will utilize a detailed coding sheet (see Appendix 1) to assess whether literature is eligible for inclusion into the review. The two RAs will both code the population of eligible studies and independently enter results into the Microsoft Word coding document which has built in macros (drop down menus) to encourage consistency in coding. Coding differences will be resolved in a conference involving the senior analyst. The coding sheet requests the following preliminary information: Eligibility Checklist Criteria for Eligibility (e.g. study includes an intervention and a comparison group). Search information Reference information (e.g. authors, publication type etc.), Where the document meets the criteria for eligibility, RAs will then record the following information: Intervention Information, Unit of Analysis, Research Design & Quality Assessment of Methodology, Outcomes Reported, Outcome Variables, Effect Size/Reports of Statistical Significance, Data, and Conclusions by Authors. Additional searches. The research team will check the reference of eligible studies to determine if there are other studies of interest that were not retrieved in the original search. Any literature of interest will be obtained and assessed for eligibility. Data entry. Once the coding of literature has been completed, RAs will enter data into Comprehensive Meta-Analysis 2.0. Quality Assessment. A random check (10 of the eligible studies) will be conducted to assess the consistency and quality of data entry. This quality of coding assessment will be conducted prior to analysis. Additional Searches. The references of eligible studies will be reviewed for additional studies of interest. Independent assessment of retrieved studies. A list of experts in the field of police legitimacy will be developed. A reference list of research eligible for the meta-analysis will be circulated to this list of experts/authors who have given significant consideration to the concept legitimacy in policing (e.g. Tyler, Murphy, Hinds, Kane, Hawdon, and Tankebe). They will be asked to comment on the reference list and asked to identify any significant research that might have been missed in the search process.
3.4 Recruitment, training and inter-rater reliability
3.4.1 Recruitment
In addition to the senior analyst managing the project, two research assistants (RAs) will be recruited. Minimum requirements will be: Experience in data entry, database management and/or involvement in meta analytic review(s), Completion or near completion of an undergraduate degree, Ability to work on site so that questions can easily be answered, and Complete an inter-rater reliability test.
Recruitment will be targeted towards students in the social sciences working towards their honors and postgraduate students, preferably with a background in data analysis or statistics.
3.4.2 Training, coding and test of inter-rater reliability
Training, double coding and testing inter-rater reliability (IRR) will be critical to ensure consistent assessment and coding of research material.
The training process will be facilitated by an initial meeting in which the senior analyst will provide an overview of the project, research objectives, systematic search, search strategy for the review, coding sheet and analysis. In addition, the senior analyst will walk each researcher through the entire process with several example documents so that the research assistants are confident with the procedure.
RAs will also be initially assigned a training task. RAs will be required to determine eligibility of 10 documents drawn from the Excel search by completing the coding sheet for each document. Where eligible, RAs will also need to assess whether any data is missing so that analysis can be conducted. The senior analyst will review the RA's assessment of the 10 documents and provide feedback on any discrepancies in determination of eligibility and/or coding. Following this training exercise, an IRR test will be performed on a further 10 documents drawn from the Excel search results.
All documents will be double coded and coding differences will be resolved in a conference involving the senior analyst.
3.5 Statistical procedures and conventions for review
3.5.1 Test statistic of experimental effect
In this study, Hedges' g will be employed to represent the relative magnitude of the experimental effect. We use Hedges' g as the effect size because most of the outcome measures of interest to this review are continuous in nature and Hedges' g is well suited to comparison group designs with continuous dependent variables.
We will, when possible, use means and standard deviations to calculate the magnitude of experimental effect. When means and standard deviations are not reported, however, effect sizes will be calculated from test statistics (e.g. t-tests, F-ratio, frequencies, odds ratios and X 2 tests for 2 × 2 contingency tables) using formulas outlined by Lipsey and Wilson (2000, pp. 199–202). In the event that studies only report adjusted results (i.e. regression coefficients or analysis of covariance), advice will be sought from experienced members of the Campbell Collaboration on methods for handling these effect sizes.
If there are binary and continuous measures of the same outcomes (within or across studies), we will use procedures recommended by Sanchez-Meca and colleagues to convert odds ratios (OR) to Hedges' g. We will also inspect the data to ensure there is no clear evidence of skewness in the distribution.
3.5.2 Meta-analysis
Data synthesis will be conducted using a specialized statistical meta-analysis software package (e.g. Comprehensive Meta-Analysis). We will be employing a random effects model for pooling intervention effects. An assumption is made that there are unexplained sources of heterogeneity across studies. Based on this assumption, the assumptions of a fixed effects model are impossible. That is, the sample of studies experimental effects provides estimates of a single population effect size (Lipsey and Wilson, 2000). Additionally, we will conduct separate meta-analyses for each outcome measure, including direct (legitimacy, procedural justice, cooperation, compliance, satisfaction) and indirect (reoffending, crime, disorder) outcomes of policing interventions.
Where studies provide several indicators of the same construct, we will use Hedges and Olkin's (1985) procedure to account for the possibility of correlated or dependent multiple effect sizes. To carry out the procedure, we will first combine effect sizes within each study to create a single standard measure for each study, and then combine these standard measures for an overall effect size estimate. This method was selected over the Rosenthal and Rubin (1986) procedure as it produces similar results but is more conservative (Marin-Martinez and Sanchez-Meca, 1999)
As well as reporting average effects, we will provide tables (by intervention) with individual effect sizes and forest plots. As part of the coding process, we will record whether an outcome is measured relative to individuals, groups, or larger units of analysis. For the purposes of this meta analysis, we will not combine studies that have reported data gathered on individuals with studies that make comparisons at larger units of analysis (e.g. communities).
It is possible that a study could include a combination of interventions directed at offenders, victims and/or third parties. In the event that a study reports more than one modality as part of the intervention, we will code each component of the intervention, identify the predominant intervention (if there is one) and conduct further sub-group analyses on mutually exclusive categories that clearly depict the nature of the intervention on the different units of analysis (i.e. offenders, victims or third parties).
3.5.3 Assessment of heterogeneity
We will assess heterogeneity in the outcome measures using the Q-statistic for each analysis. Additionally, we employ the I2 statistic (Higgins, 2002) as an additional, albeit related, method of assessing heterogeneity. An I2 value greater than 50% suggests moderate heterogeneity even if the Q statistic is not statistically significant.
3.5.4 Moderator Analysis
We will examine possible moderators of policing legitimacy. These are likely to include type of I study design (experimental vs. quasi experimental), type of participants (e.g. citizens, victims, offenders, bar managers, community groups), baseline characteristics (e.g. age, gender, race/ethnicity), intervention type (e.g. police-citizen encounters post incident, police-citizen interactions during routine patrols or during community gatherings, random breath stops, targeted traffic stops), duration of the intervention, setting type, country in which the intervention occurred, comparison condition, duration of follow-up, and type of outcome measure.
We will analyze these potential moderating factors using an analog to the Analysis Of Variance (ANOVA) (categorical moderators) and weighted least squares regression (random effects) when appropriate. The analog to the ANOVA is a method of testing the ability of a single categorical value (e.g. intervention type) to explain excess variability in a distribution of effect sizes (Lipsey & Wilson, 2000). We will use a specialized statistical meta-analysis software package (e.g. Comprehensive Meta-Analysis) to run all moderator analyses. Forest plots will be used to graphically display these moderator effects.
3.5.5 Missing data
In the event that a study failed to display suitable data for calculation of effect sizes (e.g., means and standard deviations, valid Ns), we will endeavor to contact the author(s) of the primary studies in order to obtain the missing information. Data on excluded subgroups (e.g., program drop-outs) will also be sought at the time of enquiry.
3.5.6 Sensitivity analyses
A sensitivity analysis will be conducted to test the robustness of the conclusions with respect to the quality of the available data and approaches to the analysis. We will begin by reanalyzing data and excluding studies that have poor quality indicators (e.g. lack of controls for baseline indicators). We will then compare our initial analysis with the post/sensitivity analysis to assess the impact of these exclusions. Additionally, as proposed by Sutton, Duval, Tweedie, Abrams and Jones (2000), we will assess the vulnerability of publication and small sample bias. We will achieve this by inspecting funnel plots, and utilising ‘trim and fill’ methods that estimate intervention effect by adjusting for the number and outcomes of missing studies.
3.5.7 How study quality will be assessed
We recognize that the assessment of study quality can incorporate both objective and subjective elements. Therefore, we will assess the quality of studies in terms of their respective research design, sample bias, equivalency between groups, attrition bias, integrity of intervention delivery, integrity of maintaining differences between the treatment and control conditions, level of monitoring the treatment delivery, research standards adhered to in terms of gathering outcome data, whether or not the analysis is conducted on “intention to treat” or actual evidence of treatment, whether or not mistakes in randomization occurred and how the mistakes were corrected (if at all), consistency of intervention periods and follow up/post intervention time frames both within and between experimental and comparison groups. We will create an interval level summed score of study quality from answers to the questions in Section VII of the coding sheet. This is interpreted as the lower the score the higher the quality of design.
3.5.8 Treatment of qualitative research
Qualitative studies will not be included in the current study. We will, however, acknowledge seminal pieces of research in our discussion.
4. Timeframe
Table 3 presents a preliminary work action plan for the legitimacy in policing systematic review.
Time action plan for Legitimacy and Policing Review, September 09 to June 2010
5. Plans for Updating the Review
Professor Lorraine Mazerolle, Dr Sarah Bennett and Dr Matthew Manning will be responsible for updating this review. We anticipate that this review will be updated as new evidence is collected and subsequent empirical papers written.
Footnotes
6. Acknowledgments
We would like to thank the NPIA for their positive feedback on the initial systematic search.
7. Statement Concerning Conflict of Interest
Professor Lorraine Mazerolle, Dr Matthew Manning, and Dr Sarah Bennett have no financial interest in any of the existing research being conducted in this area of research (legitimacy in policing). The authors have no vested interest in the treatments that are the subject of this review or in the outcome of the review. The authors also do not have any incentives to represent findings in a biased manner and strive to avoid any potential conflict.