PROTOCOL: Mapping the scientific knowledge and approaches to defining and measuring hate crime,hate speech,and hate incidents

Types of studies

For both objectives, this review will include the following types of documents published after 1990: (1)

academic literature in the social and psychological sciences (including criminology, sociology, political science, law and psychology, computer science and sub-fields like regional studies, religious studies and peace and conflict studies);

Additionally, this review will include current legislation, including hate crime, hate speech, and hate incidents found in federal and state statutes, including criminal and civil law statutes, and international agreements (e.g., UN; Council of Europe; American Conventions, etc.). Reported case law will not be reviewed because it is a large field that would warrant a separate project with a different approach.

Population

We focus on Canada and a sample of countries in different regions that are comparable in terms of democratic institutions, socio-political context and legislative approach to hate crime, hate speech and hate incidents.

Eligible countries include, in addition to Canada, the United States and the United Kingdom, which represent two key and different approaches to hate crime data collection globally, with the United States generally requiring a crime to present objective indicators of bias to be counted as a hate crime, and the United Kingdom accepting third party perceptions as a valid hate crime indicator at the policing stage (not at the prosecutors or court stage). We include Ireland, Germany, France, Italy and Spain, which represent a heterogeneous sample of European countries in terms of hate crime and hate speech legislation, different socio-political contexts and approaches to hate crime and hate speech data collection, as identified by recent reports by the European Union Agency for Fundamental Rights and the European Parliament. For example, Germany and Italy have unique hate speech regulations to target hate speech propaganda and hate crime associated with the Nazi and Fascist ideologies. Ireland and Spain include third party perceptions as bias indicator, while France and Germany do not, and Italy does not have a list of bias indicators. Germany and France require by law to record only specific forms of hate crime, Spain and Ireland incorporated hate crime in the general crime recording system, but in Spain recording is not compulsory while in Ireland it is, and in Italy hate crime is not incorporated in the general recording system although hate crime data is collected by law enforcement authorities. All five countries have hate crime and hate speech regulations in criminal law, civil law, media law and press-self regulation, as well as watch-group organizations collecting hate incidents reports, which make them relevant case studies. We include Australia, which is a comparable context with Canada in terms of legal, political and democratic structures (e.g., federalism, bicameral parliaments, common law legal system—although Canada has one criminal law, whereas in Australia there are different statutes in each state). Moreover, Canada and Australia have striking similarities in terms of geographical and demographic distribution (i.e., population concentration in urban areas), economic and social indicators (e.g., GDP per capita, unemployment), cultural and linguistic indicators, migrant population and attitudes to diversity, social and political issues (such as relationships between First Nations and European settlers), all of which constitute significant predictors of intergroup conflict that shape the types and forms of hate crime, hate incidents and hate speech taking place in Australia and Canada.

We include New Zealand, which after the Christchurch attack significantly increased its efforts in policy, practice and research on hate behaviors, as demonstrated by the Christchurch call to eliminate terrorist and violent extremist content online, and by the Royal Commission of Inquiry on the Christchurch attack, and its numerous recommendations relevant to regulating hate crime, hate speech and hate incidents.

The review will take into consideration academic articles in English, French, German, Italian and Spanish languages, and will review relevant contemporary legal statutes, policy and programming in North America (Canada and United States), Europe (Germany, France, United Kingdom, Ireland, Italy and Spain), Australia, and New Zealand. We focus on these languages because we aim to capture relevant documents in the countries that are included in the review.

Phenomenon of interest

This review focuses specifically on capturing both general definitions of hate crime, hate speech and hate incidents; and definitions of surrogate terms capturing hate directed towards identities based on the following perceived characteristics that appear in Canada's legislation: 1.

Racial and ethnic identity (e.g., racism, xenophobia, sinophobia);

Our review will also pay particular attention to how protected characteristics are defined and captured, and how other characteristics absent from lists are captured by existing definitions and measurement tools. In this review, we focus exclusively on definitions of malicious behaviors motivated by (or demonstrating) prejudice, or by a worldview or ideology that dehumanizes the target group and justifies aggression and violence, and how these definitions are operationalized to measure these behaviors. Behaviors that will be included in this review can be criminal behaviors (e.g., murders, violence against people and properties, and forms of hate speech regulated by the criminal code) and noncriminal behaviors (e.g., the so-called “protected hate speech” in the United States or other jurisdictions). Both on- and offline behaviors are included in our scope. Discrimination (e.g., discrimination in workplace, sale and supply of services), public order offences, sex crimes and harassment are excluded because they are different fields of literature, which we believe would need a separate independent review, especially in relation to the legal literature.

Context

Our review focuses on definitions and measurement “hate crime,” “hate speech,” “hate incidents” or any surrogate terms that are used to capture these concepts. Much of the literature focusing on “hate” and surrogate terms (such as anti-Semitism, religious hate, racial hate, etc.) uses these terms to define both behaviors and attitudes (e.g., negative attitudes to a racial or religious group). This review will only focus on definitions and measurement tools that focus on capturing behaviors including speech.

Some of the literature focusing on concepts like homophobia, Islamophobia or anti-Semitism, focuses on negative attitudes to out-groups instead of behaviors (see e.g., Huynh et al., 2020; Uenal et al., 2020). This area of literature focusing on attitudes, although using concepts that might be relevant for this review (such as homophobia, Islamophobia, and anti-Semitism), will be excluded because our main focus are definitions and measurements of behaviors, not cognitive activities. As we only focus on measurement tools capturing behaviors only (not attitudes), we will not take into consideration any psychometric scale instrument, or any tool aiming to capture attitudes related to hate or surrogate terms (e.g., racism, homophobia, anti-Semitism, etc.).

Electronic searches

We developed our search strategy based on Kugley et al. (2016) methods guide on search strategies, discussions within our team, with Campbell Collaboration, as well as with input from members of our External Advisory Board and university librarians. We will use the same search strategies to identify literature relevant to objective 1 and objective 2. However, our search strategies to identify different types of documents (i.e., academic and gray literature, federal and state statutes; see above) will be tailored to optimize the retrieval of potentially relevant documents for each type and, where practicable, each country. In selecting the databases for electronic searches, we aim to provide a broad coverage by including multi-disciplinary databases and citation indexes as well as subject-specific databases to specifically identify potentially relevant documents in French, German, Spanish and Italian (hereafter: databases). For the purpose of our review, we do not consider it relevant to include trials registers in our search strategy because our review is not a review of interventions. Please find a full list of databases in Table 2 and the specific search strategies in Supporting Information Annex 3.

For electronic searches, we have devised a search strategy that combines search terms grouped into three categories: 1.

Attribute-specific search terms: these search terms correspond to groups that commonly become the target of hate crimes, hate incidents or hate speech;

These search terms capture our broad focus on hate crime, hate incident and hate speech as well as surrogate terms. We decided against search terms that would reflect our research objectives, such as “definition” or “measurement,” because we would expect that doing so may exclude potentially relevant documents from search results, similarly to Campbell et al. (2014) observations. As we do not have specific eligibility criteria around study design, we do not consider it necessary to include study design as a criterion in our search strategy. Please see Table 1.

For each electronic search, we will adapt our search strategy depending on the capacities and specificities of the search function of the database in question. We divided the databases into two groups: the first group of databases allows us to combine all our search terms with Boolean and proximity operators into a single search string; the second group of databases has limited search functionalities which do not allow us to use our comprehensive search string. Please find a list of databases grouped into the two categories in Table 2.

Each search strategy for databases falling into group 1 includes the use of placeholders and wildcards as well as loose/approximate and exact phrases. Additionally, we will use search limits and filters, where possible, to filter out material published before 1990, as this material is beyond the scope of our review. We focus on materials published after 1990 because it is the point at which both policy responses and academic attention to hate motivated behaviors began in earnest. The Hate Crime Statistics Act (HCSA) of 1990 set into motion the structure and mechanisms for identifying and collecting hate crime data in the United States, and corresponded with a surge in academic publications on hate crime globally. For this reason, a number of hate crime data collection efforts, like Gruenewald's Bias Homicide Database (BHDB), begin in 1990.

The search terms within each column will be connected with “OR” operators; attribute-specific and behavior-specific search terms will be connected with a proximity operator where available, such as “W/n” in Scopus, or otherwise with an “AND” operator; and behavior-specific and country-specific search terms will be connected with an “AND” operator. We have chosen to use a proximity operator, where available, to connect attribute- and behavior-specific search terms to improve the precision of our searches.

For databases falling into group 2, our search strategy will be limited to eight keywords: 1.

“hate crime”

We chose these keywords because they are the most commonly used in the literature published in English, and because we expect them to have equivalents in the other languages considered in our review in the form of direct translations. For searches carried out in Italian, Spanish, French and German, we will translate those keywords into the respective languages. Where possible, we will combine these search terms in a single search combining them with the Boolean operator OR. Where this is not possible, we will carry out individual searches for each term separately. Please find the individual search strategies for each database in groups 1 and 2 in Supporting Information Annex 3. For the purpose of our review, we do not consider the use of search filters, such as the Cochrane Highly Sensitive Search Strategies for identifying randomized trials in MEDLINE, appropriate.

We will conduct one round of forward citation searching in Google Scholar of the academic studies that will be included in the review.

For each electronic search, we will document the following details to ensure reproducibility of our search strategy:

Date of the search (dd/mm/yyyy)

Searching other resources

To complement electronic searches, we will use complementary search strategies to minimize the risk of bias. These search strategies include: (1)

Hand-searching of reference lists of any systematic reviews on hate crimes, hate incidents, hate speech, or a surrogate of these terms identified throughout the review process.

To ensure reproducibility of our search strategy, for each search carried out through means other than electronic searches, we will record:

Date the search (dd/mm/yyyy)

To identify current federal and state statutes, we will draw on key gray literature providing the full list of current legislation regulating hate crime, hate speech and hate incidents in a given country. An example is the report of the Royal Commission of Inquiry into the Terrorist Attack on Christchurch Mosques on March 15, 2019, which identifies and lists all “Hate speech and hate crime related legislation.” For each country, we will develop a list with relevant current legislation based on this strategy. Prior to coding, we will send these lists to academic experts of hate crime and hate speech legislation and legislation practitioners' from each country of focus for review.

Description of methods used in primary research

Review Objective 1: Definitions

Inclusion and exclusion criteria for objectives 1 and 2 will be identical for title/abstract screening. Specific additional criteria will be added for full-text screening. The inclusion criteria for title/abstract screening are provided in Table 3.

Documents to be included in our review in relation to objective 1 need to meet the following inclusion criteria for full-text screening (Table 4).

To avoid confusion about whether any document is included or excluded in relation to objective 1, coders will be asked to assign a code upon making their decision that will label whether a document was included or excluded in relation to objective 1, objective 2, or both.

Review Objective 2: Measurement tools

The procedure for title/abstract screening for documents in relation to objective 2 will be identical to that outlined above for documents in relation to objective 1. However, consistent with the inclusion criteria outlined above, the inclusion criteria for full-text screening will differ. These criteria are provided in Table 5.

To avoid confusion about whether any document is included or excluded in relation to Aim objective, coders will be asked to assign a code upon making their decision that will label whether a document was included or excluded in relation to objective 1, objective 2, or both.

Our team will include documents in English, French, Italian, Spanish, and German. Because these are the languages spoken in the countries that we focus on, and will therefore capture virtually all relevant documents. Documents in other languages will be excluded.

Review Objective 1: Definitions

To achieve objective 1, for each document we will extract information about the components of the definition of hate crime, hate speech, hate incidents and surrogate terms. Components include: (1) how the motivation is named (e.g., racist, Anti-Semitic, hate, etc.); (2) how the nature of the behavior is named (e.g., crime, incident, speech, act, etc.); (3) how the target is described (e.g., a person, a property, a person and property, etc.); (4) whether perception is a bias indicator; and (5) the protected characteristics (if any). Additionally, we will code categorical variables identifying whether:

the definition is adopted by any governmental or nongovernmental organization or linked to any statute or legislation;

Review Objective 2: Measurement tools

To achieve objective 2, for each document we will extract information about: the type of measurement tool, defined as a vehicle or an aid to collect information and data (e.g., an online module to collect data about hate incidents, or an automated text analysis algorithm, or a survey); the metric s used to measure hate, defined as parameters (measures) or indices used for measurement, comparison or tracking performance (e.g., bias indicators); and the methods used to measure hate, defined as the process and approach involved in a systematic inquiry of hate crimes, hate incidents, hate speech or surrogate terms, and generally refer to study design or application of an analytical method to this topic.

We will code categorical variables looking at information about the target identities that the measurement tool encompasses, the bias indicators used (if any), whether it is adopted by any government or nongovernment organizations, the variables collected about the incident, the victims, and the offender, and the indicators used to assess the severity of the crime. If there is any analysis of quantitative data collected using the measurement tool, we will collect—when present—information about context and sample, data collection methods, data analysis, and any reporting about feasibility, efficacy, reliability or validity of the instrument. We will also collect information about whether the instrument is at concept development, pilot or implementation stage.

Based on the OMERACT tool to assess quality of measurement tools (Beaton et al., 2019), we will collect information relative to the domains of truth (i.e., does the instrument cover all the protected characteristics present in the relevant legislation), feasibility (i.e., is the instrument accessible? Is it available in multiple languages?) and discrimination (is there stability in situations of no change? Does the instrument detect score change in situation of real change? How well does the instrument distinguish between groups?). We expect that, while information about truth and feasibility might be available for most instruments, information about discrimination might not.

Review Objective 1: Definitions

To assess the quality of definitions, we will conduct post-hoc analyses of the definitions against the following criteria: 1)

Does the definition contain unambiguous concepts (Ritzer, 1991)? For example, we will look at whether the hate motivation is clearly defined (e.g., is the threshold between hateful and non-hateful behavior clearly defined?) and at whether there is a clear understanding of how to identify the hate motivation (e.g., what are the bias indicators?)

We will operationalize these criteria as three separate dimensions to assess the quality of definitions by attributing a score from 1 (strongly disagree), 2 (disagree), 3 (agree), 4 (strongly agree) to each.

Review Objective 2: Measurement tools

To assess the quality of measurement tools, we use the approach described by Sparling et al. (2019) in their gap map of tools, metrics and methods in the field of food systems and agriculture-nutrition. First, we will look at whether instruments are at different stages of development, which include: (1) concept development; (2) pilot; and (3) wide-spread application. Second, we will look at whether the documents report any information about (1) feasibility, efficacy or internal validity of the instrument, and (2) validity or reliability of the instrument. Third, we will consider how much information about the hate incident is included in the instrument (e.g., about the offender, victim, incident, bystanders, etc.) and whether the instrument is adopted by any governmental or nongovernmental organization. Finally, we will collect information relative to the domains of truth, feasibility and discrimination (Beaton et al., 2019). Where present, we will report reliability and validity of measurement tools included in this review, using the template in Wells et al. (2009).

Review Objective 1: Definitions

Given the qualitative textual nature of the information extracted (see Data extraction and management section) we will attribute categorical codes to the text extracted. The unit of analysis will be the definitions. For each definition, we will code components (e.g., attribute, behavior, protected characteristics, etc.) and other additional qualitative categories (e.g., whether the definition is adopted by any government or nongovernment organization, whether it allows for degrees of motivation, etc.).

Specifically, to answer each research question, the following information will be examined: ○

How are hate crimes, hate speech and hate incidents defined in the academic, legal, policy, and programming literature? We will look at definition components and other qualitative categories, and document type (whether academic, gray or legal, and—if academic—which discipline it is situated in).

Review Objective 2: Measurement tools

The majority of the information that we will collect about the measurement tools will be qualitative and textual (see Data extraction and management section), and we will attribute categorical codes to the text extracted. For each instrument, we will code tools, metrics and methods, stage of development and additional qualitative information (e.g., whether the instrument is adopted by any government or nongovernment organization).

Additionally, when reported, we will extract additional information about sample, context, and any available information about feasibility, efficacy, reliability and validity of the instrument. Specifically, to answer each research question of the review we will look at how are definitions operationalized to measure hate crimes, hate speech, and hate incidents? We will code tools, metrics and methods. We will also look at how valid and reliable are these measures? We will look at stage of development and additional qualitative information, as well as any available information about feasibility, efficacy, reliability and validity of the instrument, when reported.

Review Objective 1: Definitions

The unit of analysis will be the definitions. We plan to present univariate analysis of the categorical data, as well as bivariate analysis (e.g., cross-tabulations, bar charts) of definition components and additional qualitative categories with document type, academic field, country, language, year, and authors' affiliation (whether a university, a government or nongovernment organization). We will also present a narrative report describing definition components and additional qualitative categories, which will be used to provide an assessment of the definitions (see section Assessing the methodological limitations of included studies).

The full list of the analyses that will be performed is available in Supporting Information Annex 1.

Review Objective 2: Measurement tools

The unit of analysis will be the instruments to measure hate speech, hate incidents, hate crimes or surrogate terms. We plan to present univariate analysis of the data that will be collected and coded as part of this review, which includes categorical data, as well as bivariate analysis (e.g., cross-tabulations, bar charts) of instruments, metrics, tools, methods and additional qualitative categories with document type, academic field, country, language, year, and authors' affiliation (whether a university, a government or nongovernment organization). We will also present a narrative report describing instrument's characteristics, stage and additional qualitative categories, which will be used to provide an assessment of the definitions, together with any available information about the instrument's feasibility, efficacy, reliability and validity (see section Assessing the methodological limitations of included studies).

The full list of the analyses that will be performed is available in Supporting Information Annex 1.

Review author reflexivity

The team has a variety of disciplinary backgrounds that reflect different positions and approaches in the field of hate crime studies, including qualitative and theoretical sociology (Perry), quantitative criminology (Chermak and Freilich), mixed methods political psychology (Vergani) and mixed methods online research (Scrivens). The team is well versed in relevant theory and in the study of subfields relevant to this review, such as Islamophobia, anti-Semitism, terrorism, and violent extremism. This review's aim is to map—not summarize or meta-analyze—the existing definitions and measurement tools across different disciplines. The review team has been maintaining a reflexive position throughout all the stages of the review process, and decisions have been discussed critically and regularly among the team members with regular debriefing sessions to support with decision-making and coding. Based on the team's collective and individual experiences, it is anticipated that the findings will reveal a combination of approaches and disciplinary contributions across the whole spectrum of social and psychological sciences relevant to the study of hate behaviors. The team will remain mindful of conscious and unconscious presuppositions and support each other to minimize the risk of these skewing our analysis or the interpretation of our findings. The research team is composed of researchers who greatly contributed to the literature under investigation in this review. For this reason, we will make sure that the authors of a document will not review and assess it.