PROTOCOL: Strategies Related to the Prevention,Detection,Management and Response to Terrorism: A Campbell Systematic Review

INTRODUCTION

Social scientists have often responded to events, crises and public concerns through research. In crime and justice, for example, the increase in crime rates and imprisonment in the 1970s and 80s (see Blumstein and Wallman, 2000) and the debate over the effectiveness of rehabilitation (Lipton, Martinson and Wilks, 1975; MacKenzie, 1997) led to a large amount of research on imprisonment, sentencing schemes and alternative corrections (Blumstein, 1988). Similarly, heightened awareness of domestic violence and outrage towards police responses to these incidents resulted in increased attention to the issue by criminologists and later to policy responses reflecting research (Sherman, 1992). This same policy focus and attempts to influence current policies can be seen in research on high-profile concerns about gangs and drugs (Lane and Meeker, 2000; Laniel, 1999; Reuter, 2001) or victimization and fear of crime (Farrall, Bannister, Ditton and Gilchrist, 1997; Sebba, 2001).

Many of these responses to crises in criminal justice have centered around two related concerns: implementation and development of effective policy to counteract the crises, or the evaluation of the effectiveness of current policies. Both have become generally encompassed under “evidence-based” crime prevention approaches (see MacKenzie, 2000; Sherman et al., 1997; Sherman, Farrington, Welsh and MacKenzie, 2002). Evidence-based crime prevention has been facilitated by a vast amount of research on “what works” in crime prevention (see Lum and Yang, 2004; Sherman et al., 2002), meta-analyses and systematic reviews that attempt to organize, make sense of, and critique research, and the establishment of the Campbell Collaboration (and specifically its Crime and Justice Coordinating Group) which encourages these endeavors. This movement has served important functions in improving both the quality of research as well as properly informed practice in areas such as police patrol, corrections-based schemes, juvenile diversion, treatment of sex offenders, and sentencing practices. Evidence-based perspectives can also provide a moderating effect on policy-responses to crises that have become more influenced by “moral panics” (Cohen, 1972) than by reason or facts. For example, the use of boot camps to control the juvenile “superpredator” has been widely discounted by much research evaluating the effective ness of these programs (see Gover, MacKenzie and Styve, 2000) counter-balancing the popular movement towards utilizing these approaches in controlling juvenile crime.

The current public concern over terrorism also presents similar controversies and research challenges. While counter-terrorism and terrorism prevention programs have blossomed considerably since September 11^th, there has been little evaluation of either the effectiveness of the wide array of anti-terrorism policies or the effects of the responses themselves. Given the vast array of different strategies designed to “fight terrorism”, we know very little about which of these might actually be effective and appropriate. Millions of dollars are being expended on potentially fruitless programs, or worse, harmful responses that may promote or benefit terrorists or harm populations generally. As with boot camps or police strategies to deal with domestic violence, evaluating the effectiveness and effects of strategies geared toward countering terrorism is part of ensuring that policy is informed, evidence-based and effective.

Yet, evaluating the effectiveness of anti-terrorism strategies remains a challenging and controversial task for a variety of reasons. Unlike crime, terrorism incidents have not been easily defined and therefore, strategies may be difficult to pinpoint. If we include all political violence as “terrorism” (including ethnic conflict) then we might also have to include strategies such as war, government suppression, laws that manipulate boundaries or peace treaties as possible strategies to evaluate. Even more problematic may be political goals and biases that underlie terrorism research. For example, one might argue that anti-terrorism “effectiveness” is measured when there is political suppression of opposition parties, which may not be valued in democratic societies. Other strategies may seem effective but may be in gross violation of human rights norms, such as the inhumane treatment of prisoners or other corrective policies that may not be acceptable in many societies.

Additionally, like much of crime, political violence involves multiple points at which interventions may be effective and where outcomes might be measured. For example, anti-terrorism strategies may include prevention and alleviation of early risk factors, situational prevention of actual events, or post-event responses. Furthermore, because of the rare nature of terrorism events, it may also be difficult to determine whether these strategies actually worked. Thus, researchers may be interested in detecting “secondary” effects of prevention strategies such as the reduction of the fear of terrorism or the adequate and efficient response after an event occurs. Other strategies may focus on detecting potential terrorism events or other high risk situations.

The purpose of this review is to organize and comprehend the research on strategies designed to prevent, detect, manage and respond to terrorism and political violence in order to make sense of the focus and quality of this research and to make future recommendations. Despite the aforementioned challenges about research on anti-terrorism strategies, conducting a Campbell systematic review is a relevant pursuit for two reasons. The first is obvious – given the explosion of “anti-terrorism” policies and the millions of dollars spent on them, we need to know what “works”, “doesn't work” or is “promising” (see Sherman et al., 1997) in terms of the most optimal use of resources. This review will function as a mechanism to systematically collect and comprehend the scattered research on terrorism that takes place across multiple disciplines and countries. Secondly, a systematic review may also expose what little we really know about the effects and effectiveness of anti-terrorism strategies and areas of possible future research. This may not only shape research agendas but may also shape more thoughtful policy development as well. Currently, the state of terrorism research, although vast in many respects, does little to critically evaluate the effectiveness and effects of programs designed to combat terrorism and we have already seen harmful effects of policies which we know little about.

A PRELIMINARY REVIEW OF TERRORISM LITERATURE

Before approaching a review of anti-terrorism strategies, a preliminary and general examination of terrorism research can provide a better understanding of the scope of the research. From time to time, a few attempts at reviews of terrorism research have been undertaken (see, e.g. Halkides, 1995; Hoffman, 1992; Miller, 1988; Romano, 1984; Schmid and Jongman, 1988; Silke, 2004), although September 11^th has certainly necessitated the need for an updated review on strategies, specifically. To conduct this preliminary review, we focused on published articles from multiple disciplines. ¹ To include as much research as possible, seventeen separate literary databases were searched in January 2003 for broad and sweeping terms related to terrorism and political violence. ² The universe of literature was multidisciplinary, including subjects related to economics, gender issues, criminal justice, education, geography, the humanities, medicine, psychology, political science, public policy, sociology and the law. We located 14,006 articles from these databases after excluding for duplicate records. ³ For each article located, the author, year, title, citation, abstract and keywords were recorded.

Because of the volume of the initial literature search, publications were first divided according to the review status of their sources and only abstracts from peer-reviewed sources were individually read. Again, it should be emphasized that the filters used for this preliminary review will be re-evaluated when searching only for research on the evaluation of anti-terrorism strategies. The focus on peer-reviewed sources was not only practically necessary due to the volume of the literature, but served as a preliminary filter for understanding the quality and substance of academic research on terrorism and political violence. Many of the articles from non peer-reviewed sources were either news reports, opinion-editorials, advertisements, announcements or bulletins that would not normally be considered “research”. To separate articles from peer-reviewed sources, a list of peer-reviewed journals was generated using peer-review lists compiled by EBSCO, ⁴ OCLC, ⁵ the American Medical Association, ⁶ and the American Association for the Advancement of Science, ⁷ and by investigating individual journal titles if not listed in these databases. Each of the 14,006 articles was then matched with a peer-review status using a relational database program. 6,041 articles could be identified as coming from peer-reviewed sources. Clearly, a large amount of writing regarding terrorism appears in non-peer reviewed journals or in journals with unknown peer-reviewed status.

The first, most unique finding compared to past literature reviews was that September 11^th has had an unprecedented effect on terrorism research. As Figure 1 indicates, among the entire 14,006 journal articles, approximately 54% were published in 2001 and 2002. As many of the journals were not refereed and represented magazines and other opinion outlets, this trend could simply represent an “emotion bias” after the attacks. However, when only examining articles from peer-reviewed journals, the same trend was evident. Neither the Oklahoma City bombing in 1995 nor any other significant terrorist event, including the World Trade Center bombing in 1993, has ever generated this much research interest.

OPEN IN VIEWER

1

Yearly Distribution of Terrorism Publications (as a percentage of total N)

This exponential growth of research reflects not only the response to September 11^th by researchers, but also increases in federal funding for research and development related to terrorism over the fiscal years following the attacks (Guinnessy and Dawson, 2002; Issues in Science and Technology, 2002; Macilwain, 2002; Silke, 2004, generally). Although time will tell whether this substantial increase is a “fad”, this trend is indeed currently significant, pointing to the relevance of terrorism research generally. Furthermore, this increase in research has also been matched by the massive build-up of anti-terrorism efforts and bureaucracy, much of which we have little or no understanding as to their effectiveness or consequences. Thus, not only is interest and concern of terrorism real, but substantive and institutionalized changes have made the effect of September 11^th significant.

A number of questions come to mind with the exponential growth of this research in relation to the goals of a Campbell systematic review. How much of this research is actually “evaluation” research? What types of topics are being evaluated? What is the methodological quality of this research? What conclusions are being made from this research? Does this research reflect the types of policies that are being implemented? To begin to answer these questions, we analyzed only peer-reviewed articles to better estimate the overall state of evaluation research on anti-terrorism strategies.

To ascertain the nature of the methods used to analyze terrorism, abstracts of articles from peer-reviewed sources were read and coded for the type of methodology used. While we recognize that the abstracts might not provide detailed information about the methodology used for each study, the volume of studies did not allow us to individually read every article for this preliminary review. However, we anticipate that this limitation is a minor source of error that does not largely impact our general findings (and for the Campbell review articles will be scrutinized in-full). Three basic categorizations were preliminarily developed to facilitate further identification of articles that may be useful to a Campbell Review: “thought pieces”, “case studies”, and “empirically-based” research. Briefly, thought pieces were classified as publications that were philosophical, theoretical or opinion-based without any indication that a case study or empirical analysis of a topic was undertaken. For example, articles discussing problems with defining terrorism, legal issues surrounding 9–11, historical accounts of a particular conflict, descriptions of a particular emergency response or opinions about the government's policy towards counter-terrorism would be categorized as “thought pieces.” Case studies became a somewhat confusing category (as some case studies could be thought pieces and others empirical analyses), so only those publications in which authors specifically defined their work as a case study were included in this category. These publications tended to push the analysis of terrorism further than thought pieces by including in-depth description and qualitative analysis on a particular incident of terrorism or a particular situation or region.

Publications described as “empirical analyses” were distinguished from thought pieces in that some form of systematic qualitative or quantitative analysis was undertaken on the data gathered. As it was expected that this would be the smallest of categories, a liberal definition of “empirical” was adopted and could include a wide range of methodological strengths, ranging from descriptive trend statistics to randomized controlled experiments. For example, articles which fell under this category included the analysis of terrorist incidents across time, the effects of a particular intervention on the occurrence of terrorism, surveys on the psychological effects of terrorism, attitudes towards the U.S. after September 11th, and multivariate analysis on factors contributing to the development of terrorist groups.

Figure 2 shows the distribution of the type of methodology for all articles in peer-reviewed publications where an abstract was available. Although thought pieces certainly have their own value in furthering the understanding of any topic, the research on terrorism has been bereft of empirical research, a problem which has been endemic in terrorism research since Schmid and Jongman's review in 1988. This problem has not improved after September 11^th; when only examining the research conducted in 2001 and 2002, the proportion of empirical or case studies was similar.

OPEN IN VIEWER

2

Methodological Quality of Terrorism Research

The dearth of empirical analysis in peer-reviewed sources suggests a number of possible concerns. First, practical problems in either qualitatively or quantitatively studying terrorism are often thought to be significant obstacles to overcome. Not only are events thought of as “rare” but units of analysis (individuals or groups who employ terror violence or actual events/incidents of terrorism) may be difficult to locate and study. Especially since much political violence occurs outside of our nation's borders, a call for terrorism research presents similar difficulties that comparative criminological research does generally, including small sample sizes, difficulties in interpretation, and differences in measurements of key variables. More importantly, the scarcity of empirical analysis may also reflect the state of the science of terrorism research, which suggests that very little evaluation research on strategies designed to prevent, detect, manage or respond to terrorism events (or the potential of terrorism events) exists.

We were also interested in a better understanding of what types of subjects were covered in this research (and specifically how much of the general research reflected evaluation studies). To understand the distribution of subjects researched regarding terrorism, a coding schema was developed by initially taking a random sample of 200 articles from the peer-reviewed journals which were then individually examined to develop a preliminary list of topics. This was done so as not to force subjects into pre-defined categories. Then, each of the citations from peer-reviewed sources that had abstracts were individually read and matched to this preliminary list, which was actively modified during this process. Thirty-five general categories initially emerged which were collapsed into seventeen categories shown in Table 1. Table 1 also reports the distribution of these categorizations for just those articles which were deemed “empirical” studies.

A number of interesting findings emerged. Generally, issues related to weapons of mass destruction represented the largest proportion of articles (18.9%) followed by articles which focused on a specific issue, such as the Israel-Palestinian conflict, the IRA, Al Qaeda, or September 11^th (if it could not be categorized elsewhere). Additionally, political responses to terrorism and the “sociology” of terrorism (causes, motivations, explanations, definitions) clearly dominate the research agenda of terrorism studies. These emphases most likely represent the political science bias of the study of terrorism and speak to some of the priorities that have been formulated.

Although these matters are indeed salient, surprisingly absent from the literature were articles regarding subjects that are more likely to affect individuals, their fear, and the ramifications of anti-terrorism efforts. Given the rarity of terrorism events in American society, we are more likely to be affected by anti-terrorism responses than events themselves, concerns that are also under-researched. This may include, for example, violations of civil rights, increases in law enforcement powers with questionable constitutionality, or post-traumatic psychological effects on victims. Little is known generally about the effectiveness of law enforcement and other non-political responses to terrorist events. Although legal issues have been more researched, they still represent only a small percentage of terrorism-related research. This is also true for issues related to coping with terrorism events and the general victimology of terrorism. Additionally, when thinking about future threats, such correlations between terrorism and religion, socio-economic factors and political responses have yet to be made.

When examining those articles preliminarily deemed to be “empirical”, the findings are both encouraging and discouraging. While a quarter of the empirical work has been conducted on victimology, a subject relevant to our search for evaluation studies, and while it is anticipated that within the literature on weapons of mass destruction there may be empirical tests, we also found very little empirical work done on any type of response to terrorism or on the legal aspects of terrorism. The vast majority of empirically-based studies do not seem to evaluate the effects of anti-terrorism strategies. We expect that many evaluation studies will focus on the effects of treatment provided to victims of terrorism, the effects of anti-biological weapons programs and security related to airports.

CRITERIA FOR INCLUSION AND EXCLUSION OF STUDIES

Terrorism research presents a number of difficulties for such a review. As already shown, what might be considered as an “anti-terrorism strategy” can include a wide variety of subjects, including political, social, legal, law enforcement, economic, preventative, reactive or “after-care” responses. Limiting our search to only those responses which are law-related, for example, may exclude those responses that focus on risk factors. Certainly in other crime-related reviews of evaluations of anti-crime programs, such as the Maryland Report (see Sherman et al., 1997; Sherman et al., 2002), programs that may not be within a specified criminal justice institution (e.g., the police, courts or corrections) are still crime-relevant (such as those strategies implemented in schools or by businesses or communities). Thus, we feel that a broad approach is warranted in creating the initial base of literature to examine. We propose to include evaluations of any strategy that involves the prevention, detection, management or response to terrorism events and incidents. While this may seem to be a difficult and disadvantaging task, our preliminary analysis shows that only 3% of our peer reviewed literature is “empirically based”, suggesting that the there may only be a couple of hundred empirically-based studies to begin with. While we will have to extend our search to 2003 and 2004 as well as look at all publication mediums (books, non-peer review materials, unpublished papers, government reports), we anticipate that the number of evaluations will be manageable despite our broad inclusion criteria.

However, as already discussed, one main obstacle when seeking out evaluations of terrorism-related strategies is what we decide as the definition of “terrorism”. For our preliminary search, the goal is to be as inclusive as possible so as to not miss potential articles that may be used in the review. Thus, we will initially consider any evaluation research on terrorism which satisfies either of the following two criteria: 1)

Any evaluation in which the outcome measured follows the official definition as set forth by the United States Department of State, which has described terrorism since 1983 (Title 22 of the United States Code, Section 2656f[d]) as premeditated, politically motivated violence perpetrated against noncombatant targets by sub-national groups or clandestine agents, usually intended to influence an audience; or

While our intention of (1) is to provide some basic structure for the search, we do not wish to imply that only an American definition or perspective will be used (hence the 2^nd possibility).

Following from these broad criteria, our search will encompass the aforementioned four categories of anti-terrorism strategies (although if another relevant category arises, we will be open to creating new categories): preventative, detection-oriented, managerial and response-oriented. We take this approach as we recognize that there are a number of points before, during and after a terrorist event in which strategies may be implemented (see Schmid, 1983; Walter, 1969). Prevention strategies are strategies designed to deter future events and might include airport security measures or target hardening evaluations. Studies on the preventative effects of legal measures (such as the Patriot Act) would also be included, for example. Research on detection strategies may include interventions related to, for example, airport security, but also might include evaluations of measures used to look into containers, border-related strategies, immigration policy or other tactics used to detect people, places, things or situations involved in terror-related activity. “Managing” strategies point to tactics “after the fact” and can include evaluations of the government's threat level mechanism, new hospital procedures designed to address issues of health security, or programs put in place to help people who have been impacted by a terrorist event, psychologically or otherwise (for example, evaluations of services intended to reduce general fear or victim post traumatic stress). Evaluations looking at response strategies might focus on law enforcement responses to continual suicide bombing or the effects of economic sanctions on terrorism. It is anticipated that some of these categorizations will overlap. This Campbell review will also provide a unique opportunity for researchers to suggest a number of places where evaluation research may be needed.

Because it is not expected that randomized or even quasi-experiments will be regularly present in the empirical research on terrorism, we intend to use a lower threshold than normally considered in a Campbell review to initially identify any possible study that might satisfy the inclusion criteria above. Thus, the identification of any study based on “empirical” data will initially be considered. However, because this review will also have policy implications, we intend to specify our sample, within this group of empirical studies, to those studies that would satisfy moderate methodological requirements as described below. While we will use the Maryland Report Scientific Methods Score (see Sherman et al., 1997) of “3” as a general guideline, each study will be evaluated on its own. However, when reporting our results, we will remain extremely sensitive to the way we present our findings, emphasizing continuously that lower methodological rigor may lead to incorrect conclusions. Studies that will be excluded from this review will be “thought pieces” which have no empirical analysis of the effects or effectiveness of anti-terrorism strategies or that do not have any outcome measured for programs discussed.

SEARCH STRATEGY

The search of these studies will build upon the comprehensive review as described above but will be broadened in a number of ways. All “empirical-based” research will be identified from the review above, which was based on a comprehensive search of multiple library datasets (as previously listed in Footnote 2). In addition to the data already collected, a second search of empirical literature specifically up to the current date will be undertaken, to assure that no study may have been missed since the last search which ended in 2002. Again, our initial search is intended to be as broad as possible as we do not anticipate finding a large amount of evaluation literature on anti-terrorism strategies. Furthermore, we have also compiled a comprehensive search of organizations and data-sources of terrorism research (see Appendices A and B) ⁹ which will be searched for evaluations conducted that may not be included in academic, peer reviewed sources. Additionally, Alex Piquero, of the University of Florida, Gainesville and Gary LaFree, of the University of Maryland, College Park, have also conducted a review of hijacking which we will seek to add to our study with their permission. Other terrorism literature will also be searched, including evaluations from books and non-article mediums, government publications and unpublished material. We will also seek the assistance of a number of international scholars, who may be of assistance in locating research not present in these databases or that may not initially be in the English language. Finally, we will conduct an internet search to capture any other evaluations that may have been missed and that satisfy our me thodological criteria.

In accordance with our broad definitions as mentioned above, we will conduct a widespread search of multiple terms related to terrorism and their derivatives (which may be added to or adjusted as required). Some examples of these terms include : “terrorism”, “political violence”, “political crime”, “emergency response”, “riots”, “anti-Semitic”, “white supremacist”, “black supremacist”, “militia groups”, “bio -terrorism”, “weapons of mass destruction”, “homeland security”, “national security”, “ethnic violence”, “bombing”, “suicide bombing”, “aircraft hijacking and skyjacking” as well as significant event labels such as “September 11^th“, “Oklahoma City Bombing” or “USS Cole Bombing”.

ASSESSMENT OF METHODOLOGICAL QUALITY

Empirical studies will be determined first as those studies involving the analysis of empirical data as described above. These studies will then be individually examined to determine if the subject is relevant to the prevention, detection, response and management of terrorism events as described above. Each study will then be examined individually in terms of the methodological design used. Some guidance will be obtained from the Maryland Report (see Sherman et al., 1997; Sherman et al., 2002) in terms of the study's rating on the “Sc ientific Methods Scale” (SMS). Moderately rigorous studies will be those who could be rated a “3” on the SMS scale, although we will evaluate each study on its own merits. Furthermore, we found Farrington's (2003) criteria to also be helpful in determining the scientific validity of studies and will use both guidance form the Sherman et al.'s SMS and Farrington's discussion of validity.

CRITERIA FOR DETERMINATION OF INDEPENDENT FINDINGS

We anticipate that an evaluation research study may include multiple findings on a variety of different types of strategies or on different study samples. Each research work will be reviewed in terms of whether outcomes arise from different samples, or if different outcomes arise from the same study samples. For those arising from independent samples but reported in one article or book, we will treat as separate findings. For multiple findings from the same sample, each will be scrutinized independently as to how to either combine the findings or choose one to represent the study.

DETAILS OF STUDY CODING CATEGORIES

As already mentioned, in our preliminary analysis we collected basic information about the citation, author, title, year, and abstract of the study. We also coded studies generally as empirical, thought pieces or cases studies and labeled the source of each study and the subject matter covered. In the systematic review, we will also record the types of outcomes measured, sample sizes, type of methodological design, unit of analysis, statistical tests used, reports of statistical significance if any, effect sizes and power statistics (if any), type of intervention, and the location or venue of intervention. Furthermore, rationales for studies given by authors and conclusions made by authors will also be recorded. Because the number of studies finally used in the systematic review is anticipated to be small, multiple authors of this review will separately analyze each study and come to a consensus about the coding of the data.

STATISTICAL PROCEDURES AND CONVENTIONS

The results of each study will be summarized and presented in table format listing relevant variables such as findings, effect sizes, statistical significance, type of intervention and type of methodology used. If possible, meta-analytic approaches to combining data from studies will be used (see Lipsey and Wilson, 2001). When possible, the authors will calculate effect sizes if enough data is present in the study results to do so. We plan to use commonly accepted standardized measures of effect sizes (see Lipsey and Wilson, 2001; Rosenthal, 1991) to calculate and compare effect sizes. Furthermore, a general assessment will be made as to whether positive, negative or null findings are found. This will provide a general assessment as to whether the study investigator and the reviewers saw the conclusion to have a “positive effect”, “no detected effect or the effect was not statistically significant” or “the program or treatment had an unintended backfire effect” (Weisburd et al., 2001: 58–59).