Abstract
The use of DNA testing as part of police investigative work has increased substantially since its emergence in the 1980s. The objective of this Campbell systematic review has been to synthesize the existing evidence on the effectiveness of DNA testing as part of routine police investigative practices compared to other more traditional forms of investigation.
We have identified five studies that clearly addressed the effectiveness of DNA testing as part of a criminal investigation to improve criminal justice system outcomes. The evidence suggests that DNA testing has positive value when used to investigate a broad range of crime types. There are caveats. Other than a methodologically sound evaluation in five U.S. jurisdictions, the evidence of the utility of DNA testing for serious violent crimes is based on studies with clear methodological weaknesses.
Synopsis
The use of DNA testing as part of police investigative work has increased substantially since its emergence in the 1980s. The objective of this review has been to synthesize the existing evidence on the effectiveness of DNA testing as part of routine police investigative practices compared to other more traditional forms of investigation. We have identified five studies that clearly addressed the effectiveness of DNA testing as part of a criminal investigation to improve criminal justice system outcomes. The evidence suggests that DNA testing has positive value when used to investigate a broad range of crime types. There are caveats. Other than a methodologically sound evaluation in five U.S. jurisdictions, the evidence of the utility of DNA testing for serious violent crimes is based on studies with clear methodological weaknesses.
Abstract
BACKGROUND
The use of DNA testing as part of police investigative work has increased substantially since its emergence in the 1980s. Initially used primarily in serious cases, such as homicides and rapes, recent use has expanded to include additional crimes, such as property offenses. The science behind the accuracy of DNA testing is substantial. With the growth of DNA databases, the possibility of comparing DNA evidence collected from a crime scene against a DNA database to identify suspects has become feasible. An important empirical question is whether wide scale use of DNA testing as part of routine police investigative practices will improve case outcomes.
OBJECTIVES
The objective of this review has been to synthesize existing evidence on the effectiveness of DNA testing as part of routine police investigative practices compared to other more traditional forms of investigation.
SEARCH CRITERIA
We were inclusive with respect to research design, including both experimental RCT designs and observational or quasi-experimental studies. All studies must have provided an estimate of the effect of DNA testing (yes/no or degree of) as part of the investigative phase of a criminal case on a criminal justice system outcome, such as the identification of a suspect or a conviction.
SELECTION STRATEGY
We searched 35 electronic databases and reviewed the references of seminal works in the area. This produced more than 10,000 titles that we scanned for potentially eligible works. These potentially eligible works were examined more carefully and evaluated against our eligibility criteria. This process resulted in five studies that met our inclusion criteria. Four of these five studies were dissertations or government reports.
DATA COLLECTION AND ANALYSIS
Studies were coded with respect to a range of methodological and substantive features. When possible, odds-ratios were computed to reflect the effect of DNA testing on criminal justice system outcomes. Given the diversity of designs, we did not meta-analyze results across studies. We did, however, use meta-analysis to synthesize results across multiple sites within two of the five studies.
MAIN RESULTS
Across studies we found generally positive results regarding the utility of DNA testing. The results from the single experimental study on the effectiveness of DNA for property crimes were consistently positive across the included sites. A time-series analysis found a relationship between the size of a local DNA database and clearance rates for most crime types. Two of the remaining three quasi-experimental designs found positive, and sometimes large, effects for the benefits of DNA testing. A study of homicides found a negative effect of the value of DNA. Clear alternative explanations for this finding are plausible.
REVIEWERS' CONCLUSIONS
The evidence suggests that DNA testing has value when used to investigate a broad range of crime types. There are caveats to this conclusion, and additional high quality evaluations are needed to establish the robustness and generalizability of these findings. Other than a methodologically sound evaluation in five U.S. jurisdictions, the evidence of the utility of DNA testing in investigative practices for serious violent crimes is based on studies with clear methodological weaknesses.
1 Background
Despite the strong advances in police innovations and their evaluation in the 1980s and 1990s (Weisburd and Braga, 2006), there is more generally little evidence regarding whether technology (e.g. advanced computing, automatic finger print systems, and DNA analysis) has impacted on the success of investigations (Committee, 2004). In this regard Horvath et al. (2001:5) argue that “in many fundamental respects, the police criminal investigation process has remained relatively unaffected by the significant changes that have occurred in policing, the crime problem and technology in the past thirty years.” However, the use of DNA testing as part of police investigative work has increased substantially since its emergence in the 1980s (e.g., Roman et al., 2008; Home Office, 2005; Lovrich et al., 2003), even in the absence of such evaluations. Initially used primarily in serious cases, such as homicides and rapes, recent use has expanded to include additional crimes, such as property offenses (e.g., Asplen, 2004, Home Office, 2005). The fundamental question motivating this review is: Does the use of DNA testing improve the effectiveness of the police in identifying and convicting perpetrators of crime, particularly if expanded beyond its traditional use in serious and violent offenses?
Assessment of the importance of the use of DNA in police investigations comes in part from the low clearance rates (i.e. rates of solved crimes) that are often achieved using conventional investigative techniques. This problem was noted in the National Research Council report on Fairness and Effectiveness in Policing (Committee, 2004) which concluded “that most property crimes and many violent crimes are unsolved” (p. 227). The problem is particularly acute for property crimes where clearance rates are often much below 20% (Cordner, 1989; Weisburd et al., in press). Low clearance rates in property crime are the result of a combination of factors, but are often attributed to a lack of evidence in property crime investigations (where eye witness accounts are rare) and the large number of cases relative to investigators available (Eck, 1983; Greenwood, Chaiken, & Petersilia, 1977). The NRC report (2004) also notes that studies have shown “that if clues pointing to specific suspects were not provided by citizens to the first responding officers, then follow-up investigators had great difficulty solving the case” (p. 228). Forensic DNA offers a means of providing such critical early information and identifications to investigators in cases where it may not have otherwise been available (Neyroud, 2010).
DNA is essentially a long string of information that is represented by combinations of four possible acid pairings of adenine, cytosine, guanine, and thymine (AT, GC, TA, CG). Long sequences of these pairs contain each individual's genetic information, much the same way strings of 1s and 0s in binary code can contain the information used by computers (i.e. 00110101101100). While over 99.9% of the sequence of acid pairs is exactly the same in everyone's DNA, the relatively small portions of DNA that are not the same are very unique for each individual (United States, About Forensic DNA). Within this unique portion, there are several known points, or loci, where short sequences of these acid pairs are repeated over and over. By counting the number of times these short sequences repeat at each of these loci, it is possible to determine, with a very high degree of certainty, whether a sample of DNA came from a particular individual. In a recent report from the National Research Council, the Committee on Identifying the Needs of the Forensic Sciences Community investigated many aspects of the forensic sciences currently being used in the United States. While the report was critical of many of the other methods, the committee concluded that DNA was the only “forensic method [that] has been rigorously shown to have the capacity to consistently, and with a high degree of certainty, demonstrate a connection between evidence and a specific individual source” (Committee on Identifying the Needs of the Forensic Sciences Community, 2009, p. 7)
Measuring the number of times a short string of acid pairs repeats at a particular locus is known as STR (short tandem repeats) analysis. STR analysis has largely replaced the previously used RFLP (restriction fragment length polymorphism) method of analysis, which measured much longer strings of repeating acid pairs over much larger portions of the DNA string than the several loci used in STR analysis. The FBI has identified 13 specific loci that are used in forensic analyses, and when the number of repeats for each of these loci is the same in two samples, the odds of the similarity being a coincidence are about one in a billion (U.S. Department of Energy, 2008). This method of DNA analysis can result in three possible outcomes: inclusion, exclusion, or inconclusive result (United States, Possible Results from DNA Tests). It is important to recognize that an inclusion is not actually the same as a “match.” Rather, it only means that the odds of a sample coming from another source are extremely remote.
Just as the STR method improved DNA analysis from the earlier RFLP method, other developments in the scientific aspects of forensic DNA are continuing to advance the capabilities of the discipline. The new techniques of mtDNA (mitochondrial DNA) and Y-STR (Y chromosome STR) analysis can be used to identify people through their familial lines (United States, Mitochondrial Analysis; United States, Y-Chromosome Analysis). Automation and robotics are being incorporated into crime laboratories “to improve the speed and to reduce the cost of DNA analysis” (United States, Miniaturization and Automation). New portable analysis systems are currently being developed for use in law enforcement (NEC, 2007; NPIA, 2010). More sensitive methods of collecting samples are allowing analysis of even the minuscule amounts of DNA left from simply touching a surface (Gill, 2001).
As fast as these new technologies related to forensic DNA have been developing, new policies and applications of that science have developed just as quickly. As mentioned earlier, the use of DNA testing as part of police investigative work has increased substantially since its emergence in the 1980s and DNA databases, such as CODIS (Combined Offender DNA Index System) in the United States and NDNAD (National DNA Database) in the United Kingdom, have also been developed to provide an entirely new method of investigating crime. Instead of using DNA analysis simply to corroborate the guilt (or innocence) of a previously identified suspect, DNA databases can themselves identify suspects before there is any other evidence implicating the individual. The offenses for which DNA is collected and entered into these databases have broadened so that within the United States, as of June 2009, 47 states collect DNA samples from all convicted felons, and 37 states collect samples from those convicted of certain misdemeanors (DNA Resource, 2009). Many states are either considering, or have already implemented, policies of collecting DNA samples from arrestees. Recent research focusing on such databases have begun to create models for future assessments of database effectiveness (Walsh, et al., 2010) and to project their criminal deterrent effect (Bhati, 2010).
In addition to identifying suspects, forensic DNA has also contributed to the exoneration of those wrongfully convicted of crimes they did not commit. Thus far, forensic DNA has contributed to the exoneration of 255 individuals convicted of serious crimes in the United States, 17 of who have served time on death row (The Innocence Project, 2010).
Along with the benefits of any new advance, there are also challenges. The utility and popularity of forensic DNA has also proved to be one of its shortcomings. Misconceptions about the capabilities of DNA analysis and other forensic sciences, known as the CSI effect, are common (Schweitzer & Saks, 2007). With the promise of forensic answers in difficult criminal investigations, increased submissions to crime laboratories have made backlogs of unprocessed evidence a frequent occurrence (Lovrich, et al., 2004). Stories of thousands of rape cases remaining unanalyzed and sitting on lab shelves are unfortunately frequent (CA NOW, 2009; Nadler, 2002; Weiner). These public misconceptions and implementation challenges can threaten important perceptions of police legitimacy, but rigorous empirical evaluations (such as randomized controlled studies) can provide the understandings necessary to avoid such difficulties.
The best time to implement randomized controlled studies is before a technology becomes routine practice. The expanding use of forensic DNA in criminal investigations naturally fits experimentation. The random assignment of cases to existing practice or an experimental approach is both ethical and a wise use of resources when there is reason to believe that the experimental approach may improve practice (Wilson, McClure, Weisburd, 2010). However, as the use of forensic DNA in criminal investigations continues to expand, the apparent benefits for investigations may begin to preclude the opportunities to use such rigorous empirical methods. Though their decisions may not be supported by research findings, few decision-makers would allow for random assignment of DNA analysis for serious/violent crime investigations. As a result, the intuitive benefits and rapid expansion of forensic DNA in criminal investigations may ironically limit opportunities to conduct the empirical evaluations that are necessary for forensic DNA to maximize its potential benefit for criminal investigations.
To address this issue, the goal of this systematic review is to summarize the relevant and accessible evidence on the effectiveness of DNA testing in routine police work. Specifically, the fundamental question motivating this review is: Does the use of DNA testing improve the effectiveness of the police in identifying and convicting perpetrators of crime, particularly if expanded beyond its traditional use in serious and violent crime investigations?
2 Objectives
The objective of this review has been to synthesize the existing evidence on the effectiveness of DNA testing as part of routine police investigative practices compared to other more traditional forms of investigation. Of interest are the effects on the apprehension of individuals responsible for crimes and reductions in the likelihood of the involvement of innocent individuals in the criminal justice system. We were also interested in the effect of DNA on the cost, speed, clearance rates, arrest rates, and conviction rates of investigations.
It is anticipated that this review will help inform policy makers and the police department decisions regarding the routine use of DNA testing in investigative police work. Many police agencies are expanding the use of DNA testing, and a critical examination of the existing evidence is warranted.
3 Methods
3.1 CRITERIA FOR CONSIDERING STUDIES FOR THIS REVIEW
3.1.1 Types of interventions
The scope of the review will be limited to the use of DNA testing by police as part of their investigations of crime. We will not consider the use of DNA testing by criminal defendants or by prosecutors. Of particular interest is the routine or expanded application of DNA testing in cases that often do not make use of available DNA evidence. The primary basis for eligible studies requires some variation in the use of DNA in the investigation.
3.1.2 Types of studies
Given the limited amount of research in this area, we included a broad range of study designs. All designs, however, must have provided an estimate of the effect of DNA testing relative to an alternative or more limited type or application of analysis (i.e., varying degrees of DNA testing).
The ideal design type would have randomly assigned cases to either a DNA testing condition or a traditional investigative practice condition, and then assessed the outcomes of both conditions from the same time-frame. We considered any such designs that varied the degree of DNA testing used and examine one or more of the outcomes discussed above. The comparison condition did not need to represent the absence of DNA analysis, but simply a variation of DNA from the treatment condition.
We also considered quasi-experimental designs in which there was a control group that either matched the DNA testing group, or was identified as comparable. To include these less methodologically rigorous designs required statistical justification of the suitability of the control groups identified.
Interrupted time-series designs were also considered for inclusion in this review, along with other regression-based analyses that estimate the impact of DNA testing on a relevant outcome. These designs were handled separately, as they can be especially vulnerable to historical threats to validity. An essential feature of time-series designs is the multiple baseline estimates of the rate of interest (e.g., identification of a suspect). This allows for an assessment of both the natural change over time and change that may be associated with the start of an intervention, such as the use of DNA testing and some other change related to the use of DNA in police investigative work.
Basic pre-post designs with a single pre-DNA and a single post-DNA estimate were eligible, but these designs were handled separately as they provide a weak basis for drawing causal inferences. Other quasi-experimental designs were eligible, such as a design that contrasts the clearance rates for different police agencies (without statistically justifying the validity of the comparison), but these were to be reported separately.
3.1.3 Types of outcome measures
This review included all crime types. We recognize, however, that the utility of DNA testing is likely to vary substantially across crime type. The current trend toward increased use of DNA testing in burglary cases reflects that burglary is often a high volume crime, with offenders engaged in serial burglaries (Roman et al., 2008). The serial nature of this crime increases the likelihood that an offender may be identified in an existing database and helps police connect crimes committed by the same individual. The added value of DNA testing may be less for other crimes. As such, we examined the evidence separately by type of offense.
DNA testing may improve outcomes at several stages of the investigative process. It may facilitate the identification of suspects through the use of DNA databases, such as CODIS in the United States or the UK National Criminal Intelligence Database in the United Kingdom. DNA testing also may help eliminate suspects or identify one suspect among multiple suspects found through traditional police investigative methods. These processes may increase the likelihood of an arrest and a conviction, raising the number of cleared cases. Studies of the effectiveness of DNA testing may examine the effectiveness of DNA testing on one or more of these outcomes. As such, eligible outcomes included the following: the rate at which suspects are identified, the arrest rate of a suspect, the conviction rate, length and speed of an investigation, the cost of the investigation, and the case clearance rate (i.e., how often cases are successfully solved).
The source of the data will generally be from official records or reports of some form. However, we will not restrict eligibility based on the source of the outcome data. All sources will be considered.
3.2 SEARCH METHODS FOR IDENTIFICATION OF STUDIES
Preliminary searches were conducted with the terms “DNA” and “Police” or “Policing” or “Investigation”. Additional terms were developed through these preliminary searches. In addition to searching the databases listed below (see electronic sources), we also searched the UK Home Office website for relevant publications. Google Scholar was also used to identify publications not already captured through the formal databases. Key individuals working in this field were solicited for assistance in identifying relevant studies (e.g., Peter Neyroud at the National Policing Improvement Agency). We also made efforts to identify studies in languages other than English.
3.2.1 Search Strategy for the Identification of Relevant Studies
In consultation with an information retrieval specialist, a search strategy for electronic databases was developed in order to reach an optimal balance between potentially relevant search results and the large number of results using similar terms, yet addressing issues other than those of interest in this project. This search strategy was broad, but avoided the combination of terms that would include the large body of technical research done on DNA, as well as studies exclusively addressing non-DNA forensic sciences.
3.2.2 Search Terms
Two categories of keywords were developed for this search. The first category addressed the technology of interest (DNA). The second category addressed the application of DNA testing in police investigative work, and included terms such as policing, detective, arrest, etc. The intention of separating the terms in this manner was to include all the potentially relevant results, while simultaneously excluding the large bodies of literature on DNA from non-forensic disciplines. These two sets of keywords were combined with a Boolean AND. Unfortunately, the body of literature on the application of DNA to criminal investigations is much smaller than the literature on aspects of DNA addressed by other disciplines (i.e., the basic science of DNA testing). This resulted in search results in the low hundreds for some databases. Particular Technology of Interest DNA or “DEOXYRIBONUCLEIC ACID” Application context FORENSIC! or LAW or LEGAL or COURT* or TRIAL! or CSI or C.S.I. or “CRIME SCENE” or “CRIME LAB*” or ANALYSIS or INVESTIGATION! or POLIC* or DETECT* or PROSECUT* or DEFEN* or CRIM* or CODIS or C.O.D.I.S. or “COMBINED DNA INDEX SYSTEM” or NON-VIOLENT or “RAPE KIT!” or IDENTI* or ARREST! or COST! or CLEARANCE! or CLOSURE! or SPEED or “COLD CASE!” or EXCULPAT* or “WRONGFUL CONVICTION!” or “ACTUAL INNOCENCE” or BACKLOG
These search terms were derived from a review of literature on the topic of forensic DNA, such as text books, websites, and journal articles. The terms were separated into the two categories above, each with a different level of generality. The broadest level simply included the search terms “DNA” and “Deoxyribonucleic Acid.” As at least one of these two search terms were certain to be in any relevant resource, each search began with these two terms. Only using these search terms in many of the databases produced an unwieldy number of results, most of which were clearly not relevant for this review. The second level of search terms covered terms that described the application of DNA in the criminal justice system. The terms in this second level were joined with the first level by the appropriate connectors for each database (a Boolean “AND”). As the related literature on this topic is quite significant, even these two levels of search terms frequently returned more results than could feasibly be reviewed, so the second level of search terms were broken into groups as necessary. This restricted the search results from those containing at least on term from each group to results that had at least one results from all the groups. This search strategy and the form of the search terms (i.e. including search devices such as ! * ?) were modified to suit the requirements of each specific database.
3.2.3 Electronic Sources
The search strategy described above was applied to the following databases, which cover both the more accessible sources as well as the grey literature.
Association of Chief Police Officers ACPO Association of Chief Police Officers of Scotland ACPOS Association of Police Authorities APA AIC – Australian Institute of Criminology Australian Research Council Centre of Excellence in Policing and Security (CEPS) ASSIA – Applied Social Science Index and Abstracts Canadian Police Research Centre CINCH (the Australian Criminology Database) Criminal Justice Abstracts Dissertation Abstracts EconLit ENFSI – European Network of Forensic Science Institutes HeinOnline Her Majesty's Inspectorate of Constabulary HMIC Ingenta Jill Dando Insitute of Crime Science (JDI) JSTOR Medline/Embase NCJRS (National Criminal Justice Reference Service) NCSTL (National Clearinghouse for Science, Technology, and the Law) Policy Archive PolicyFile PROQUEST Public Affairs Information Service RAND Documents ResearchNow Science Direct Scottish Institute for Policing Research SIPR Social Sciences Citation Index Social Services Abstracts Sociological Abstracts SSRN – Social Science Research Network Worldwide Political Science Abstracts
3.3 DATA COLLECTION AND ANALYSIS
3.3.1 Assessment of risk of bias in included studies
The methodological quality of the studies was assessed by coding the features contained in the attached coding forms. The types of information that were considered included: (1) nature of assignment to conditions; (2) use of matching of cases or the use of statistical controls, such as regression analysis to adjust for potential selection bias in the case of non-random assignment to conditions; (3) representativeness of the sample of cases (e.g., census, random sample, convenience sample); (4) attrition of cases from the study: and, (5) replication of findings in multiple jurisdictions.
3.3.2 Measures of treatment effect
The odds-ratio was the effect size of choice for all outcomes of a dichotomous or binary nature. For example, an odds-ratio was computed to represent the effect of DNA testing relative to an alternative on the proportion of cases cleared. In contrast, the standardized mean difference effect size was used for outcomes measured on a continuous measure or outcomes that represent counts or rates. Standard methods of computing effect sizes were used (see Lipsey and Wilson, 2001). In the case of quasi-experimental designs, preference was given for effect sizes adjusted for baseline differences or other covariate. For example, a coefficient for a treatment dummy variable from a logistic regression model can be converted into a treatment effect odds-ratio that is adjusted for the other variables in the model. Similarly, covariate adjusted means can be used in the computation of a standardized mean difference of effect sizes (Lipsey and Wilson, 2001). Preference was also given to quasi-experimental designs that use a control/comparison condition that is assessed at the same time as the treatment condition.
3.3.3 Unit of analysis issues
The primary unit-of-analysis within four of the five studies of interest was the criminal case, such as a sexual offense or a burglary. Effects from these studies that reported results separately for distinct and independent policing units, such as units in different cities, were coded separately and treated as statistically independent.
Multiple reports or manuscripts based on the same study or data were treated as a single entity for purposes of this review. We selected the most complete references if all of the relevant information in secondary reports (e.g., journal manuscript) was contained within the primary report (e.g., technical report). However, if the multiple reports each provided unique information (e.g., different outcomes or different jurisdictions), then all the reports were included and coded as part of the review.
3.3.4 Assessment of reporting biases
Reporting bias was addressed primarily by searching for, and including, unpublished works, such as technical reports, dissertations, and government reports. We planned in the protocol to perform statistical tests for publication bias, such as the Tweedie and Duvall's trim-and-fill method and funnel plots (see Duval and Tweedie, 2000). We were unable to perform these analyses given the nature of the studies included. However, four of the five studies reviewed represent grey literature.
3.3.5 Data synthesis
We did not anticipate being able to perform a meta-analysis on the effect sizes generated from the studies included in this review. Our a priori decision rule for performing meta-analysis was as follows: two or more studies, each with a computable effect size of a common outcomes construct (potentially measured in different ways), and similar comparison condition. Instances that satisfy this decision rule were to be meta-analyzed using standard methods (i.e., inverse-variance weighted, random effects model; see Lipsey and Wilson, 2001). We did, as planned, use meta-analysis to synthesize results across multiple sites within a single study when these results were reported separately.
In the absence of meta-analysis, synthesis of the findings across studies and inferences about the effectiveness of DNA use in policing were based on the size and direction of the effects and related confidence intervals. Greater emphasis was placed on the direction of effects and consistency of effects across similar studies than on statistical significance. A single high quality evaluation, with an effect size of a meaningful magnitude that was also statistically significant, was interpreted as evidence that DNA use can be effective, assuming there was not an equally strong study with a negative result. Positive findings of a meaningful magnitude were interpreted as promising, but were also viewed as providing limited evidence of the potential effectiveness of forensic DNA use in police investigations.
4 Main Results
4.1 SEARCH RESULTS
This search strategy identified over 10,000 potential documents across databases. The titles of these documents were reviewed to identify potentially eligible studies for inclusion in this review. At this stage, the process was generously inclusive, and included any title that was remotely likely to be eligible. Next, the full abstracts of these potentially eligible studies were reviewed to identify the documents most likely to meet our eligibility criteria. The full texts of these documents were then reviewed for final eligibility. Final eligibility was determined by two members of the research team and resulted in the identification of five studies. Several studies addressed issues peripheral to the objective of this review and are listed in an appendix (References for Ineligible for Relevant Studies).
4.2 DESCRIPTION OF METHODOLOGY USED IN ELIGIBLE STUDIES
Each of the five eligible studies shared the common feature of a design intended to assess the impact of the use of DNA on aspects of the criminal justice system. Beyond this, the studies were quite dissimilar.
Roman et al. (2008) used a true experimental design to assess the impact of DNA on several case outcomes, including the identification, arrest, and prosecution of property crimes, including both residential and commercial burglaries, as well as theft from auto. This study was implemented in five police departments in different cities or counties in the United States (Denver, Colorado; Los Angeles, California; Orange County, California; Phoenix, Arizona; and Topeka, Kansas). Funding from the NIJ grant supporting this work was used to provide each site with additional resources for conducting DNA analysis on these property offences. The intervention or conditions to which cases were assigned were DNA analysis as soon as possible or DNA analysis only after 60 days. Random assignment to these conditions occurred after officers recovered biological evidence from a property crime scenes. Thus, all cases had the potential for DNA testing and represent a subset of property crime cases. Randomization was controlled by the research team. The target sample size per site was 500 or 250 per condition. Three of the five sites either met or exceeded this target number, another site finished with nearly 400 cases, and the last site finished with a little over half of the targeted amount. There were a total of 2,160 cases across the five sites. The outcomes measured in this study included whether a suspect was identified, an arrest was made, and whether the case was referred for prosecution. This study also collected cost data, reporting both the marginal cost of the DNA testing (average cost for each case) and the cost effectiveness or cost of an additional conviction.
Dunsmuir et al. (2008) examined the effect of an expanding DNA database in NSW Australia on clearance rates for eight different crime types. In January of 2001, NSW Australia began testing the DNA of all prison inmates and adding the results to their DNA database. As such, the size of the NSW DNA database began to grow substantially. Dunsmuir et al. examined whether the expansion in the DNA database improved police effectiveness through a times-series analysis of monthly clearance rates, charge rates, and the ratio of charge to clearance rates from 1995 through 2007, inclusively. It was reasoned that there would be a lag between the increase in the DNA database and the improvement in clearance rates and that this lag would vary with crime type. Prison inmates cannot commit crimes, at least against the general public, until after they are released from prison. As such, Dunsmuir et al. tested for different lags in the timing of the increase police effectiveness after January 2001. The ARIMA models used to examine this affect assumed a simple linear relationship between the expanding DNA database and three different dependent variables. They argued that there was no theoretical basis on which to justify exploring alternative functional forms.
Briody (2004), Schroeder (2007), and Tully (1998) used similar quasi-experimental designs that compared criminal justice outcomes in cases with DNA testing relative to cases without DNA testing. Briody's study was conducted in Australia, whereas Schroeder's and Tully's studies were conducted in the United States.
Briody (2004) examined whether the presence of DNA evidence affected the acceptance of cases for prosecution, the rate at which defendants plead guilty, and the conviction rate. The research design matched DNA cases with non-DNA cases based on the seriousness of the crime. Additionally, only cases that had reached a final disposition and cases with complete records were included. A total of 750 matched cases from Queensland, Australia were used. Separate analyses were performed for sexual offense crimes, homicides, serious assaults, and property crimes. Logistic regression models were used to assess the influence of DNA on case outcomes, along with numerous other case characteristics, including the existence of fingerprint evidence. Unfortunately, the presence of DNA-evidence was dropped from models if it was not significantly correlated with the outcome of interest.
Schroeder (2007) examined clearance rates of homicide cases in the Borough of Manhattan, New York, and whether DNA evidence facilitated case closure. From a potential population of 957 homicide cases from 1996 through 2003, Schroeder extracted data from 602 with available case files and categorized the cases into groups based on the way DNA had been used in those investigations. He found 230 cases where DNA evidence was available, but had not been used in the investigation, and another 40 where DNA evidence was both available and used in the investigation. Schroeder compared the clearance rates between these two groups. Furthermore, he examined the utility of DNA evidence in the 40 cases where it was used (i.e., did it identify a suspect, etc.).
Tully (1998) examined the effect of DNA testing on plea bargaining, convictions and sentence length. Tully collected official data, and coded them for case characteristics from four counties in the State of Maryland in the United States. Only three of these counties (Anne Arundel, Montgomery, and St. Mary's) included a comparison condition. As such, only the data from these three counties were used in this review. DNA cases were compared to two forms of controls: (1) pre-DNA historical cases from 1979 through 1986 where biological evidence was collected, and (2) current cases with biological evidence but no DNA testing. Unfortunately, these two comparison groups were combined in the report and we could not examine them separately. Only 107 DNA cases and 92 non-DNA cases were available across the three counties. The study provides limited information on how these cases were selected.
4.3 ASSESSMENT OF METHODOLOGICAL QUALITY
The studies varied substantially in methodological quality (see table 2). Only one study used a true experimental design with random assignment to conditions (Roman et al., 2008). This study provides the strongest basis from which to infer a causal connection between the use of DNA and improved police effectiveness in solving volume crimes. Furthermore, this study maintained the integrity of the random assignment and the analyses are based on intent-to-treat (i.e., original assignment to conditions). The authors do note that some treatment cases did not receive DNA testing in less than 60-days. As such, these cases were no different than control cases. This treatment integrity issue would downwardly bias any true effectiveness of DNA testing. The external validity of this study is enhanced by the multi-site design. The five distinct police jurisdictions differed in important ways, providing a range of contexts in which the effectiveness of DNA use was assessed.
The four remaining studies were quasi-experimental relying on existing criminal justice system data. Selection bias is a major concern for three of these (Briody, 2004; Schroeder, 2007; Tully, 1998). The homicide cases with DNA results available during the investigative stages in the Schroeder study are highly likely to differ substantially from the cases with DNA utilized evidence. A full 74 percent of the latter cases were cleared, indicating that the police were highly successful in solving these cases without the assistance of DNA. The very low clearance rate for homicide cases with DNA evidence and testing strongly suggests that these cases were fundamentally different and more difficult to solve. Another weakness of this study was the truncation of sample size based on the availability of case files. A full 37.4% of the eligible homicide cases were not included in the study because the case files were unavailable. Although these may represent a random sample of cases, there is no way to assess this assumption, raising the possibility that available cases differ in some important way. Additionally, this reduced the sample size. Only 270 cases involved the collection of DNA evidence and in only 40 of these was DNA testing available during the investigative stage. Thus, we judge the evidence regarding the utility of DNA testing in homicide cases as questionable.
The Briody (2004) study reduced the threat of selection bias through matching of DNA and non-DNA cases based on case seriousness and by only using cases that had reached a final disposition. Furthermore, Briody assessed the effects of DNA in a multivariate logistic regression model, adjusting for many potential confounds. Without a better understanding of the mechanisms to determine why DNA evidence is tested in some cases and not in others, it is difficult to assess the extent to which selection bias has been reduced. Our main methodological concern with this study is outcome selection bias. Only the independent variables strongly correlated with the dependent variable, often including the use of DNA testing, were included in the regression models. As such, we are unable to assess the magnitude and direction of effects for the several models in which the independent variable for DNA testing was not included. This raises the possibility that the observed positive effects are upwardly biased, representing those effects favored by chance.
Similar to Schroeder (2007) and Briody (2004), Tully also compared the outcomes of cases that used DNA testing versus cases that did not. All cases had DNA evidence available for testing. No attempt was made to assess the similarity of the cases and the very small sample sizes resulted in highly unstable results across the three counties.
The primary methodological concern with the time-series analysis by Dunsmuir et al. (2008), discussed in detail by the authors, is the threat of history—that some other historical event or events account for the observed improvement in clearance rates, such as changes in police practice unrelated to DNA testing. Attempts were made to address this through the use of covariates in the analysis, but the potential threat cannot be completely ruled out.
4.4 FINDINGS OF ELIGIBLE STUDIES
The findings of the five eligible studies are summarized in Table 4. Where possible we report the percentage of cases in the DNA and control condition with the specified outcome (e.g., percentage of suspects identified by condition) and the odds-ratio and 95% confidence intervals for the relationship between DNA testing of the specific criminal justice outcome. The diversity of designs and research questions examined by this collection of studies precluded meaningful meta-analysis. However, meta-analytic methods were used to aggregate results across the five sites in the Roman et al. (2008) and the Tully (1998) studies. In the latter case, only the aggregated results are reported due to the very small sample sizes of the individual sites (as low as 21 for one site).
Looking across studies we found generally positive results regarding the utility of DNA testing. The results in the Roman et al. (2008) study were consistently positive. Nineteen of 24 models performed by Dunsmuir et al. (2008) showed an improvement in the clearance rates as the size of the DNA database increased. At the aggregate level, the results from Tully (1998) are all positive, although only 3 of the 9 aggregated odds-ratios are statistically significant (the odds-ratio reported at 1.0 is actually slightly positive, 1.045). Furthermore, this specific effect was for convictions for rape cases. The author notes that a major complication with the rape cases was an understandable unwillingness of victims to testify, even with the existence of strong forensic evidence. The Schroeder (2007) study produced a large negative effect for the utility of DNA testing. As discussed above, this study suffers from serious methodological weaknesses. The small number of homicide cases in which available DNA evidence was actually tested may reflect limited applicability to this type of crime. In general, homicides have very high clearance rates (often in the range of 75%, as observed in the non-DNA testing cases in Schroeder's study). DNA testing may be highly useful in a small subset of homicide cases (and cases of convicted individuals being exonerated must not be ignored). The Schroeder study does not provide a credible test of this possibility.
The sizes of the effects from the Roman et al. (2008) study are remarkable. The use of DNA testing in high volume property crimes more than tripled the odds of identifying a suspect and making an arrest, and more than doubled the odds of a case being accepted for prosecution. Although the results varied some across sites, all sites observed increased identification, arrest, and prosecution. This single high quality study provides compelling evidence of the value of DNA testing for solving property crimes.
The Dunsmuir et al. (2008) study estimates the 12-month percent change in the cases cleared, charge rate, and charge to clearance rate ratio as a function of the increase in the size of the local DNA database. These estimates ranged from a negative effect of -.9 to a high of 8.1 percent. Most of the effects were positive. The results varied substantially across crime types, with no clear explanation for this pattern.
Finally, Briody (2004) reported odds-ratio ranging from 2.1 to 33.1 for the independent variable representing the use of DNA testing in logistic regression models that predicted various criminal justice system outcomes. Unfortunately, results for DNA testing were not included in all the models.
4.5 COST ISSUES
Cost issues related to the use of DNA are changing rapidly as the technology advances. Only the Roman et al. (2008) study examined costs. The average cost of adding DNA testing to property crime investigations varied from a low of $815 to a high of $2,481. The clearance rates for property crimes are generally quite low. Even with DNA testing, less than half result in a prosecution (across the five sites in Roman et al., the range was from 7% to 46% with DNA testing). As such, it is important to examine the cost effectiveness of DNA testing, that is, the cost per one additional conviction. Across the five sites included in Roman et al., the additional costs to the system to achieve an additional conviction ranged from just under $2,000 to just under $13,000. The latter number is largely a function of the high costs of DNA testing at one of the sites. The authors note that there were unique circumstances at this site that contributed to these high costs.
The costs of DNA testing are a moving target. As the technology advances and economies of scale come into play, costs are decreasing substantially. We have heard reports of costs under $100 for DNA testing. However, costs can vary dramatically depending on the type of analysis conducted and the condition of the sample. The UK police service is experimenting with a brief case size portable DNA testing kit that can be brought to a crime scene and produce DNA results in under an hour. The rapid changes in this area make cost-effectiveness and benefit-cost assessment difficult at best.
5 Reviewers' Conclusions
The objective of this review was to assess the utility of DNA testing in solving crimes. It is important to recognize that we were not questioning the science behind DNA testing. As pointed out in a recent report by the Committee on Science, Technology, and Law of the (US) National Academies (2009), DNA is the only forensic technology based on strong science. In the case of serious crimes, such as homicide and sexual assault, it can be argued that solving one additional crime is of value. For example, of the 40 homicide cases in the Schroeder study that involved DNA testing, 16 cases provided a lead from a DNA database and one of these cases was cleared. Although we cannot know for certain whether this case would have been cleared in the absence of DNA testing, the possibility that DNA will solve an otherwise unsolvable case in these serious crimes is compelling. This is not the case for less serious crimes, such as property or robbery crimes. The use of DNA requires time and fiscal resources. As such, it is important to ask whether DNA testing improves the effectiveness of police investigative practices when DNA is used more broadly.
The evidence from this review generally supports the positive utility of DNA testing. The strongest evidence for the effectiveness of DNA for volume crimes comes from the randomized controlled trial in five jurisdictions conducted by Roman et al. The improvements in the number of suspects identified, arrest, and prosecuted were impressive and represented a two to three fold increase in the percentage of these cases solved. This is particularly valuable given the smaller percentage of cases of this type that are typically solved. It can be argued that as the size of the DNA databases expand, the effectiveness of the use of DNA testing on a broad scale will increase. The Dunsmuir et al. (2008) study provides support, albeit with caveats, for this claim.
It is important to recognize that the use of DNA testing for volume crimes represents an important shift in the way in which DNA is used in the investigative process. In the early days of DNA testing it was used solely to compare the DNA of suspects identified through traditional investigative practices against DNA samples obtained at a crime scene or off of a victim's body. The development of large-scale DNA databases allows to the ability to test DNA crime scene samples against the database to identify suspects.
Three of the five studies examined (Roman et al., 2008; Dunsmuir, 2008; and Briody, 2005) all provide evidences that this approach has value in solving property crimes.
The evidence for the value of DNA in more serious crimes is generally positive but based on weak evidence. The Briody (2005), Dunsmuir et al. (2008), Schroeder (2007), and Tully (1998) studies all provided evidence related to more serious crimes, such as sexual assault, rape, homicide, and serious assault. With the exception of the Schroeder study that suffers from serious selection bias issues, the evidence pointed in the direction of positive overall benefits from the use of DNA testing.
In summary, these results are encouraging and suggest the conclusion that DNA testing has value when used to investigate a broad range of crime types. There are caveats to this conclusion and additional high quality evaluations are needed to establish the robustness and generalizability of these findings. Other than a methodologically sound evaluation in five U.S. jurisdictions, the evidence of the utility of DNA testing in investigative practices for non-high-volume crimes is based on studies with clear methodological weaknesses. Additional studies are clearly needed, particularly those using the methodologically rigorous randomized experimental designs, such as Roman et al. (2008).
6 Acknowledgments
We thank the Campbell Collaboration for its financial support of this systematic review. We also thank the members of the Crime and Justice Steering Committee that provided recommendations of databases that would be appropriate to include in this review and for substantive comments and recommendations.