Abstract
Cognitive bias is a popular criticism of medicolegal death investigation. However, many of these criticisms are misplaced. Some have criticized the use of history into medical diagnosis. However, this criticism ignores the fact that the practice of forensic pathology is a medical discipline; it is not a simple mechanical test. History is an integral and necessary part of medical practice, and the lack of history is more dangerous than its use. Criticisms of manner determination often reflect an incorrect understanding of what manner is and what it is properly used for. Many criticisms wrapped in the flag of cognitive bias are in fact errors in scientific orthodoxy, real scientific disagreements, issues of ignorance, and lack of competence or certification, ethical lapses, and political pressure. Solutions that address cognitive bias do not address these issues. Many of the proposed solutions have not themselves been proven to be effective. Alternatives, such as intensive peer review, might address both cognitive bias and these other issues. Any proposed interventions should themselves be able to demonstrate a sound empirical basis.
Introduction
There has been much written and said recently about cognitive bias in forensic science, with proposed solutions such as “linear sequential unmasking” (1). Certainly, cognitive bias is by definition an issue in any human endeavor, and forensic practitioners are not an exception. However, wrapping all failures, or worse, all disagreements, in the flag of “cognitive bias” ignores the real problems and suggests inappropriate solutions. A one-size-fits-all approach to this issue such as the linear sequential unmasking scheme suggested for all disciplines is as bad, or worse, than the problem itself. There are many reasons.
Not all forensic sciences are DNA analysis, and not all interpretations are supposed to be done in a vacuum. Integrative disciplines involve the knowledgeable integration of information into their conclusions; that information is not extraneous and its integration does not constitute cognitive bias.
This is most clear in the integration of history and scene evidence in death investigation.
Discussion
History is Important, Necessary, and a Fundamental Part of Medical Practice
The use of history (and in forensic death investigation, scene information) is an integral part of medical diagnosis. Declining to use history is not an example of removing cognitive bias, but is instead simple malpractice. The literature regarding the importance of history in medical diagnosis is voluminous.
As referenced by Barry Scheck in the plenary session of the American Academy of Forensic Sciences in 2015, one of the primary problems in forensic pathology is that the pathologist is given incomplete history, not that he or she is given too much. As one author wrote last year:
The inference of manner is much like the inference of cause of death. One creates the equivalent of a differential diagnosis, ranks and prunes the possibilities, and comes to a conclusion as to which is most likely. The difference is in the degree to which the determination relies on external information. There is often little about a bullet hole that tells one who created it; many wounds are equally consistent with homicide, suicide, or even accident. It is necessary to consider investigational data, scene data, and history.
Some people in the legal community have disparaged the use of these external data as evidence of “cognitive bias” in our determinations. This is no more true for the medical examiner than it is for any other physician who makes a diagnosis integrating history and circumstances with physical examination and laboratory tests. It is the practice of medicine. A physician evaluating a patient with a fever and malaise may try to discover all sorts of things other than what the immediate physical examination provides – travel to other countries where certain disease are endemic, what was recently eaten for dinner, what medications the patient may be on, whether or not other people in the home also have fevers, etc. In light of the recent Ebola outbreak, it is not mere “cognitive bias” to ask about recent travel to Africa when evaluating someone in the emergency room, even if that travel is not a physical finding. In fact, failing to do so would be a lapse in care. The same thing is true with manner determination. It relies on history and circumstances as well as the immediate physical examination.
The importance of history and circumstances in medical diagnosis is sometimes misunderstood by those who are not physicians. In one study of 630 medical cases, for instance, history was the most important component of diagnosis in 56% of cases whereas physical examination was most important in only 17% of cases and laboratory investigation was most important in 23% of cases. In another study of 80 cases, history was most important in 76% of the cases while physical examination was most important in 18% of the cases. These findings have been replicated many times. To dismiss the importance of history and circumstances in medical diagnosis as “cognitive bias” represents a severe misunderstanding of inferential processes in medicine (2).
Sanders, in his critique of expert witness testimony notes the following 12 errors of unethical consultation (3). They include:
Falsifying data
Not performing the investigation at all
Altering data
Accepting conditional engagement (i.e., payment based on conclusion)
False testimony
Intentionally ignoring available data
Recanting a prior position
Accepting or performing an assignment beyond one's competence
Unauthorized lawyer influence
Inadequate time to complete assignment
Reaching a conclusion before completing research
Conflict of interest
It is no accident that “intentionally ignoring available data” is included on this list. It would unfortunate if we were to institutionalize this error rather than work to eliminate it.
No Element of History is 100% Perfect, But that is Not an Argument to Ignore It
Much has been made of the use of confession information and other evidence of low empiric value in the determination of cause and manner of death (4). Evidence, at least in medical diagnosis, is not a toggle between 100% and zero. Evidence must be weighed and evaluated by the physician. When a patient comes to a doctor, complains of pain, and asks for drugs for relief, it makes a difference in the evaluation of that complaint if the patient has a long history of drug seeking behavior or if he or she is a stoic person who almost never complains of anything. Much of the information that a pathologist evaluates is not of 100% specificity and sensitivity, even when just considering anatomic findings. That is why it is of paramount importance that the pathologist weigh all available information. But this is true in all integrative disciplines. David Schum, writing about the interpretation of intelligence data notes:
The problem is that most inferences involve processes or variables that are nonindependent in various ways, with genuinely interesting evidential subtleties. A causal assumption of complete independence among identified processes would in most cases invite inferential calamity. So we have no choice but to do our best at capturing what we believe are avenues of probabilistic dependence among processes of concern. To do so, we link nodes representing these processes by various patterns of arcs. I can think of no inference problem, outside the classroom, whose structure is either provided for us of immediately apparent. Constructing a network representation of an inference problem is a purely subjective judgmental task, one likely to result in a different structural pattern by each person who performs it (5).
This does not imply that two different pathologists will end up with different diagnoses, but they will likely get to the same diagnosis via different paths. This important truth is ignored by those who would impose some sort of cookbook inferential process on the pathologist or death investigator. Medical practice is not the technical performance of a laboratory test. An attempt to enforce one pathologist's inferential processes onto another by acting as a gatekeeper will result in calamity because the second pathologist cannot replicate the first pathologist's thought processes, and the first pathologist cannot predict the second pathologist's inferential net. Thus, neither of them will work.
Misinterpretation and Inappropriate Weighting of Historical or Scene Data is Not an Indication that the Data is Bad or should be Ignored. It Means that Better Training for that Individual is Needed in that Interpretation
The fact is that errors are made because people inappropriately weigh the value of information they have. That is not the fault of the information, nor does it mean that the information should not be available. It means that there needs to be work done to vet how the evidence is weighed by the person making the medical diagnosis.
Worse, limiting information by gatekeepers will increase rather than decrease the inappropriate weighting of information. It should be remembered that those in the forensic pathology world, particularly those who are engaged in the “shaken baby” controversy started as people with heterodox opinions (and those opinions are still heterodox in much of the medical world outside of forensic pathology). It is precisely because those practitioners had information and were able to make their decisions away from the limitations of gatekeepers that they were able to form and promulgate those opinions and engage in the appropriate intellectual discussion that has changed how many forensic pathologists view this subject.
The same thing is true for medicolegal death investigators. While much of the information they gather at a scene is done personally, they also derive information from police, emergency personnel, etc. It is neither reasonable nor useful to demand that death investigators either ignore important information at the scene simply because they are given that information by a police officer or, worse, try to act like police to get that information first hand and interact with dangerous people in dangerous situations for which they are not trained. A more correct solution would be better training in how to weigh and interpret the data they are given, not enforce blind ignorance. Recognized certification of medicolegal death investigators can provide confidence in the competence of obtaining and evaluating information relevant to the death scene.
Enforcing strict orthodoxy through gatekeepers and limitations on information will decrease, rather than increase, this intellectual progress. Instead, it will make diagnosis a static process based on bureaucratic, rather than scientific criteria.
Ignoring History and Scene Information Causes More Problems than it Fixes
An example of the disaster of an investigation free of the “cognitive bias” effects of history may be illustrative. Because this is an ongoing investigation, details have been changed. Last month, one of the authors of this manuscript was approached by an investigator who had reopened an investigation into a death from years ago. A clerk had been rearranging boxes containing old cases and saw a folder that said “Rape-Homicide/natural causes.” She was struck by that combination and opened the file.
The cognitive-bias-inducing story is as follows. In the early 1970s, the body of a young African-American woman was found in the woods. She was nude and her body had been posed with her hands above her head and her knees flexed and legs externally rotated to expose her genitalia. There were drag marks from a path to the location of the body. The scene had been documented with photographs, and the case labeled “Rape-Homicide.”
The body was delivered to a pathologist at a local medical school, and an autopsy was performed. The autopsy pathologist was provided no history whatsoever. The police acted as “gatekeeper” and “case manager” in the cognitive bias sense, and “protected” him from the “cognitive bias” associated with knowing the scene findings. At autopsy, no anatomic cause of death was noted, as often happens in cases of asphyxia (a common method of killing in rape/homicides). Accordingly, the autopsy was signed out as “unknown causes.” The next day, the police declared the case to be a presumed natural death, and the case was closed.
When the pathologist, who recently retired, was informed of the case he said he remembered it. He was under the impression this was simply a sudden death at home. Had he known the circumstances, he would have signed it out as a probable asphyxiation with a manner of homicide. The case has been reopened and an exhumation is planned. The white police officer who investigated the case, recently released from prison for the attempted rape of a different African-American woman years ago, is also still alive. He claims that there was testimonial evidence from a third party that the woman had died suddenly while engaged in sex in the secluded spot, though there is no documentation of that testimony.
Because of the soil conditions in the area, it is unlikely that an exhumation will be helpful, but the effort will still be made. It is also unlikely that the person who raped and killed this young woman will be brought to justice – all because the pathologist was “protected” from history by the police “gatekeeper.”
This is common. The biggest problem is that there is not enough history, not that there is too much.
Manner Determination and Determination of Cause of Death Should Not be Conflated
Manner Determination Has no Place in Court
A common criticism of medicolegal death investigation centers on so-called “errors” in manner determination. Such claims, brought forward primarily by those in the legal profession, ignore what manner determination is, and why it is done. Manner determination is not a legal determination. It is a public health classification for statistical analysis. It is absurd to pretend that manner determination has inherent legal meaning, and it is a misuse of manner to act as if it does.
Manner determination is a task peculiar to medicolegal death investigation systems in the United States (and systems derived from it). It came about at the beginning of the 20th century to allow the collection of aggregate statistics from death certificates for public health purposes. Because of this, the “rules” for determining manner of death are vague and vary among offices.
There is heterogeneity as to whether or not “intent” is important in homicide determination. There is heterogeneity in the degree to which supportive information is necessary to make a determination of suicide. There is heterogeneity in the degree of certainty needed to make a determination. Manner determination integrates social, policy, and cultural conventions. The same case might well receive a different manner of death determination depending on the jurisdiction and the training tradition of the pathologist.
For example, in some jurisdictions, such as New York and San Francisco, an automobile collision that results in death will be classified as “homicide” if the driver is found to be drunk. The thinking is that driving drunk represents such severe recklessness that it is indistinguishable from a volitional act of killing. In Georgia or North Carolina, automobile-related deaths are classified accidents unless the car is intentionally used as a weapon. In New York, a case is normally not classified as a “homicide” unless the pathologist is certain “beyond a reasonable doubt” that it is a volitional killing. In other areas this standard is considered an inappropriate mixing of legal and medical concepts, and the standard of “most likely” or “preponderance of evidence” is sufficient. In some jurisdictions, “most likely” is the equivalent of “greater than 51% chance” (however subjectively the percentages are estimated), while in others it is merely a plurality: if the likelihood is 45% homicide, 30% accident, and 25% suicide, then it is classified as “homicide.”
Because it is a matter of weighing information for which there may not be much certainty, virtually every serious discussion of manner accepts that in many cases there is no “right” or “wrong” answer. As the widely recognized “Guide to Manner Determination” notes:
It must be realized that when differing opinions occur regarding manner-of-death classification, there is often no “right” or “wrong” answer or specific classification that is better than its alternatives. When promulgating guidelines, however, one of the available options needs to be selected as the one recommended for use. Thus, the recommendations herein are ones selected to foster a consistent approach amongst certifiers, not because the recommended approach is the “right” or the “better” one (6).
Structurally, manner determination is arbitrarily limited to only five options: homicide, suicide, accident, natural, and undetermined. As one might expect, many cases don't naturally fall into one of these classes. Where do therapeutic complications and medical mistakes fall? Are they “accidents” or “natural?” What about so-called “parasuicides” where people engage in activities that are gestures, but may result in death?
What about so-called “misadventure” when people engage in very dangerous behavior because the very danger is addictive? Even homicide has problems translating into some “legal” corollary.
A few years ago, a survey asked practitioners about how much certainty was necessary to call certain manners (7). In that study, one group of respondents indicated that 51% or above was sufficient for the determination of homicide, and another group indicated that 95% or above was necessary. The author wrote:
Given the uncertainties involved in the determination of intent (or even volition), it is not surprising that most medical examiners accept that their determinations of manner are made with uncertainty. As the survey demonstrates, medical examiners show a wide range in the uncertainty they will accept in a manner determination. A demand for absolute certainty, or even “reasonable medical certainty” (however defined) would result in the classification of “undetermined” in all but a few cases. While some might view this heterogeneity with distaste, it may reflect a practical response to local needs…(7).
The use of manner in court varies from jurisdiction to jurisdiction. In some states, it is routinely admitted into court. In many others, it is not. In North Carolina, for instance, it is the policy of the Office of Chief Medical Examiner that manner not be included in an autopsy report (though it is placed on the death certificate) because including manner may preclude acceptance of the report in court.
It Doesn't Matter Whether or Not Manner Determination is “Correct” in Most of the Cases Where the Legal Profession Complains
That heterogeneity is accepted, however, because statistically it doesn't matter. Many critics forget that in almost all cases the manner is trivially obvious and unquestioned. In fact, most determinations of manner are natural and are certified by the clinician who treated the decedent. Only a relatively small proportion of deaths are medical examiner or coroner cases; only a very small proportion of that small proportion display any ambiguity; in only a very small proportion of that very small proportion is that ambiguity not dealt with by some policy (as with drunk driving deaths). While these policy-based changes can have effects on statistical analysis (as is occurring in the sudden infant deaths/sudden unexpected infant death arena), this small number of ambiguous deaths does not. Thus, if every single case where the manner is disputed had been incorrectly determined by the medical examiner or coroner's pathologist, it would be statistically irrelevant. From the perspective of the purpose of manner determination, it simply does not matter whether or not some individual case in litigation is determined incorrectly.
It turns out, however, that we don't get every single such case implausibly wrong – even in the absence of any ground truth or gold standard. One measure of our success is that numerous groups take manner determination and use it for what are essentially off-label uses. Insurance companies use it for benefit determination. Lawyers introduce it into court. Families use it for closure. Newspapers print it on the front page. That's flattering to the profession, and some forensic pathologists enjoy opining about manner in court. However, it is inappropriate to ignore the actual purpose of manner and claim that manner determination should be changed so that it can be misused more egregiously.
The answer is thus not to “fix” a determination that is not broken, but instead to stop pretending it is something it is not. Accusations of “cognitive bias” in manner determination criticize it because it does not work well as a tool for a task it was not designed for. It is essentially the complaint that a hammer makes a bad saw blade because it doesn't have sharp enough teeth. It would be better to acknowledge what manner determination is, how it is appropriately used, and evaluate it accordingly.
Issues of “Cognitive Bias” in Manner Determination Should Not be Used as Accusations of “Cognitive Bias” in Cause of Death Determination
Claims of “cognitive bias” in medicolegal death investigation often seem to conflate manner and cause of death determination. The methods, standards, and criteria for these are completely different. It is inappropriate to make accusations of one on the basis of misperceptions of the other. However, we often see supposed errors in manner determination used to attack determination of cause of death. This is simply wrong, and, frankly, an example of “sagecraft” as described below.
Most Errors in Forensic Pathology and Death Investigation are Not those of Simple Cognitive Bias
While cognitive bias is an important issue, most of the issues raised regarding medicolegal death investigation in publicized complaints are something else. It is clear that many agendas are being wrapped in the flag of “cognitive bias.” Those who wish to regulate how pathologists practice medicine by claiming pervasive and rampant “cognitive bias” might well benefit from reviewing their own perspectives.
Errors in Scientific Orthodoxy Do Not Constitute “Cognitive Bias”
History is filled with scientific “facts” and beliefs that have not held up under scrutiny, from dietary guidelines to classical physics. One recent study by Ioannidis points out the inevitable high rate of error in “good” scientific literature, simply due to the basic mathematics of type 1 and type 2 errors. In his work, he proposes that up to 50% of the peer reviewed literature is simply wrong (8). Worse, “peer review” has been shown to be almost useless in vetting studies, since almost no peer reviewer actually reperforms the study, but primarily looks over manuscripts for internal consistency and acceptance of scientific norms. In one study from 2011, the results of two-thirds of peer reviewed studies could not be reproduced (9).
It is not surprising, therefore that scientific ideas that were commonly accepted a few years ago come under attack and are modified or discarded. These errors do not constitute “bad science” nor do they constitute “cognitive bias.” They constitute the natural evolution of scientific knowledge. Further, neither accepting nor questioning scientific orthodoxy is evidence of “cognitive bias.”
Scientific Disagreement is Not Cognitive Bias
In this context, it is no surprise that there is some disagreement on how much to accept any given study or any given set of studies. It is unfortunate that this real scientific disagreement is so often cast by advocates of one side or another in a controversy as being evidence of incompetence, ignorance, or, more recently, having “cognitive bias.” This is most obvious in the discussions of so-called “shaken baby syndrome.”
However, casting this disagreement as “cognitive bias,” while attractive as a label for use in the courtroom or in the political or regulatory arena, is simply inappropriate. If simply having an opinion counter to current orthodoxy constitutes “cognitive bias,” then those who originally questioned the diagnosis were the heterodox physicians early on. Were they engaging in “cognitive bias” because they discounted the studies supporting shaken baby syndrome and had more confidence in those studies that were critical? Or is it only those who discount the studies that do not support shaken baby syndrome and have more confidence in the studies that support the diagnosis who engage in “cognitive bias?”
The application of the label “cognitive bias” seems to have little to do with actual science, and more to do with advancing a particular perspective by labeling and silencing scientific discourse. There is a saying that floats around political Internet discussion groups that “An ideologue is someone who disagrees with me and is insufficiently apologetic.” It sometimes seems that a similar definition applies to cognitive bias: “Cognitive bias occurs when someone disagrees with my scientific belief and is insufficiently apologetic.”
That's too bad, because this kind of skeptical dialectic is how science actually advances. It is not through government-imposed orthodoxy. It is through conflict. Karl Popper, the father of the so-called “scientific method,” envisioned science as a natural selection of ideas (10).
Richard Feynman, the Nobel Prize-winning physicist once wrote about this process. He noted:
It is our responsibility not to give the answer today as to what it is all about, to drive everybody down in that direction and say: “This is a solution to it all.” …I believe, therefore, that although it is not the case today, that there may some day come a time, I should hope, when it will be fully appreciated that the power of the government should be limited; that governments ought not to be empowered to decide the validity of scientific theories, that it is a ridiculous thing for them to try to do…(11).
The current political climate is not supportive of this view, and we have decided that it is the purpose of the government to determine the validity of scientific theories. The wisdom of this is beyond the scope of this editorial. However, it is unfortunate that we are now attempting to silence real scientific discourse by invoking claims of “cognitive bias.”
Ignorance and Incompetence are Not Cognitive Bias
A common problem in scientific debate is that some knowledge is old, some is unproven, and some is incomplete. The vast volume of the scientific literature makes it almost impossible for any given person to be fully aware of all of the literature on any given subject. This issue is not limited to forensic pathology and medicolegal death investigation.
In individual diagnoses, a pathologist is necessarily limited by her or his knowledge base. Because of the changes in literature, there is simply no regulatory way to ensure that every pathologist has all information available for all issues. Certification examinations and all of the contrivances of Maintenance of Certification simply cannot do this – which is why no certification process actually claims to ensure competence.
There are profound barriers, however, between forensic pathologists and the current literature. Most medical examiners are not part of an academic institution, and thus do not have access to medical school libraries or broad online journal access. This results in a profound asymmetry of information. Pathologists without easy access to the literature must rely on what they were taught in medical school, residency, and fellowship, minimally augmented by conferences and occasional discussions. Current copyright limitations make sharing of articles difficult.
Moreover, many studies involve disciplines other than medicine. In the shaken baby controversy, it is often the case that neither those physicians that tout biomechanical studies nor those who discount them are competent to critically read them. Thus, the arguments about these studies reduce to arguments from authority. Similarly, some of the authors of the biomechanical studies have, in the past, shown profound misunderstanding of clinical information and a surprising misuse of statistical analysis.
Younger pathologists may have better access to literature, but lack practical experience, which is of paramount importance in a field that relies heavily on perceptual cognition. A recent study, accepted pending minor revision for publication in the Journal of Forensic Sciences, has shown a surprising heterogeneity in diagnosis of patterned injuries from photographs. A follow-up study, which has not been submitted, demonstrates that there are significant differences in how young and old pathologists explain why they differed from the consensus response. Younger pathologists were much more likely to state that they simply did not recognize the lesion. Older pathologists were more likely to state that the consensus was incorrect and their response was more correct; interestingly, they were often right.
Often, medicolegal death investigators will not be given complete information by police or first responders. They may not be given complete access to the scene. While death investigators are trained to weigh the information they are given, it is unfortunate that they are sometimes kept in the dark regarding facts in a case. However, the solution is to increase the independence of the investigator and his or her ability to obtain and weigh that information, not hide more information from her or him.
It is not surprising that many people, particularly retired consultants with minimal access to the literature, take positions that are out of date or that do not reflect an intimate knowledge of the literature, or that younger pathologists rely too heavily on things they have been told rather than their own perceptual skills. It is not unreasonable for death investigators to present the information they have, rather than the information that was denied to them. However, drawing reasonable conclusions based on inadequate information or insufficient experience is not “cognitive bias.” Worse, the “solution” of hiding yet more information as proposed by some will exacerbate, not ameliorate this problem. We should be working on how to get more information, both of scientific information and case information to the investigators, not less.
In addition to a lack of access to the literature, medicolegal death investigation is often performed by people who are not adequately trained to begin with. While it is clearly absurd to deny the importance of board-certification in the practice of surgery, the importance of certification is often ignored in medicolegal death investigation, whether it be board-certification for forensic pathologists or appropriate certification for death investigators and coroners. Many of the valid complaints made against those who make inappropriate determinations can be explained by the fact that the determinations were simply made by unqualified people. Assertions of “cognitive bias” do not address this severe and ongoing problem.
Unethical Behavior is Not Cognitive Bias
The unfortunate fact is that there are forensic pathologists who will modulate their reports to please counsel. This is not supported by the profession, and we believe that it is very uncommon (though there is no metric for it). While this may be considered cognitive bias in the large sense, it can also be argued that it is not in the sense that it is not unconscious. More important, a person who sells his or her word will not be affected by data hiding.
Political Pressure May be Cognitive Bias, But it is Not Addressed by Linear Sequential Unmasking
In contrast to the kind of cognitive bias that would be addressed by “linear sequential unmasking,” a real issue among forensic pathologists and death investigators is the issue of independence. In one survey of forensic pathologists, 21% of pathologists working in coroner offices and 4.5% of those in medical examiner offices were directed by their superiors to change their findings. Approximately 10% were asked to sign documents such as reports that did not reflect their findings in an autopsy. Eighty-two percent had experienced pressure from stakeholders to change a finding, and 22% had suffered political pressure to change a diagnosis. Approximately one-quarter of those who resisted these pressures suffered a substantial consequence of their refusal. In another study, approximately 13.5% of respondents admitted to modifying their findings as a result of threat of litigation. Forensic pathologists have been threatened with termination because their testimony was favorable to the defense (12).
This is a real and continuing issue. Unfortunately, “linear sequential unmasking” will not deal with it.
Standards Should be Subject to the Same Kind of Scientific Rigor that Scientific Processes are Subject to
There is no large study that demonstrates that “linear sequential unmasking” is effective or helpful in forensic pathology. It may, for the reasons noted above, be harmful.
A paradox in the regulation of science is that those who would institute standards in order to increase the “scientific” validity of work often decline to subject the standards themselves to the same rigor. There is no scientific study that demonstrates that “cognitive bias” is a severe or driving problem in medicolegal death investigation. It is not enough to show that “cognitive bias” (however broadly and poorly defined) exists in some experimental situation. It needs to be shown that it is an actual and significant problem in practice. If it is not, then perhaps we should focus on those issues that we know are of immediate and significant importance. More importantly, there is no evidence that schemes such as “linear sequential unmasking” will fix this unproven issue even were it a significant problem. In the case of linear sequential unmasking, there is essentially no literature that shows that it would be of use. Instead, it seems that the literature is largely of people quoting each other's speculations that it should work.
Linear sequential unmasking, gatekeepers, and such may be of use in some disciplines, particularly those that do not involve any integration of information. However, before imposing those requirements on disciplines that do involve such integration, it would be useful to demonstrate its scientific validity. Forensic science is in need of improvement. That is not a reason to use it as an experiment of nature for schemes that have no substantial scientific bases themselves.
Before imposing an untested hypothetical solution to an unmeasured problem, it would perhaps be useful to measure the problem and test the solution.
It would be interesting to perform schemes such as “linear sequential unmasking” in the forensic pathology discipline, and the federal government might consider funding such a study in some limited jurisdiction before imposing it on the nation at large. However, in doing so, it will have to implement some way of indemnifying that jurisdiction from the disasters that we believe will be associated with it.
The linear sequential masking paradigm is unworkable for forensic pathology.
The Use of “Gatekeepers” and “Case Managers” Implies a Uniformity of Judgment that is Not Present
The concept of “gatekeepers” and “case managers” is based on the belief that every person's inferential network is the same. As noted by Schum above, this is a false assumption in nontrivial real-world cases. It may be true in trivial cases where no inferential judgment is actually used, but that is not the case in forensic pathology. The use of a “gatekeeper” in such a case does not “protect” the pathologist from “extraneous” information. It simply substitutes one person's judgment of what is or is not important for another person's. Worse, because each person's inferential chain is different, it suggests a high likelihood that neither will work correctly. The gatekeeper will not really be able to impose his or her inferential processes onto the pathologist, and the pathologist will not be able to adequately evaluate the case because he or she is denied information he or she needs.
The Iterational Model of Analysis is Not Practical in Autopsy Pathology
A paradigm that has been suggested for forensic pathology is that the person performing an analysis will first perform the analysis blind. Then he or she will ask for information that the gatekeeper will then decide to provide or not to provide. If provided, he or she will then repeat the analysis and note how the information will affect the process. This will be repeated an arbitrary number of times (13).
This is nonsensical for a death investigation. First, of course, one only gets one uncompromised scene investigation and complete autopsy. Once the scene is processed, it is released. Once the autopsy is completed, the body is returned to the family. One cannot hold a body for months to allow iterative autopsies. More important, the autopsy is destructive. So-called “second autopsies” only provide a small amount of information. While it is possible in some cases to go back and do new stains on histologic sections, there are a number of tests that cannot be done unless there is a suspicion of a particular diagnosis at the time of initial prosection.
A classic example is that of a tension pneumothorax, in which air is under pressure in the chest cavity. Once the chest cavity is opened, the air escapes. It cannot be put back in, and if one does not do a test for it at the time of autopsy, it will likely not be noticed. Unfortunately, it is not possible to do all conceivable tests in every autopsy both for monetary and logistical reasons. Some tests are mutually exclusive and one much choose to do one at the expense of another. One cannot procure eyes for tissue donation, extract vitreous for toxicology testing, and preserve the eyes for careful sequential sectioning and examination for hemorrhage. One must choose one or two, but not all three.
Worse, it is not practical to argue that the autopsy pathologist ask all possible questions in all possible cases before performing the autopsy. There are simply too many, and too many variations on how questions cascade.
The Use of Gatekeepers Will be Extraordinarily Expensive and Will Result in Sloppy Errors
As discussed below, the usefulness of peer review can be diminished when pathologists are given too heavy a load of work to review. There is no reason to believe that the same will not be true for a “gatekeeper” or “case manager.” Theoretically, the case manager will review the case records – sometimes consisting of hundreds or even thousands of pages of material – and then choose just those things that are important to the case. This is unlikely, because the gatekeeper will have his or her own cases to do. Instead, those cases will be quickly reviewed and snap decisions will be made regarding what will and will not be allowed to the investigating pathologist. And the investigating pathologist, who, unlike the gatekeeper, will be motivated to find the details, will not know what gems might be hidden there. This will result in significant error, and it will further overburden an already taxed system that has far too few active board-certified forensic pathologists.
A recent example from the private consultation practice of one of the writers of this response again comes to mind. In a medical malpractice case, a woman fell from a wheelchair and suffered a fracture of the spine. She became less and less mobile, developed a urinary tract infection and then sepsis, and died of septic shock. The family sued the caretakers.
When the writer was retained as an expert by the defense, he was presented with approximately 2500 pages of medical and historical information. Hidden within that data were two pages that mentioned that the decedent suffered a sudden loss of mobility that in turn contributed to a urinary tract infection that in turn contributed to fatal sepsis. Had he not carefully looked at every one of those 2500 pages, it would have been missed. But he did, because it was his case. Now, consider a gatekeeper who has no particular interest in the case. Is it really proposed that he or she would search through those 2500 pages as diligently as the person whose signature is on the report? On a daily basis? While doing his or her own 250 cases per year?
This will not happen. The gatekeeper will act as many peer reviewers do: perform a superficial investigation and leave it at that. The real investigator will be left adrift.
There are Paradigms that Have Been Demonstrated to Work in Medicine. Why Not Use Them?
There are multiple paradigms that have worked in medicine. Peer review is the most obvious. It may be appropriate to use those that have some supporting scientific and literature basis rather than those that do not.
Peer Review Has a Mixed Record, But at Least it Has a Record
It is important that one of the problems of peer review is that it institutionalizes common biases. A common complaint among heterodox scientists is that they cannot get published because of biases towards orthodoxy among peer reviewers. Criticisms of peer review abound, arguing that it is biased, slow, expensive, arbitrary, capricious, and inconsistent (14). In contrast, peer review when done under appropriate circumstances by appropriate people can increase the value and organization of manuscripts.
Peer review also is subject to the issue of incomplete information. In one case, for instance, a “blue ribbon panel” of pathologists was retained by a jurisdiction and asked to provide a critique of the work of a pathologist. The panel found numerous severe deficiencies. In the medical examiner's response, he noted that the jurisdiction had acted as a gatekeeper and had failed to provide much of the information that addressed the concerns of the panel. One of the primary flaws of peer review is that the reviewers do not replicate the studies, but take them on faith, instead looking for errors in writing, concept, and consistency. Since many of the problems involve the actual acquisition and analysis of data, many issues are not noticed because these are the very tasks that are not performed by the reviewer.
Peer review sometimes substitutes the cognitive bias of the group with the cognitive bias of the individual. Nonetheless a good argument can be made that peer review represents a reasonable way of providing the “gatekeeper” function of looking for the improper integration of external information by following the inferential chain of the pathologist. The primary difference is that it is performed before the report is released rather than before it is created.
Peer Review at the Forensic Pathology Level Can Avoid Some of the Known Pitfalls
More important, while peer review has its flaws, they are known flaws, which is substantially superior to imposing a scheme that has been subjected to no scientific evaluation in the field of forensic pathology whatsoever. Peer review at the forensic pathology case level can avoid the primary pitfall of manuscript peer review in that a careful peer review will review the photographs of the case, the histology, the primary data such as first responder reports, etc.
Peer Review Will be Expensive
Peer review, if done fully, is expensive and time-consuming, which is why most traditional peer review fails. For the purpose of further discussion, such extensive peer review will be called “deep” peer review. It takes approximately eight hours for a forensic pathologist to perform an autopsy and generate a report on average. Depending on the depth of the peer review, it may take as many hours for the reviewing pathologist(s) to evaluate the results. It is not clear that there are sufficient funds or personnel to perform this function.
In the experience of the writers, when peer review becomes too time-consuming, the depth of the reviews decreases. Review of the histologic slides may be skipped. The external records may not be reviewed. At worst, it becomes a simple proofreading to look for copy-editing errors and gross inconsistencies. But, still, even at its worst, a superficial peer review would be less disastrous than sloppy gatekeeping that denies a pathologist important information that he or she needs.
It Would be Possible to Establish Protocols that Would Facilitate Deep Peer Review
The primary issue with manuscript peer review is the lack of the ability to examine and analyze the original data. A similar issue in forensic pathology can be minimized by establishing standards for photographic documentation of injuries and explicit description of inferential chains. This would allow a peer reviewer or peer reviewers to evaluate data as close to the primary data as possible. A formal description of how that data is integrated into the conclusion would serve the same purpose as the descriptions that would theoretically be obtained with linear sequential unmasking.
Deep Peer Review Might Address Many of the Other, Better-Demonstrated Problems with Death Investigation
The major problems with death investigation do not revolve around “cognitive bias” per se, but the other issues detailed above. In contrast to linear sequential unmasking, deep peer review would address these as well. Deep peer review that involved more than one person would decrease the risk that some fairly well known forensic scientific knowledge was ignored. It would decrease the risk of unethical conclusions. It would help with issues of heterodoxy and in the acknowledgement of areas of contention. Peer review that involved reviewers outside a given jurisdiction may well ameliorate issues of independence. It would, unfortunately, not address areas where scientific orthodoxy is wrong, but neither would data hiding schemes.
Any Proposed Solution Should be Supported by Scientific Studies
Any proposed intervention must be supported by scientific data that examines its benefits and possible negative consequences. There are problems in medicolegal death investigation, and it can be improved. Proposed “solutions” that will likely introduce more problems, rather than solve them, should not be imposed without a firm scientific basis that shows that they have a real expectation of success.
Conclusion
In conclusion, there are significant challenges to the practice of good medicolegal death investigation. Cognitive bias is ubiquitous, and is by definition one of those challenges. However, the degree to which it is a significant issue is not scientifically determined. In contrast, many of the other issues are well known, and we should focus on these issues. It is inappropriate to wrap agendas regarding specific issues such as shaken baby syndrome in the flag of “cognitive bias” and attempt to achieve a propaganda rather than scientific victory on that issue.
In 1940s, the Polish sociologist Florian Znaniecki wrote a landmark book on the sociology of scientists called “The Social Role of the Man of Science” (15). In that work, he provided one of the first non-Marxist evaluations of how scientists act in society and among themselves. He discussed various roles that scientists take on, such as classifiers, popularizers, etc.
Znaniecki contrasts two seemingly similar but very different roles as that of the “fighter” and the “sage.” The “fighter” was a scientist who defended a particular scientific position against those who posited a different theory. The dialectic between the “fighters” resulted in a synthesis that advanced scientific knowledge. The “sage,” in contrast, focused on using “science” to defend a particular social policy. Often, they were hired by political parties or social activists to provide a patina of “scientific” validity to their position. Sages perverted science in order to advance their social agenda, by denigrating the work of others and ignoring contrary data, by overstating data that supported their agenda, and by making personal attacks on scientists with whom they disagree.
Making a shibboleth of “cognitive bias” in the service of these agendas does a disservice to the real science and real advances that would help us deal with the issues that limit our ability to do our job well, including real cognitive bias.
Those who would use the label of “cognitive bias” to advance specific legislative, policy, or regulatory agendas should beware of leaving real science behind in that desire for that policy victory. Any intervention must be supported by scientific data that examines the benefits and possible unanticipated negative consequences of that intervention. There are problems in medicolegal death investigation, and it can be improved. Proposed “solutions” that will likely introduce more problems, rather than solve them, should not be imposed without a firm scientific basis that shows that they have a real expectation of success.
Footnotes
The authors have indicated that they do not have financial relationships to disclose that are relevant to this manuscript