Can We Trust Best Practices? Six Cognitive Challenges of Evidence-Based Approaches

Challenge 1: Characterizing problems

The notion that we can apply a rule—“If this condition arises, apply that treatment”—ignores the ability to judge whether the condition has arisen. Much, if not most, of the challenge is in understanding what is anomalous and the nature of the underlying problems (Klein, Pliske, Crandall, & Woods, 2005). Once a problem is accurately identified, selection of a treatment can be straightforward. Best-practices approaches, such as EBM, tackle the easier part of the equation—how to address each problem—and do not give adequate coverage to the detection and identification of the problem, the variability within a problem category, and the nature of other interacting problems, conditions, and treatments. As a result, they undervalue the diagnostic expertise that is essential for effective health care. For example, the following vital signs—blood pressure = 90/60, heart rate = 130, respiratory rate = 30, oxygen saturation = 88%—open an immediate list of situations of concern (sepsis, congestive heart failure, emphysema flare) that are sorted out by expertise as the physician takes into account what is the context and who is the patient, a newborn baby or an 85-year-old woman with diabetes and a cough? Clinicians use heuristics (useful tactics that gain responsiveness and robustness but fall short of optimality) and pattern recognition (requiring diverse experiences) to cope with complexity. Best practices rely on categorizations as a form of organizing knowledge. These generalizations are simultaneously valuable and limited when a practitioner is confronted with variety, time course, and interactions across disease conditions. Heuristics, pattern recognition, and other forms of expertise lose their vitality, adaptiveness, and context sensitivity when reduced to rules.

Challenge 2: Gauging Confidence in the Evidence

Many physicians would like to access a database of best practices to identify a preferred treatment regimen for a patient, but too often the choice of treatment depends on judging the quality and relevance of the evidence, judgments that often depend on experience and are made under uncertainty and time pressure.

Evidence by itself can mislead us. Witness the frequent revision of claims that had empirical support but were modified or overturned by later evidence in fields such as cancer treatment (Kaiser, 2015). In “garden path” problems, early-arriving pieces of evidence points to one hypothesis, but later, more subtle patterns of evidence reveal a different problem is actually present (Woods & Hollnagel, 2006).

Even if we ignore the problem of studies that cannot be replicated, we also have to worry about negative findings that are used to dismiss treatments that do have value. The data collection may have been flawed because of variables that were not well understood at the time. A well-known example is the set of best practices for treating peptic ulcer disease, thought to arise from excess stomach acids, likely caused by stress. When Barry Marshall explored the hypothesis that ulcers were actually caused by Helicobacter pylori, he conducted a critical experiment searching for H. pylori in the gut of ulcers victims. But the search failed. None of the patients showed any sign of H. pylori. That should have scuttled the hypothesis, but the problem was that the lab technicians were discarding the cultures after only a few days, the standard procedures they used for strep infections. Once the lab technicians accidentally gave the cultures more time to grow, the link to H. pylori was demonstrated definitively (Marshall, 2005). And the best practice shifted from surgery to antibiotics.

Some of the gold-standard sources of EBM have proved to be misleading—the Framingham Heart Study was performed focusing only on White males. However, many of the key symptoms for differentiating a heart attack from other diseases (i.e., “feels like an elephant sitting on my chest”) are found in only 5% of women. After one emergency physician realized the consequences of these limitations, she explained her horrified reaction: “I think about all the women I sent home to their deaths because I was following the algorithm for best practices for chest pain” (personal communication, April 6, 2012).

Excessive faith in evidence by itself can also literally blind people (Chatterjee, 2015) and lead to fixation on one view, which impairs the search for and utilization of new, emerging, alternative indicators. The 1981 Nobel Prize in Medicine, given to Torsten Wiesel and David Hubel, focused on their finding of a critical period of neuroplasticity for vision. On the basis of this research, clinicians stopped performing corrective surgery for children born with cataracts if the intervention occurred after the age of 8. However, a researcher in India decided to challenge this dogma. When children’s cataracts were removed and artificial lenses inserted, vision was at least partially restored. The original Nobel Prize research, performed on primates, had led the medical community astray.

Examples like these show that the implications of evidence are not clear-cut, and the search for evidence is ongoing. Researchers may be unaware of the variables that are obscuring the relationships they are studying. As a result, the medical community has to be continually prepared to revise its faith in evidence that may seem solid today. Increasingly, researchers are appreciating the value of being uncertain (Timmermans & Berg, 2003). This is a critical finding in cognitive engineering—performance depends on how well people, teams, and organizations revise assessments as new evidence comes in (Woods & Hollnagel, 2006). A clear-cut example occurred in the run-up to the Columbia space shuttle disaster as managers discounted new evidence about new kinds of safety risks (Woods, 2005). EBM appears to offer a reassuring set of stable best practices, but in reality, we find that yesterday’s best practices become questioned and eventually revised or even discredited. Although once established, these practices may take a long time to be modified or discarded even when new evidence builds up.

Challenge 3: Deciding What to Do When the Generally Accepted Best Practices Conflict With Professional Expertise

This conflict is an issue because the professional judgment of physicians is credible, and their expertise and intuitions need to be taken into account. Physicians build up expertise and acquire pattern repertoires over years of experience. When they have ample opportunities for feedback about their judgments, their intuitions—their use of experience to make pattern-based judgments—are valuable (Klein, 1998). Kahneman and Klein (2009) assert that intuitions are useful under two conditions: a reasonably stable environment and an opportunity for people to learn from feedback. For example, the stock market does not constitute a reasonably stable environment, and we are highly skeptical about claims of expertise or intuition in selecting stocks for investment. Another example is organizational decision making. Most people who work on the administrative side of organizations fail to get frequent, consistent, or accurate feedback and thus fail to develop expertise and credible intuition.

In contrast, medicine satisfies both of the conditions for credible intuitions. It is a reasonably stable environment, and physicians do get some feedback. Intuition here is not a random or mystical process but simply the use of pattern matching that is based on experience in a reasonably stable environment.

Admittedly, physicians do not achieve the levels of expertise found in chess grandmasters. Chase and Simon (1973) described how chess grandmasters accumulate tens of thousands of patterns that enabled them to rapidly size up situations. Chess is a highly stable environment—the positions of the pieces are unambiguous. And chess players receive clear feedback about the quality of their decisions. They know that they have won or lost a game and can go over the moves to determine where they made mistakes. Physicians do not receive the same level of feedback on their decisions. When they refer patients to specialists, they may not be informed about the results. Worse, feedback is abundant on common conditions but limited on rare conditions, such as early rabies. And worse yet, not all feedback is equal. Losses loom larger than successes. Ghaffarzadegan, Epstein, and Martin (2013) have shown that a false negative (e.g., failing to do a needed C-section) will have a greater impact than a false positive (doing an unnecessary C-section). That is why physicians are sometimes cautioned not to trust their intuition. Indeed, this warning is part of the rationale for EBM.

We are not advocating for physicians to uncritically trust their intuitions. Rather, we see the importance of taking judgments and intuitions into account when making decisions, which will occasionally generate a conflict between best practices and professional judgment. Even though medicine is not as stable and well structured a domain as chess, and does not permit the same quality of feedback, it is not a zero-reliability domain, like stock selection. Experience lets skilled physicians identify reasonable courses of action and reasonable treatments upon diagnosing a condition. These intuitions can sometimes be misleading, which is why they need to be checked by deliberately scrutinizing the conditions and likely consequences of different actions. Still, in complex situations, heuristics and pattern recognition are essential and cannot be replaced by sets of rules (DeAnda & Gaba, 1991; Wears & Schubert, 2015).

Challenge 4: Applying Simple Rules to Complex Situations

Problem solving is applied to both well-ordered and complex situations. Well-ordered situations are highly stable and are easily captured as a set of procedures. Insertion of a central line is a well-ordered situation in which a checklist approach has markedly reduced line infections (Pronovost et al., 2006). A central line is inserted into a large vein to deliver blood, medications, fluids, or nutrients for an extended period of time. In situations like this, there are fairly unambiguous tasks to be accomplished and clear criteria for success.

Complex situations, in contrast, contain many variables that must be taken into account. These variables all interact and vary over time as the patient’s status changes. Further, a patient may be suffering from several different problems, making the patient’s current status difficult to assess. A patient may have both asthma and diabetes. Severe asthma requires the use of steroids, but the steroids will drive up blood glucose. The physician cannot rely on either an asthma protocol or a diabetes protocol alone but will have to take both conditions into account, performing trade-offs that also reflect the characteristics and lifestyle of the individual patient. Much of health care involves wicked problems (Rittel & Webber, 1973) that do not have an unambiguously correct solution, a right treatment—but even so, some solutions are better than others.

Rules and evidence are about populations, but physicians have to treat individual patients. The evidence may focus on average patient response data, downplaying the distribution and ignoring considerable individual variations. A given drug may be ineffective when averaged over an entire sample but may work at one of the extremes of this sample.

Physicians have to connect the general or categorical guidance of best practices to fit the individual patient in front of them. And that is difficult because the research base consists of studies that vary one thing at a time, so it is hard to adapt the findings to all of the factors present in an individual patient. The research base cannot vary all of the relevant factors without quickly running into combinatorial explosions and confounded variables.

Global rules do not necessarily apply blindly to specific patients. Patients, diseases, and comorbidities are more variable than the categories of best practices. As a result, no set of rules by themselves can completely specify how to treat individual patients under complex conditions. This finding holds regardless of whether sets of rules are organized as best practices, written as procedures, or mechanized in computers; rules are resources for considered action (Suchman, 1987; Woods, Roth, & Bennett, 1990). Skilled physicians certainly use statistically based generalization, but they also use other processes and inputs for situated cognition and action (Suchman, 1987).

One of the primary attractions of best practices is simplicity—the potential to dial in a course of treatment once a diagnosis is made. However, simplicity is also a limitation when confronting the complexity of individual patients.

Challenge 5: Revising Treatment Plans That Do Not Seem to Be Working

Health care providers often must rapidly determine when a treatment plan needs to be adapted. Plans have to be adapted for several reasons: They may not be working, complex situations are continually changing, and patient status may fluctuate. Plans are often created for these “wicked problems,” involving ill-defined goals that need to be revised. Medicine is practiced with patients who improve or deteriorate or develop additional symptoms and problems over time.

However, EBM is punctate; given these data, here are the recommendations. EBM is not well suited for plan adaptation. Practicing physicians who want to adhere to EBM have to wrestle with several challenges: the need to pick up early signs that a best practice is not producing the intended effects, early signs of expectancies that have been violated. They have to determine when and how to gather evidence to test these concerns and when to revise or withdraw a best practice.

Plan revision places great demands on expertise in understanding the treatment regimen and the individual patient so that revisions can be made quickly and effectively. Physicians need skill in revising a treatment plan without being too impatient to give it a chance to work or waiting so long that the patient’s chances of recovery are diminished. Rudolph, Morrison, and Carroll (2009) has examined the dynamic of hanging onto a plan for a while but being prepared to drop it when required. Research on dynamic and adaptive decision making can inform medical practice and be extended to enhance patient safety (Amalberti, 2013; Brehmer, 1987, 1992; Kylesten, 2013).

Woods (1994) and Woods and Hollnagel (2006) model anomaly response and plan revision based on studies in abnormal events in aviation, nuclear power emergencies, and space shuttle missions. As anomalies occur, judgments about when and how to revise a plan are difficult. Gaba, Maxwell, and DeAnda (1987) documented this need for adaptation in their study of anesthesia crises. Klein (2007a, 2007b) has found that revising a plan can be more difficult than initiating it because once a plan is under way, it can be difficult to disentangle the condition from the treatment. The patient may be getting worse or might be having a bad reaction to the medication even as the underlying disease is being brought under control. Plan revision can consist of changing a schedule to a more serious modification of adding or deleting tasks, to an even more serious step of replacing one strategy with another in order to achieve the goals, and in the most extreme cases, to revise or replace the goals themselves. Applying EBM assumes the best practice has the desired effect and fails to speak to the cognitive and teamwork issues in plan revision.

Challenge 6: Considering Remedies That Are Not Best Practices

Best-practice guidelines may not be available that cover all of the situations physicians face or may not address all aspects of a patient’s case, yet physicians still have to make treatment decisions. How should a practitioner use other evidence not derived from controlled studies that seems relevant or fill gaps for the patients they encounter? We appreciate that not all EBM advocates insist on using evidence only when derived from double-blinded studies and that in many circumstances, it will be impossible to set up this kind of design. This raises the question of the role of less rigorously documented practices. Should physicians not apply remedies unless they are part of prescribed/sanctioned “best practices” even when practice turns out to be incomplete or insufficient for the patient who is suffering now? How should physicians act now when research trials to expand best-practices knowledge may not be completed for years? An all-or-none stance on rigor leaves a gap between the development of rigorous knowledge and the need to act now when knowledge is incomplete. And what to do about meta-analyses that throw out all information from studies that fail to meet the highest levels of rigor even though the study may provide some provisional information? Studies may fail to generate significant results because of variability in the data, but that variability may reflect subpopulations that have different reactions to treatments. Thus, meta-analyses can mask different reactions among various subpopulations.

These examples identify another judgment that EBM downplays but has been studied in cognitive engineering—the sufficient-rigor judgment (Zelik, Patterson, & Woods, 2010). Although one would like to make decisions based only on the most rigorous evidence, in the real world, there are always constraints from limited resources and time pressure. This means in practice there is a judgment about what level of rigor is sufficient at that point in time given the uncertainty and the risks of acting too late or too early. There are ways to assess and assist the sufficient-rigor judgment as decision makers struggle under pressure.

A related challenge is the role of practitioners in providing evidence. Does the evidence base always fall within the provenance of medical researchers, or will data from other sources be accepted? Unexpected twists in a case, unexpected reactions, coincidences, all have contributed new hypotheses and ideas. Spectacular progress in surgery over the past century—joint replacement, cardiac valve replacement, reconstructions, and so forth—have been achieved without initial randomized control trials (O’Sullivan, 2010).

Finally, it seems foolish to ignore “rock stars”—individual practitioners who achieve much higher success rates than others. Staszewski (2004) provided an example of a rock star in the domain of mine detection. Conventional mine detectors proved ineffective in the face of a new generation of mines that relied on plastic parts instead of metal, so the Army invested $38 million over 9 years to develop the next generation of handheld mine detector. Unfortunately, tests showed that the new version was not any better than the previous version; both achieved only 10% to 20% accuracy. However, there were a few specialists, one in particular, who were able to achieve success rates greater than 90% with the new equipment. By studying their strategies, it was possible to develop a training course that allowed soldiers to jump from 20% accuracy to 90% accuracy. When there are only a few experts, or just a single expert, statistical evaluation will be impossible. However, these standouts, sometimes referred to as positive deviants (Pascale & Sternin, 2010), offer important lessons. Of course, there is a risk of drawing the wrong lessons from standouts. For example, one risk of naive copying of successful cases involves undersampling failure (Denrell, 2003; Denrell & Fang, 2010). Organizations may try to learn from other, successful organizations and conclude that bold, risky decisions are essential. However, this conclusion misses the cases of organizations that failed because they took unfortunate risks; the failures are no longer in business and unavailable for study. The Staszewski example was not naive copying of standouts but was a detailed study to understand what made the standouts successful. In contrast, “best practice” as a paradigm in health care easily slides into naive copying. The above example from Staszewski also illustrates another aspect of experience and expertise. Expertise is not expressible in the form of rule sets and so can be easily dismissed as mere intuition and unreliable. But forms of expertise, such as the recognition of patterns of relationships, can be identified, modeled, and supported (Hoffman et al., 2014).

(a) Develop and sustain expertise

In order to maintain a balance of evidence and expertise, the cognitive engineering/naturalistic decision making (NDM) community has advocated for ways to build expertise (Ericsson, Charness, Feltovich, & Hoffman, 2006; Klein, 2005; Hoffman et al., 2014). Ericsson (2004) found that the longer it had been since general physicians had completed their formal medical training, the greater the reductions in accuracy and consistency of cardiac diagnoses. That is why it is so important to apply methods for training and sustaining skilled performance in areas such as health care, aviation, intelligence analysis, and so forth. Expertise often takes the form of tacit knowledge, so training in perceptual skills, pattern recognition, anomaly detection, and mental models becomes critical. Decision makers cannot rely on explicit knowledge (facts, rules, procedures) alone. One example of how development of tacit knowledge is being supported is through the ShadowBox training (Klein & Borders, in press), which focuses on speeding the development of expertise through comparisons and introspection.

(b) Support adaptation

Expertise depends on tacit knowledge and, in particular, the use of tacit knowledge to judge when to change treatments and when to depart from best practices for individual cases. For example, in health care, we suggest replacing global prescriptions about populations with expectations that physicians will adapt treatment plans to the needs of specific patients. Physicians can shift to customized and annotated plans, not only about medication management but generally about patient treatment strategies.

(c) Combine evidence with experience

We see little benefit in contrasting evidence and experience. Both provide a basis for practitioners to make decisions. Sometimes they will conflict, and the decision maker needs to navigate through the conflict. Balancing the competing claims of evidence and experience, and the strengths and limitations of each approach, is not unique to medicine. This duality appears in many different disciplines. For example, the new Pearson Q interactive assessment tool allows clinicians to select tests to be given (Delis, 2014) but makes suggestions based on other experts (Eric Saperstein, personal communication, January 14, 2015). Experienced behavioral researchers may review the results of a study in terms of means and standard deviations but then look at individual participants for signs of anomalies and unusual patterns.

(d) Balance generic evidence with experiential evidence

There is a further conflict within evidence-based approaches: the confidence placed in general evidence drawn from populations versus experiential evidence drawn from the individual cases. In health care, EBM encourages the medical community to rely on generic evidence rather than the evidence of their own patients. However, both types of evidence seem important in making treatment decisions. There are several ways to address this. Gigerenzer (2002) has used frequency data to present generic evidence, with the goal of contextualizing the generic evidence. Another approach is to make better use of displays that illustrate the various parameters and context of a situation to allow for greater resilience (Hollnagel, Woods, & Leveson, 2006; Nemeth, O’Connor, Klock, & Cook, 2006).

(e) Represent evidence

Publication of scientific papers is not sufficient. We need to provide data in more easily digested forms to help decision makers see how to personalize the findings to specific cases. Additionally, we advocate ways to offer clearer presentation of effect sizes and clearer presentation of variability, even speculating about the clusters of study participants who gained the most and the least. Using the language of statisticians, we are advocating for ways to highlight Subject × Treatment interactions.

(f) Appraise evidence

Much of the evidence we believe in today will be discarded in 5 to 10 years’ time. Decision makers cannot blindly accept the latest studies. They have to gain skills in judging how much confidence to place in evidence. At least, they have to be able to determine that a small difference, although statistically significant, may not warrant too much confidence. The cognitive engineering/NDM community has encountered challenges about appraising evidence with populations, such as intelligence analysts, who are always alert to the possibility of accidental, erroneous, or even deceptive data points. Panel operators of petrochemical plants need to constantly be alert to the possibility that sensor data may be erroneous. They are trained on garden-path scenarios involving flawed evidence and mistaken initial assessments.

(g) Share evidence

Expanding the use of information-exchange mechanisms will help the entire medical community learn about new treatments that have not yet been vetted by the standards of best practices. Several communities have made powerful use of information exchanges, for example, the informal and impromptu lessons learned from chat rooms that sprang up during Operation Iraqi Freedom to trade observations about topics such as detecting improvised explosive devices. The health care domain has established the Patient-Centered Outcomes Research Institute, tasked with sifting data from electronic medical records to identify promising therapies that could make a valuable contribution without having to rely on double-blind experiments. With appropriate oversight, synthesis, and medical legal protection, such forums would provide an opportunity for a broader dialogue on how to balance best practices and expertise.

(h) Support collaborative decision making

Cognitive engineering/NDM researchers pay a great deal of attention to effective teamwork. Best practices should not be narrowly drawn as the province of the decision maker. The decision maker has to establish trade-offs and coordination with team members. For example, within health care, the team includes physicians, nurses, and various other professionals. The team concept should be broadened to include patients. A best practice has little value if the patient is unable or unwilling to adhere to the regimen. Physicians can blame the patient, but a more useful stance is to take the patient’s abilities and motivations into account in designing a treatment program, even if it means departing from the generally accepted best practices. A suboptimal regimen that a patient can sustain may be better than an optimal one that the patient will ignore. Health care professionals can use what we have learned about adherence (e.g., D. Klein, 2009) in designing individual treatment programs.