Quality and Efficiency of the Clinical Decision‐Making Process: Information Overload and Emphasis Framing

Abstract

The healthcare industry has invested heavily in electronic health records and other clinical information systems in order to improve caregivers' access to information and ability to share information with other care providers. It has been shown that these systems can readily induce in their users a state of information overload, where the volume and complexity of information overwhelms the user, leading to lower decision speed and quality. This research introduces and tests a cognitive technique called “emphasis framing” as an operational tactic to help mitigate the effects of information overload, thereby improving the quality and timeliness of clinical decision‐making. Emphasis framing is the highlighting or stressing of some aspect or component of the information being exchanged in order to make it more easily processed, or more likely to be processed, by the recipient. We conducted a controlled laboratory experiment with emergency department physicians experiencing information overload to measure the effect of emphasis framing on two operational performance metrics: (1) the quality of the physician's clinical evaluation, and (2) the efficiency (timeliness) of the physician's clinical decision‐making. Our findings show that the emphasis frame helped mitigate the effects of information overload and increased the quality of clinical decision‐making. Contrary to expectations, however, we found decision‐making took longer with the emphasis frame, reinforcing the need to consider the impacts of quality/speed trade‐offs. Implications for theory and practice are discussed.

Keywords

behavioral operations healthcare decision‐making experiment

Introduction

In September of 2014, an American who had just returned from West Africa arrived at a Texas emergency room complaining of various flu‐like symptoms. The patient informed the nurse of his recent trip and it was documented in the electronic health record (EHR). Upon review of the EHR, the attending physician failed to notice the travel history documented by the nurse and the patient was treated and released from the hospital that day. Dr. Daniel Varga, Chief Clinical Officer for the hospital's parent company said, “We had that piece of information [his travel history] in the electronic health record. It was there to be found” (Goodwyn 2014). With the large amount of information that was present in the EHR, a critical piece of information was overlooked and dire consequences ensued. The patient died from Ebola 2 weeks after his initial visit to that Texas emergency department (Botelho and Wilson 2014). In this age of automation, information overload is a significant problem. Knowing how to manage a knowledge‐worker's focus and ensure the visibility of relevant information is more critical than ever before.

Over the previous century, the field of operations management contributed to a fifty‐fold increase in the productivity of the manual worker (Drucker 1999). But, as we move through the 21st century towards a more knowledge‐based economy, there is significant need to further develop the science of, and remove the barriers to, efficient and high‐quality knowledge‐work (Davenport et al. 2002, Drucker 1999, Froehle and White 2014, Hayes 2002, Hopp et al. 2009, Ramírez and Nembhard 2004). While definitions vary, knowledge‐work is inherently more cognitive than physical in nature as knowledge‐workers leverage their experience, education, and intellectual capacity to transform information into some form of “product” (Davenport et al. 2002, Davis 1999, Drucker 1999, Hopp et al. 2009, Ramírez and Nembhard 2004).

The healthcare industry is one of the largest and fastest‐growing industries in the United States and healthcare processes are characterized by knowledge‐intensive tasks like clinical decision‐making. Two landmark reports from the Institute of Medicine (IOM) painted a picture of a healthcare system wrought with quality and efficiency problems. Removing the barriers to efficient and high‐quality care is of great need. In 1999, “To Err is Human: Building a Safer Health System” exposed the prevalence of medical errors and their implications on the overall cost and quality of healthcare in the United States. In 2001, “Crossing the Quality Chasm” made an urgent call for a redesign of the American healthcare system with a focus on providing high‐quality (safe, effective, evidence‐based, and patient‐centered) care that is delivered in an efficient and equitable manner. These reports prompted a national focus on healthcare reform, with the dialogue still continuing 15 years later as healthcare facilities focus on both reducing costs and improving quality (Russell et al. 2015).

A key finding in both reports was the potentially important role of health information technology. The federal government committed an unprecedented $27 billion to promote and expand the adoption of electronic health records (EHR) through the Health Information Technology for Economic and Clinical Health Act of 2009, which has created an increase of over 600% in the adoption of EHR systems by hospitals since 2008 (Adler‐Milstein et al. 2014). Improvements in the health system, however, have thus far fallen short of expectations (Black et al. 2011, Jones et al. 2012, Kellerman and Jones 2013). This apparent contradiction between investment in these operational technologies and lack of realized improvement highlights the fact that we do not adequately understand the factors that contribute to efficient and effective information systems (Angst et al. 2011). More alarmingly, the quality of patient care may be suffering as a result.

A recent study found that over one‐third of physicians reported missing test results in an EHR system because they are simply overwhelmed by the amount of information contained in these systems (Singh et al. 2013). That study highlighted a significant problem facing physicians today: information overload. While the most‐informed decision is often the best decision, humans have only a limited capacity to absorb and process complex problems and information (Croskerry 2005). In healthcare, EHRs provide data and information that could potentially increase the quality and efficiency of clinical decision‐making and improve patient care. But, the evidence suggests that EHRs are currently failing to meet those objectives and, even worse, potentially contributing to errors in clinical decision‐making (Ash et al. 2004, Kuperman 2011, Singh et al. 2013).

In this study, we introduce and test “emphasis framing” as an approach to mitigate the effects of information overload. From a cognitive processes perspective, a “frame” serves as a simplifying structure for categorizing new information (Davies and Mabin 2001), which decreases cognitive load and potentially improves decision‐making performance. Emphasis framing is the highlighting or stressing of some aspect or component of the information being exchanged in order to make it more likely to be, or more easily, internalized by the recipient (Druckman 2001, Entman 1993). This study evaluates the effectiveness of emphasis framing for improving cognitive categorization and decreasing cognitive load, as measured by the improvement in the quality and efficiency of decision‐making of emergency physicians. Emphasis framing can help physicians more effectively use the large amount information stored in EHRs, thereby supporting medical decision‐making, enhancing diagnosis, and, ultimately, improving patient outcomes.

The section that follows provides an introduction to the information overload and emphasis framing literature from which provides the foundations for this study and will be followed a detailed description of our experiment in section . We will then describe the analysis and an overview of our results in section and discuss the implications for both future research and practice in section 3.

Background Literature and Hypotheses

Information Overload

As an exemplar of knowledge‐work, clinical decision‐making in medicine is a complex, knowledge‐intensive process that involves a careful analysis of harms and benefits associated with different treatment options (Kushniruk et al. 1995). These decisions, often associated with high stakes and important long‐term consequences, are frequently made in the face of competing priorities, limited resources and information, and an incomplete clinical picture. Physicians are regularly faced with making high‐quality clinical decisions under these challenging circumstances, and the prevalence of medical errors has become well known.

In an attempt to provide information that could potentially improve the quality and efficiency of clinical decision‐making, and ultimately patient care, a significant investment in health information technology was made in the United States. But, Jones et al. (2012, p. 2244) contend that, “swapping out of the medical record cabinet and prescription pad for a computer is proving insufficient to realize the benefits of health IT.” According to research from IBM, humanity creates approximately 2.5 quintillion bytes of data each day—so much that 90% of the data in the world has been created in the last 2 years alone (IBM, 2012). Similarly, the more than 600% increase in the adoption of EHR systems by hospitals since 2008 has increased the intensity of information being presented to physicians (Ash et al. 2004, Harrison et al. 2007, Kuperman 2011, Singh et al. 2013). They are increasingly facing a significant problem: information overload.

Information overload occurs when the information intensity of a task increases to a point where its information‐processing requirements exceed the information‐processing capacities of the individual engaged in the task (Eppler and Mengis 2004). Information overload is largely determined by the quantity and complexity of information needing to be processed and plays an important role in the ability of a decision‐maker to accurately and efficiently process information (Eppler and Mengis 2004, Hiltz and Turoff 1985, Keller and Staelin 1987, Schick et al. 1990, Schneider 1987, Speier et al. 1999).

While collecting a broad array of information from many sources may best inform decision‐making, the Theory of Bounded Rationality argues that humans have only a limited capacity to process complex problems and information (Simon 1957). Up to a certain point, decision‐making performance is positively correlated with the amount of information a decision‐ maker receives. Beyond that point, the information‐processing requirements of a task exceed the information‐processing capacities of the decision‐maker, sending them into a state of information overload (Eppler and Mengis 2004). As a result, decision‐making performance decreases.

Information overload is readily seen as an important topic of research in the medical literature in the hopes that addressing it will help minimize cognitive errors in clinical decision‐making. According to Kahneman's dual‐process theory of decision‐making, cognition generally occurs in two ways: System 1 thinking occurs automatically and quickly with little or no voluntary effort, whereas System 2 thinking is a deliberate allocation of attention to effortful mental activities. The medical literature suggests that clinical decision‐making occurs with some interactive combination of System 1 and System 2 thinking (Graber et al. 2012).

According to Croskerry et al. (2013, p. ii58), it is the “flaws in clinical reasoning, rather than lack of knowledge that underlie cognitive errors,” and the majority of clinical errors occur “on the front line” when clinicians are under immense time pressures and their cognitive resources (e.g., focus and attention) are in short supply. Under those trying conditions, they posit that physicians are subconsciously looking for shortcuts, or ways to reduce the excessive cognitive load (information overload) and inappropriately revert to trained, procedural knowledge executed with System 1 thinking instead of the more appropriate System 2 thinking. This phenomenon has been observed in other contexts as well (Fransoo and Wiers 2006).

Therefore, in order to minimize cognitive errors and improve clinical decision‐making, it is important to find ways to reduce cognitive load. This will better enable physicians to avoid the pitfalls associated with using intuitive System 1 thinking and move more deliberately into the System 2 mode of thinking that is more appropriate for the complex, knowledge‐ intensive nature of clinical decision‐making (Graber et al. 2012).

EHRs have demonstrated the ability to effectively convey information about patients and their medical history and capture procedures to generate more accurate charges (Hamm and Zubialde 1995, Schiff and Bates 2010). EHRs have also been hailed for their potential to reduce information overload and significantly improve clinical decision‐making by better supporting collaborative thinking and promoting feedback. However, recent findings have suggested that, instead of helping solve this problem of cognitive load, EHRs may actually be contributing to the problem, potentially even decreasing the quality of clinical decisions (Ash et al. 2004, Kuperman 2011, Singh et al. 2013). In direct contrast to the need for enhanced collaborative thinking and the assistance of decision‐support tools to augment an individual physician's own knowledge and capabilities, research from the IOM (2012) suggests EHRs have actually reduced the type and amount of direct interactions between health care providers. This increased reliance on computer‐mediated communication (Froehle and Roth 2004), which can lack richness, highlights one of the many reasons we may not be realizing improvements despite the significant investment made in health information technology.

The likelihood of information overload occurring, as well as the effects if it does, can be influenced by the characteristics of both the task and the decision‐maker. First, task characteristics have been shown to affect the likelihood that a decision‐maker will experience information overload (Eppler and Mengis 2004, Iselin 1988, Keller and Staelin 1987, Schneider 1987, Speier et al. 1999). A high volume of information, high task complexity, and high novelty of information all increase the information processing requirements of a decision‐ making task. Complex, knowledge‐intensive tasks like the kind typically involved in clinical decision‐making can make the knowledge‐worker more susceptible to information overload.

Second, certain characteristics of the decision‐maker have also been shown to influence information overload (Eppler and Mengis 2004, Mackay and Elam 1992, Ramamurthy et al. 1992, Speier et al. 1999). The more qualified and experienced the decision‐maker is the faster and more efficient the individual will be at processing information related to their experience. Therefore, an experienced and well‐trained physician may be less likely to experience information overload in a medical decision‐making context than, for example, a first‐year resident.

As the research from IOM suggested, EHRs have reduced the type and amount of direct interactions between health care providers, implying that technology may be shifting the kind of communication present in many medical contexts from one of convergence (i.e., seeking shared understanding through frequent back‐and‐forth conversation) to conveyance (Dennis et al. 2008). In situations where conveyance is the dominant communication mode, technology tends to be asynchronous with no opportunity for interactive questioning. This is likely to increase the cognitive load and make information overload more probable by requiring the user (i.e., in our context, the physician) to absorb large amounts of information rapidly without the added benefits of synchronous clarification and nonverbal cues.

Emphasis Framing

In the context of cognitive processes, a “frame” serves as a simplifying structure for categorizing information, ideas, and concepts (Davies and Mabin 2001). The use of frames can decrease cognitive load and potentially improve decision‐making performance. There are two types of cognitive frames identified in the literature: equivalency framing and emphasis framing.

Equivalency framing looks at how the use of logically equivalent information, which is usually presented in a positive or negative way, influences the preferences of an individual. This type of framing occurs when “different, but logically equivalent, phrases cause individuals to alter their preferences” (Druckman 2004, p. 671). Historically, this is has been investigated to develop an understanding of how materially identical descriptions (e.g. 95% employment vs. 5% unemployment) influence political opinion formation.

This study focuses on the second kind of framing—emphasis framing—which occurs when the decision‐maker's focus is directed to a specific aspect or component of the information being exchanged. Unlike a Eureka task, which is designed to provide an obvious solution when one key piece of information is discovered (Ahissar and Hochstein 1997, Bonaccio and Dalal 2006, Simon 1965), an emphasis frame is designed to encourage explicit consideration of that information or to provide a specific context or interpretation for that piece of information (Druckman 2001, Entman 1993). For example, the abstract at the beginning of a scientific journal article could be seen as an emphasis frame because it directs the reader to an explicit subset of the most meaningful pieces of information contained in the overall article. The abstract does not contain new information; rather, it emphasizes a small set of facts distributed throughout the study. While considering the entire study could put the reader into a state of information overload, the abstract is meant to emphasize those facts and aspects that the authors deem most important. As a result, reading the abstract is generally recommended before consuming the entire study.

Based on the above description, some may incorrectly infer that we are proposing a new name for the tactic used in computerized physician order‐entry (CPOE) systems, and other similar types of systems, where algorithms automatically check things such as drug prescriptions against safety standards and interaction databases. For example, in the case where a physician incorrectly enters a value for a drug prescription (e.g., 100 mg/ml instead of 10 mg/ml), the system would flag the entry with an alert. Similarly, if a patient is already taking a drug that may create a dangerous interaction with a new prescription, an alert will appear on the screen, forcing the physician to acknowledge it before moving on. There has been significant work examining the efficacy of this approach and the “alert fatigue” it can produce (van der Sijs et al. 2006). In contrast with this tactic of alerting the decision‐maker to new information relevant to a decision that has already been made, emphasis framing is bringing existing information to the fore that could be important to an impending decision.

Cognitive concepts related to and underlying the concept we globally refer to as “emphasis framing” have been studied in various contexts, yet the literature shows no evidence that they have so far been examined by, or applied to, the field of operations management. The OM field has largely overlooked the possibility that framing could affect the productivity of knowledge‐workers, leaving it up to other fields to examine questions indirectly related to the issues that concern OM scholars and practitioners.

For example, political scientists have examined how emphasizing different aspects of a position in political communication can affect public opinion (Chong and Druckman 2007). In the aviation industry, NASA co‐developed the Advance Weather and Reporting Enhancements (AWARE) tool, which aids the agency by highlighting key pieces of information among the large weather datasets they review prior to flight (Ruokangas et al. 2006). Acting in a way similar to emphasis framing, AWARE helps pilots identify and focus on key pieces of weather information so they can complete their pre‐flights safely and more efficiently.

In the education literature, cognitive learning theory (CLT) has suggested that when learners are working on tasks with high‐intrinsic complexity, it can be beneficial to provide select, high‐level information (termed just‐in‐time information presentation) prior to their starting a task (Van Merriënboer et al. 2003). In the context of e‐commerce, consumer behaviorists have been studying consumer‐purchasing decision‐making when faced with a nearly unlimited supply of online choices. Häubl and Trifts (2000), as an example, found that interactive decision aids—tools that help online shoppers to compare and evaluate product information—helped consumers reduce cognitive load and make quicker and more effective purchase decisions. An interactive decision aid, much like emphasis framing, does not change the information being presented; it simply helps the consumer focus on the product features most important to their purchase decisions.

These studies use a wide range of terms to describe the phenomenon under investigation; we identify them all as specific implementations of emphasis framing. In light of this realization, we aspire to understand how emphasis framing might apply to operations in general, to knowledge‐intensive decision‐making more specifically, and to medical diagnosis tasks in particular. Medical diagnosis (a.k.a. differential diagnosis) is a probabilistic assessment of a patient's symptoms in order to determine the underlying disease(s) and identify the best course of treatment. Since physicians cannot always know what data are important, it would be helpful to build systems that help ensure caregivers are aware of potentially crucial information.

Because emphasis frames are specifically designed to accentuate valuable pieces of information, it seems likely that emphasis frames will improve the quality of decision‐making by ensuring the decision‐maker is aware of critical inputs. Also, because emphasis frames highlight useful or important pieces of information that might otherwise require extensive search to discover, it seems reasonable to expect that the presence of a well‐constructed emphasis frame will improve decision‐making efficiency by reducing the time required to gather needed information and arrive at an accurate conclusion. Therefore, this research hypothesizes:

H1 H1

In knowledge‐intensive work environments, emphasis framing improves the quality of decision‐making.

H2 H2

In knowledge‐intensive work environments, emphasis framing reduces decision‐making time.

Methods

In order to test these hypotheses, it was deemed infeasible to identify real‐world situations where data would be collected as a matter of course, so secondary data could not be used. Relying on surveys or other uncontrolled sources of primary data was similarly rejected due to the likelihood of respondents being unable to reliably identify and assess the effects of emphasis frames being either present or absent in their daily decision‐making. Experiments offer the benefits of precision, control of variables, and ability to replicate (McGrath 1981). Therefore, we conducted a controlled laboratory experiment designed as a balanced, fixed‐effects means model. We measured the effect of emphasis framing on two operational performance metrics when physicians are under information overload (i.e., using an EHR to examine a complex and extensive patient medical record): (1) the quality of the physician's clinical decision; and (2) the efficiency (timeliness) of the physician's clinical decision‐making.

Experimental Participants

Our study seeks to measure different metrics related to decision‐making where timeliness and accuracy are both valued. Physicians were an ideal target population from which to draw experimental participants. In order to ensure that the experience navigating within the EHR and subject‐matter effects were similar across participants, we selected physician participants employed within a single healthcare organization. Moreover, while all board‐certified physicians within a hospital share some level of common knowledge, the practice of medicine across specialties can vary significantly. Because emergency medicine often involves rapid decision‐making in the face of uncertain, life‐threatening situations, emergency physicians are likely to care strongly about both the quality and timeliness of their medical decision‐making.

As the Ebola example illustrated earlier, emergency medicine routinely involves making decisions after considering information that is both structured and unstructured, recent and historical. Because we desired to draw from a range of experience levels, from new residents to attending physicians with decades of experience, an additional concern was due to the potential effects of experience discussed above. Considering all this, we recruited experimental participants from the population of emergency medicine physicians at a large, urban, trauma‐certified, academic medical center. This single‐organization approach enabled us to reduce sources of measurement error because it would be reasonable to assume that participants would meet or exceed the same minimal level of functional expertise and knowledge required to fulfill their primary role as caregivers and to be suitably familiar with the terminology, technology, and best practices upon which their clinical decision‐making would be evaluated.

A total of 28 emergency department physicians from an urban academic emergency department agreed to participate in the study, representing nearly a third of the eligible participants from this organization. We stratified participants by skill level (resident vs. attending) and then randomized participants into two treatment groups, control vs. experimental. Of the 28 individuals who participated in the study, four were excluded for the following reasons: Two physicians were forced to quit mid‐study due to technical issues that prevented them from accessing the EHR system; one physician was interrupted mid‐study to address an unexpected work related issue; and one physician encountered a technical issue with the software tool used to conduct the experiment. Of the remaining 24 physician participants, nine attendings and three residents were assigned to each treatment group. The mean experience across all participants was 8.9 years (SD = 6.2) since medical school graduation. The mean years of experience for the two groups were not statistically different: the control group = 7.7 years (SD = 4.9) and the experimental group = 10.0 years (SD = 7.2). In our physician panel, 29.2% were female and 70.8% male. Over the entire population of emergency physicians in this department, 23.5% are female and 76.5% are male, similar to our panel.

The participants were recruited via email sent to all physicians in the department of emergency medicine. In an attempt to mitigate the effects of response bias, they were invited to take part in what was advertised as an EPIC (EHR software) usability study. Notifying participants that we were examining the effects of information overload might have lead participants to behave in a manner different than their true responses to the amount of information being presented; our IRB concurred. Of specific concern was the potential for social desirability bias. Because we were measuring the quality and timeliness of their decision‐making, we conjectured that participants would unnaturally adjust the amount of time spent reviewing the information presented in an attempt to mitigate the effects of information overload. As a recruitment incentive, participants were offered the choice of a $5 coffee shop gift card or a $10 charitable contribution to a medical education fund.

Experimental Protocol & Design

Summarily, the experiment involved providing a medical scenario to a physician, giving the physician access to the medical record system, and then asking the physician to document his/her recommended course of care for the patient. This section provides details regarding the design of the experiment and how it was executed.

Overview of the Experimental Protocol

This study was individually administered in a controlled lab environment where distractions and interruptions were minimized. Upon entering the laboratory environment, the principal investigator (PI) provided an overview of what the participant would be doing during the study and informed the participant that the PI would not be staying in the laboratory during the experiment, but would be readily available just outside the laboratory to assist if necessary. The participant was provided with a pen and blank paper in the event he/she wanted to take notes.

Physician participants began the experiment by clicking the “start” button on the computer screen, which opened a Web‐based survey environment. The first two survey pages provided the IRB‐required information sheet and consent as well as instructions that the participant should not disclose the experiment with others. The third page collected demographic information. The fourth page provided the participant more explicit details as to what he/she would be doing during the experiment and introduced the details of the problem scenario. On the fifth page, the physician clicked on a link to listen to one of two pre‐recorded reports from Air Care, the system's helicopter emergency medical services unit. Depending on experimental condition (see below), the Air Care report described a hypothetical patient who would be arriving via helicopter in a few moments. Following the message, on the sixth page, the physicians were provided the medical record number for the arriving patient; the medical record was that of a real patient selected with the approval of the IRB. Physicians were instructed to use their regular credentials to sign into the EPIC EHR system and conduct a historical chart review for the patient. As the final step, the physician documented his/her expectations for the patient's clinical course and recommended diagnostic and therapeutic interventions.

Experimental Interface

Two computer interfaces were used to conduct this experiment. The first interface was provided by Qualtrics (Provo, Utah: v. 2009) survey software. For this experiment, it was used to guide the participants through the multiple phases of the study listed above, to collect data on how much time each physician spent on each aspect of the study, and as a place for physicians to document their final clinical decisions and recommendations.

The second interface was provided by EPIC, one of the most commonly used EHR systems in the United States (Off. of the Nat. Coord. for HIT, 2015). The use of EPIC was important because it not only enhanced the authenticity of the experiment (more than, say, a mocked‐up fictional EHR would have), but it also contributed to the reliability of our experiment because the study participants use the EPIC EHR system in their role as physicians and we were confident were all familiar with the EHR software for the purpose of reviewing the patient's medical history. While the experiment was artificial, the physician participants viewed an electronic medical record of an actual patient who had previously visited this medical center. To ensure patient safety, the medical record was viewed in a “read‐only” environment that does not allow any modifications to be made to the live medical record.

Experimental Condition and Emphasis Frame

The control group was presented with all information needed to arrive at a clinical decision, but it did not receive an emphasis frame. The experimental group was presented with all of the same information as the control group as well as the emphasis frame. The emphasis frame highlighted a critical piece of information found in the patient's electronic record, but which could be easily overlooked due to the amount of information present there. The emphasis frame was represented by a single sentence contained in the Air Care report, which provided the physician with the context of the (fictitious) inbound patient's circumstances:

“Air Care 1 will be en route with a 43‐year‐old male restrained driver in a motor vehicle accident who is agitated on scene and was unable to be controlled and intubated for airway protection. He is notable to have a heart rate of 130 and blood pressure 150 over 87. A respiratory heart rate of 28, satting 99%, his breath sounds are clear bilaterally, his belly is soft, he has no obvious extremity trauma. He is noted to be somewhat rigid after the intubation. Of note, his wife did mention that he has a history of allergies to anesthesia. ETA is approximately 5 minutes. Any questions?”

The bolded sentence serves as the emphasis frame in this experiment and was the only difference between the control and experimental groups. It highlights information that is present in the EHR, but which may be easy to overlook: The patient has a history of dangerous reactions to succinylcholine, a paralytic agent, due to a rare, underlying genetic condition called malignant hyperthermia (MH). This condition is unknown to the EMT and emergency physician taking care of the patient at the accident site (who do not have access to the patient's medical record). It is noted that the patient begins to exhibit tachycardia (rapid heartbeat) and signs of muscle rigidity in his extremities after intubation. Highlighting the information that the patient has had bad reactions to anesthesia in the past is expected help the physician participant more quickly and accurately identify that the rigidity and tachycardia are likely attributed to succinylcholine‐induced malignant hyperthermia rather than any of the significantly less risky complications related to the motor vehicle accident (MVA), such as shock, seizure, or neuroleptic malignant syndrome (NMS). This information is important because the immediate care plan for an MH patient that has been administered succinylcholine (i.e., timely administration of the drug Dantrolene) is notably different than for a non‐MH patient experiencing shock, seizure, or NMS. If not immediately treated, the body fails to supply sufficient oxygen to the body, remove sufficient carbon dioxide from the body, or appropriately regulate body temperature, and can ultimately lead to circulatory collapse and death.

When a patient arrives to this emergency department via Air Care, the emergency physicians on shift at the hospital are first notified of the details of the arriving patient via a phone call from the Air Care physician. While an actual phone call would most closely mimic the real ED environment, there were four reasons why we chose to instead present the situation to the physician participants via a pre‐recorded Air Care message.

First, it helps to increase the external validity of our experiment because it more closely resembles how the physicians receive this information than reading it on‐screen would. Second, a recording enables us to ensure the information presented to participants is consistent throughout the experiment. We know that the same information is delivered in exactly the same way (speed of voice, intonation, etc…) for each participant. Third, using the physician who usually makes these phone calls in practice for this ED, we emphasized the authenticity of the situation, hopefully increasing the physicians' motivation to perform at a high level as well as their perceived time sensitivity. Fourth, an emergency physician's perception of a patient's clinical condition can vary depending on the source of the information, especially with out‐of‐hospital intubations. Research from the emergency medicine literature has shown that out‐of‐hospital intubation by paramedics not only fails to improve morbidity or mortality, but also may lead to unfavorable clinical effects, adverse events, and errors (Wang and Yealy 2006). Thus, to increase the likelihood of consistent interpretation about the hypothetical patient's clinical condition, it was important that the patient arrive via the physician‐staffed Air Care helicopter, which guarantees the patient's out‐of‐hospital care would have been provided by an emergency department physician and not a paramedic. Using a pre‐recorded audio message achieved all of these objectives.

Experiment Validation and Pilot Tests

In order to ensure experimental validity and fidelity vis‐à‐vis practice, we consulted with the vice chair of research for the department of emergency medicine (also a co‐author of this study); the department's medical director; an attending physician who also has a master's degree in clinical education and serves as an education and training instructor within the department; and a fourth‐year resident who was an air care lead. Each of these individuals helped ensure the experiment exhibited sound medicine and held true with the department's standard procedures while permitting key experimental data to be obtained.

Prior to the start of our data collection, we conducted a pilot test with two emergency department physicians and adjusted our experiment based on two key findings. First, the on‐ screen “button” that physicians were required to click to play the Air Care message was too small and led to confusion. The button size was increased for the full experiment. Second, while it was noted in two different places that the participants were to review the patient's “historical” record, both commented that it was important to inform participants that no information about the current (hypothetical) situation will be included in the medical record (i.e., it should be treated more as a retrospective chart review). This information was provided to participants in the full experiment.

Performance Measures: Quality and Efficiency

We measured the effect of emphasis framing on two operational performance metrics when physicians are under information overload: (1) the quality of the physician's clinical decision, and (2) the efficiency (timeliness) of the physician's clinical decision‐making.

Quality Measure

Because medical decision‐making quality is a multidimensional construct, the quality performance measure is necessarily complex. The quality measure is based on the free‐text response provided by the physician participants. Upon completion of the chart review, each physician was presented with the following instructions:

Assume no changes in the patient's clinical exam. Please answer the following questions:

What are the top items in your differential?

What is the anticipated clinical course for this patient in the first hour?

What are the key factors in your responses?

These three open‐ended questions were constructed with help from the emergency physician educator and based on standard terminology used to capture clinical decision‐making. The request to “assume no changes in the patient's clinical exam” was important for instrument reliability. In the practice of emergency medicine, it is possible for a patient's clinical condition to rapidly change in a short period of time. Thus, we wanted to ensure the physician participants viewed the presented information as a “snap shot” to avoid any effects from a time‐series bias in their clinical decision‐making. The three questions were followed by a large text box where the physician participant could type as much or as little as he or she wished.

In order to judge the quality of the participant's response, a baseline expectation was constructed with the help and guidance of experienced emergency physicians. The hypothetical patient in the experiment was supposedly involved in a minor motor vehicle accident (MVA) while he was restrained in his safety belt and did not exhibit any obvious extremity trauma. This information was important to highlight a low probability of severe injury. The patient was fairly young with a history of alcohol‐related injuries (as noted in his medical record), but an otherwise unremarkable medical history (e.g., no history of diabetes, cardiovascular disease, seizures, or neurological issues). Thus, physician participants should have perceived a low probability for significant complications from these diseases. The patient was noted to be agitated on scene, which was most likely due to alcohol intoxication (because of patient history), or information presented in the initial physical exam that could indicate the possibility of traumatic brain injury and/or intracranial hemorrhage. Having the patient agitated on scene was vital to justifying the Air Care team needing to intubate him for airway protection.

We developed a quality scoring rubric based on best practices as determined by emergency medicine physicians familiar with the scenario and patient history presented in this study. The rubric reflects completeness, accuracy, recognition of the underlying problem, and identification of the correct protocol moving forward. Four items compose the rubric as follows:

Quality of the physician response to possible causes of the patient's altered mental status/agitation on scene;

Quality of the physician response in terms of addressing the rigidity of the patient post‐intubation;

Overall quality of the physician's decision‐making; and

Overall clarity and thoroughness of the physician's response.

We asked four physician judges to independently score the quality of the physician participants' decision‐making. The scorers did not know each other, were not affiliated with the experiment or department, and independently conducted their evaluations of all 24 responses. They were instructed to score each response on each of the four questions (A–D above), using a scale from 0 to 100. To enhance inter‐rater reliability and to reduce experimenter bias, the judges were presented with clearly stated scoring guidelines for each question. After ensuring appropriate levels of interrater reliability, the individual experimental participant's scores for the four questions were combined into a single quality metric, weighted as follows: A = 25%, B = 40%, C = 25%, and D = 10%. These weights were based on advisement from the physician experts described earlier according to their potential contributions to the morbidity and mortality (outcomes) for this patient in this hypothetical scenario.

Efficiency/Timeliness Measure

To evaluate the efficiency, or timeliness, of decision making, the survey tool collected information on how long each physician spent during each phase of the experiment. We collected data on all aspects of the experiment, but were specifically interested in how much time each physician spent on the following activities: reading the problem description, chart review, and decision‐making. We defined decision‐making time, our timeliness performance metric, as the duration (in seconds) from the timestamp the physician completed the chart review (i.e., clicked to go to the next page of the Web survey tool) to the timestamp when documentation of clinical decision‐making is completed (i.e., the participant clicked the button to conclude the study).

Results

Quality of Clinical Decision‐Making

Our first hypothesis predicted that the presence of an emphasis frame would improve the physician's decision‐making quality by ensuring the decision maker is aware of critical inputs. We started our analysis of this performance measure by examining the inter‐rater reliability (IRR) of the four judges. We calculated the intra‐class correlation coefficient (ICC) for each of the four scores, which measures the proportion of variance of an observation due to between‐subject variability in the true scores (Ebel 1951). The higher the ICC values, the greater the IRR, with an ICC estimate of 1 indicating perfect agreement and 0 indicating only random agreement. It is the most appropriate measure of inter‐rater reliability for interval scale data. Our results indicate high inter‐rater reliability across all four questions with an average of 86.9% variance explained (Table 1).

Table 1

Intra‐class Correlation Coefficients

	Question A	Question B	Question C	Question D
Intra‐class Correlation Coefficient (ICC)	0.832	0.877	0.880	0.886
p‐value	0.000	0.000	0.000	0.000

The mean aggregate quality score for the control group was 44.6 (SD = 24.1), and 70.2 (SD = 30.0) for the experimental group (Figure 1). According to Levene's test (Levene 1960) for homogeneity of variance (p ≤ 0.05), it was determined that the variances did not differ significantly; F(1,22) = 0.437, p = 0.515. Therefore, we conducted an analysis of variance (ANOVA) and determined there was a significant main effect for the treatment; F(1,24) = 5.313, p = 0.031.

Figure 1

Summary of Results

We extended our analysis to see what effect the emphasis frame had after accounting for the effect of the covariate: skill level, defined by the number of years since the physician graduated medical school. We observed a significant emphasis framing effect on quality even after controlling for skill level; F(2,24) = 2.605, p = 0.098.

As an alternative measure of quality, we conducted a chi‐square test to determine if the proportion of physicians who correctly identified the patient as potentially having MH differed by whether the emphasis frame was present or not. We found that only 33% of participants in the control group correctly identified the MH, but 75% of participants in the experimental group did. We found that the proportion of participants who correctly identified the patient as having MH did differ statistically depending on the presence of an emphasis frame; χ ² (1, N = 24) = 4.20, p = 0.041.

Timeliness of Clinical Decision‐Making

Our second hypothesis predicted that the presence of an emphasis frame would improve the physician's decision‐making efficiency by enabling him or her to arrive at a conclusion more quickly than without the frame. Contrary to that expectation, the average time required to arrive at a decision and document a course of action increased significantly for the group with the emphasis frame.

The mean decision‐making time for the control group was 266 seconds (SD = 69.1) and 421 seconds (SD = 245.7) for the experimental group (Figure 1). According to Levene's test for homogeneity of variance (p ≤ 0.05), it was determined that the variances are significantly different; F(1,22) = 8.321, p = 0.009. Thus, we conducted a two‐sample, non‐paired t‐test assuming unequal variances and concluded that the presence of emphasis framing increased the average clinical decision‐making time, t(12) = −1.86, p = 0.047.

Discussion

Statistically significant results suggested that hypothesis 1, which predicted an improvement in decision‐making quality, was supported and emphasis framing increased decision‐making quality. Hypothesis 2, which predicted that presence of an emphasis frame would improve decision‐making efficiency, was not supported; instead, we found that emphasis framing increased decision‐making time.

Surprisingly, the emphasis frame was found to increase clinical decision‐making time. While we hypothesized it would enable the participant to make decisions more quickly, it appears that was not so. However, the presence of the frame significantly improved the quality of the medical decisions and significantly increased the likelihood of successfully identifying the patient's MH. One potential explanation is that the emphasis frame forced the decision‐maker to consider a piece of important information that might have otherwise escaped notice, thereby compelling the physician to move away from System 1 thinking into the more appropriate System 2 mode of thinking and examine the medical record more thoroughly and spend more time, not less, considering the evidence.

Implications

The relationship between speed and accuracy, as a signature of the decision process itself, has been examined in the behavioral science and economics literatures (Becker 1958, Stone 1960, Wald 1947). In OM, the potential trade‐off between production speed and product quality has drawn significant attention over the years. Operations strategy researchers have long debated the question of whether, at the corporate or industry level, speed and low cost are inherently at odds with quality, typically studied in the context of manufacturing (Boyer and Lewis 2002, Rosenzweig and Roth 2004). In customer‐facing services, however, this relationship can be even more complex due to the value (and perceived quality) that customers might attribute to the service provider spending more, rather than less, time with them (Anand et al. 2011). In knowledge‐work services, especially those processes occurring in the back‐office (Chase 1981), there can be a managerially frustrating mix of productivity goals and customer‐service expectations (Froehle and White 2014). This creates a situation where a speed‐quality trade‐off can be particularly important for both researchers and practitioners to identify and incorporate into their modeling and/or planning.

This study provides a unique contribution to the existing literature as it builds upon this trade‐off paradigm in the context of medical decision‐making in particular and knowledge‐work operations in general. The volume of information available to inform medical decision‐making is greater now than at any point in history. In the presence of a focus on improving the quality and efficiency of the national and regional healthcare systems worldwide, there is a significant need to develop a better understanding of the relationship between the speed and accuracy of medical decision‐making. Emphasis framing can help improve how healthcare information contained in EHRs is used, enhancing medical decision‐making and, ultimately, patient outcomes.

Our results show the inclusion of an emphasis frame may have the valuable effect of improving the quality of decision‐making when the decision‐maker is working in a state of information overload, at least in an emergency medicine context. This has several potential implications for emergency medicine: improved patient outcomes as a result of more accurate medical diagnoses; more efficient resource allocation and; improved risk management.

Our results also demonstrated that the presence of an emphasis frame can slow decision‐ making. As this result could potentially have positive or negative implications, it needs to be further explored. On one hand, there may be times when a wrong or incomplete decision that is made quickly is actually more beneficial than a complete and accurate decision that took too long to produce. However, there are situations when the additional time spent producing a high‐quality decision more than offsets the increased time costs, such as when it can produce significantly improved clinical benefits (patient outcomes), more efficient allocation of emergency department resources, and improved patient or customer satisfaction.

Limitations and Future Work

As with any study, regardless of precautions taken, the results have limitations and could be extended in additional ways. First, our results are based on data from a single, urban, academic, emergency medicine department, using a specific EHR. It is not guaranteed that our results are generalizable to different medical specialties, in a different organization, when using a different EHR platform, or even outside medicine. The design of our medical scenario was uniquely created to match the participants and the environmental context, which, typical of experiments, limits its generalizability (McGrath 1981). Second, we cannot rule out the possibility that our two participant groups, despite being randomized on skill level, systematically differ in other, unknown ways that influence our results. Finally, we relied on a small set (four) of physician expert judges to evaluate the quality of the participants' medical decision‐making; a broader set of judges may have produced different quality scores (although sensitivity analyses suggest that this is unlikely to change our conclusions).

For next steps and future research extensions, several clear opportunities present themselves. First, our study's findings suggest the potential benefit of emphasis framing for improving the quality of clinical decision‐making, but to extend greater generalizability we could capitalize on a more diverse sample to establish if these results will hold in different medical scenarios or even outside the field of medicine.

Second, managerially speaking, further research is needed to address how and when to implement the cognitive component into knowledge‐intensive workflows. How would the information contained in the emphasis frame be first identified and then highlighted for the decision‐maker? Identifying critical information that is easily overlooked might be a worthwhile task for our increasingly capable information automation tools, such as machine learning and artificial intelligence (Jordan and Mitchell 2015). These proactive alerts, highlighting potentially important information in a medical record, much like an emphasis frame, already exist in nascent form. Our research suggests that, if decision‐making quality is to be improved, these tools should receive considerable effort to improve and deploy more widely. However, we should not aspire to entirely replace the human element in medical decision‐making. As Chen and Asch (2017) said: “Combining machine learning software with the best human clinician ‘hardware’ will permit delivery of care that outperforms what either can do alone.” Using analytics to provide emphasis frames that enhance human decision‐making could be among the current best approaches to achieving this balance.

Third, interruptions affect the way information is processed by the decision‐maker and can lead to a decrease in the quality and efficiency of decision‐making performance (Froehle and White 2014, Speier et al. 1999, Tucker and Spear 2006). And interruptions are frequent and pervasive in clinical practice (Tucker and Spear 2006). While this study did not specifically consider interruptions, future work in this area may benefit from exploring whether the decreased cognitive load attributed to the emphasis frame could improve the quality of decision‐making when a task is interrupted.

Finally, our study's findings illustrated that emphasis framing has the potential to improve the quality of decisions at the expense of timeliness and substantiates the importance of further exploring the potential trade‐off between the speed and accuracy of knowledge‐intensive decision‐making. Further research should consider parameterizing the trade‐off function that characterizes the threshold between speed and accuracy, thereby optimally balancing the benefits to the patient and efficiency to the system. This study represents a first step towards achieving these advances.

References

Adler‐Milstein

DesRoches

C. M.

Furukawa

M. F.

Worzala

Charles

Kralovec

Stalley

Jha

A. K.

. 2014. More than half of US hospitals have at least a basic EHR, but stage 2 criteria remain challenging for most. Health Aff. 33(9): 1664–1671.

Ahissar

Hochstein

. 1997. Task difficulty and the specificity of perceptual learning. Nature 387(6631): 401–406.

Anand

K. S.

Paç

M. F.

Veeraraghavan

. 2011. Quality–speed conundrum: Trade‐offs in customer‐intensive services. Management Sci. 57(1): 40–56.

Angst

C. M.

Devaraj

Queenan

C. C.

Greenwood

. 2011. Performance effects related to the sequence of integration of healthcare technologies. Prod. Oper. Manag. 20(3): 319–333.

Ash

J. S.

Berg

Coiera

. 2004. Some unintended consequences of information technology in health care: The nature of patient care information system‐related errors. J. Am. Med. Inform. Assoc. 11(2): 104–112.

Becker

G. M.

1958. Sequential decision making: Wald's model and estimates of parameters. J. Exp. Psychol. 55: 628–636.

Black

A. D.

Car

Pagliari

Anandan

Cresswell

Bokun

Procter

Majeed

Sheikh

. 2011. The impact of eHealth on the quality and safety of health care: A systematic overview. PLoS Med. 8(1): 188.

Bonaccio

Dalal

R. S.

. 2006. Advice taking and decision‐making: An integrative literature review, and implications for the organizational sciences. Organ. Behav. Hum. Decis. Process. 101(2): 127–151.

Botelho

Wilson

. 2014. Thomas Eric Duncan: First Ebola death in U.S. Available at http://www.cnn.com/2014/10/08/health/thomas-eric-duncan-ebola/index.html (accessed date June 29, 2017).

10.

Boyer

K. K.

Lewis

M. W.

. 2002. Competitive priorities: Investigating the need for trade‐offs in operations strategy. Prod. Oper. Manag. 11(1): 9–20.

11.

Chase

1981. The customer contact approach to services: Theoretical bases and practical extensions. Oper. Res. 29(4): 698–706.

12.

Chen

J. H.

Asch

S. M.

. 2017. Machine learning and prediction in medicine—Beyond the peak of inflated expectations. New Engl. J. Med. 376(26): 2507–2509.

13.

Croskerry

2005. Diagnostic failure: A cognitive and affective approach. Henriksen

Battles

J. B.

Marks

E. S.

Lewin

D. I.

, eds. Advances in Patient Safety: From Research to Implementation (Volume 2: Concepts and Methodology). Agency for Healthcare Research and Quality (US), Rockville, MD.

14.

Chong

Druckman

J. N.

. 2007. Framing theory. Annu. Rev. Political Sci. 10: 103–126.

15.

Croskerry

Singhal

Mamede

. 2013. Cognitive debiasing 1: Origins of bias and theory of debiasing. BMJ Qual. Saf. 22(Suppl 2): ii58–ii64.

16.

Davenport

T. H.

Thomas

R. J.

Cantrell

. 2002. The mysterious art and science of knowledge‐worker performance. MIT Sloan Manage. Rev. 44(1): 23–30.

17.

Davies

Mabin

V. J.

. 2001. Knowledge management and the framing of information: A contribution to OR/MS practice and pedagogy. J. Oper. Res. Soc. 52(8): 856–872.

18.

Davis

G. B.

1999. A research perspective for information systems and example of emerging area of research. Inf. Syst. Front. 1(3): 195–203.

19.

Dennis

A. R.

Fuller

Valacich

. 2008. Media, tasks, and communication processes: A theory of media synchronicity. MIS Q. 32(3): 575–600.

20.

Drucker

P. F.

1999. Knowledge‐worker productivity: The biggest challenge. Calif. Manag. Rev. 41(2): 79–94.

21.

Druckman

J. N.

2001. The implications of framing effects for citizen competence. Polit. Behav. 23(3): 225–256.

22.

Druckman

J. N.

2004. Political preference formation: Competition, deliveration, and the (ir)relevance of framing effects. Am. Polit. Sci. Rev. 98(671–686).

23.

Ebel

R. L.

1951. Estimation of the reliability of ratings. Psychometrika 16(4): 407–424.

24.

Entman

R. M.

1993. Framing: Toward clarification of a fractured paradigm. J. Commun. 43(4): 51–58.

25.

Eppler

M. J.

Mengis

. 2004. The concept of information overload: A review of literature from organization science, accounting, marketing, MIS, and related disciplines. Inf. Soc. 20(5): 325–344.

26.

Fransoo

J. C.

Wiers

V. C. S.

. 2006. Action variety of planners: Cognitive load and requisite variety. J. Oper. Manag. 24(6): 813–821.

27.

Froehle

C. M.

Roth

A. V.

. 2004. New measurement scales for evaluating perceptions of the technology‐mediated customer service experience. J. Oper. Manag. 22(1): 1–22.

28.

Froehle

C. M.

White

D. L.

. 2014. Interruption and forgetting in knowledge‐intensive service environments. Prod. Oper. Manag. 23(4): 704–722.

29.

Goodwyn

2014. Was CDC too quick to blame dallas nurses in care of Ebola patient? National Public Radio. Available at http://www.npr.org/2014/10/24/358574357/was-cdc-too-quick-to-blame-dallas-nurses-in-care-of-ebola-patient (accessed date October 18, 2015).

30.

Graber

M. L.

Kissam

Payne

V. L.

Meyer

A. N.

Sorensen

Lenfestey

Tant

Henriksen

LaBresh

Singh

. 2012. Cognitive interventions to reduce diagnostic error: A narrative review. BMJ Qual. Saf. 21(7): 535–557.

31.

Hamm

R. M.

Zubialde

. 1995. Physicians' expert cognition and the problem of cognitive biases. Prim. Care 22(2): 181–212.

32.

Harrison

M. I.

Koppel

Bar‐Lev

. 2007. Unintended consequences of information technologies in health care—An interactive sociotechnical analysis. J. Am. Med. Inform. Assoc. 14(5): 542–549.

33.

Häubl

Trifts

. 2000. Consumer decision making in online shopping environments: The effects of interactive decision aids. Market. Sci. 19(1): 4–21.

34.

Hayes

R. H.

2002. Challenges posed to operations management by the “new economy”. Prod. Oper. Manag. 11(1): 21–32.

35.

Hiltz

Turoff

. 1985. Structuring computer‐mediated communication systems to avoid information overload. Commun. ACM 28(7): 680–689.

36.

Hopp

W. J.

Iravani

S. M. R.

Liu

. 2009. Managing white‐collar work: An operations‐oriented survey. Prod. Oper. Manag. 18(1): 1–32.

37.

IBM . 2012. What is big data? Available at https://www-01.ibm.com/software/data/bigdata/what-is-big-data.html (accessed date January 24, 2017).

38.

Institute of Medicine . 2012. Health IT and patient safety: building safer systems for better care. National Academies Press, Washington, DC.

39.

Iselin

E. R.

1988. The effects of information load and information diversity on decision quality in a structured decision task. Acc. Organ. Soc. 13(2): 147–164.

40.

Jones

S. S.

Heaton

P. S.

Rudin

R. S.

Schneider

E. C.

. 2012. Unraveling the IT productivity paradox—lessons for health care. N. Engl. J. Med. 366(24): 2243–2245.

41.

Jordan

M. I.

Mitchell

T. M.

. 2015. Machine learning: Trends, perspectives, and prospects. Science 349(6245): 255–260.

42.

Keller

K. L.

Staelin

. 1987. Effects of quality and quantity of information on decision effectiveness. J. Consum. Res. 14(2): 200–213.

43.

Kellerman

A. L.

Jones

S. S.

. 2013. What it will take to achieve the as‐yet‐unfulfilled promises of health information technology. Health Aff. 32(1): 63–68.

44.

Kuperman

G. J.

2011. Health‐information exchange: Why are we doing it, and what are we doing? J. Am. Med. Inform. Assoc. 18(5): 678–682.

45.

Kushniruk

Patel

Fleiszer

. 1995. Analysis of medical decision making: A cognitive perspective on medical informatics. Gardner

R. M.

, ed. Proceedings of the Nineteenth Annual Symposium on Computer Application in Medical Care. Hanley and Belfus, Philadelphia, PA, 193–197.

46.

Levene

1960. Robust tests for equality of variances. Olkin

Ghurye

S. G.

Hoeffding

Madow

W. G.

Mann

H. B.

, eds. Contributions to Probability and Statistics: Essays in Honor of Harold Hotelling. Stanford University Press, Menlo Park, CA, 278–292.

47.

Mackay

J. M.

Elam

J. J.

. 1992. A comparative study of how experts and novices use a decision aid to solve problems in complex knowledge domains. Inf. Syst. Res. 3(2): 150–172.

48.

McGrath

J. E.

1981. Dilemmatics: The study of research choices and dilemmas. Am. Behav. Sci. 25(2): 179–210.

49.

Qualtrics . 2009. Computer software. Provo, Utah: Qualtrics.

50.

Ramamurthy

King

W. R.

Premkumar

. 1992. User characteristics‐DSS effectiveness linkage: An empirical assessment. Int. J. Man Mach. Stud. 36(3): 469–505.

51.

Ramírez

Y. W.

Nembhard

D. A.

. 2004. Measuring knowledge worker productivity: A taxonomy. J. Intellectual Capital 5(4): 602–628.

52.

Rosenzweig

E. D.

Roth

A. V.

. 2004. Towards a theory of competitive progression: Evidence from high‐tech manufacturing. Prod. Oper. Manag. 13(4): 354–368.

53.

Ruokangas

C. C.

Mengshoel

O. J.

Uckun

Rand

T. W.

Donohue

Tuvi

Rockwell Scientific Licensing LLC . 2006. Aviation weather awareness and reporting enhancements (AWARE) system using a temporal‐spatial weather database and a Bayesian network model. U.S. Patent 7,081,834.

54.

Russell

R. S.

Johnson

D. M.

White

S. W.

. 2015. Patient perceptions of quality: Analyzing patient satisfaction surveys. Int. J. Oper. Prod. Manag. 35(8): 1158–1181.

55.

Schick

A. G.

Gordon

L. A.

Haka

. 1990. Information overload: A temporal approach. Acc. Organ. Soc. 15(3): 199–220.

56.

Schiff

G. D.

Bates

D. W.

. 2010. Can electronic clinical documentation help prevent diagnostic errors? N. Engl. J. Med. 362(12): 1066–1069.

57.

Schneider

S. C.

1987. Information overload: Causes and consequences. Hum. Syst. Manag. 7(2): 143–153.

58.

van derSijs

Aarts

Vulto

Berg

. 2006. Overriding of drug safety alerts in computerized physician order entry. J. Am. Med. Infor. Assoc. 13(2): 138–147.

59.

Simon

H. A.

1957. Models of Man, Social and Rational: Mathematical Essays on Rational Human Behavior in a Social Setting. John Wiley and Sons, New York.

60.

Simon

H. A.

1965. Administrative Behavior. Free Press, New York.

61.

Singh

Spitzmueller

Petersen

N. J.

Sawhney

M. K.

Sittig

D. F.

. 2013. Information overload and missed test results in electronic health record–based settings. JAMA Intern. Med. 173(8): 702–704.

62.

Speier

Valacich

J. S.

Vessey

. 1999. The influence of task interruption on individual decision making: An information overload perspective. Decis. Sci. 30(2): 337–360.

63.

Stone

1960. Models for choice‐reaction time. Psychometricka 25(3): 251–260.

64.

Tucker

A. L.

Spear

S. J.

. 2006. Operational failures and interruptions in hospital nursing. Health Serv. Res. 41(3 pt 1): 643–662.

65.

VanMerriënboer

J. J.

Kirschner

P. A.

Kester

. 2003. Taking the load off a learner's mind: Instructional design for complex learning. Educ. Psychol. 38(1): 5–13.

66.

Wald

1947. Sequential Analysis. Wiley and Sons, New York.

67.

Wang

H. E.

Yealy

D. M.

. 2006. How many attempts are required to accomplish out‐of‐hospital endotracheal intubation? Acad. Emerg. Med. 13(4): 372–377.