To Computerised Provider Order Entry system: A comparison of ECF,HFACS,STAMP and AcciMap approaches

Abstract

Different accident analytical approaches have been utilised in safety-critical industries for analysing accidents and formulating safety recommendations. This study presents a ‘health informatics’ case incident of a patient adversely affected due to a medication dosing error resulting from a combination of contributing factors including those relating to the Computerised Order Provider Entry System. A comparative study was carried out using selected accident analytical approaches: Human Factors and Classification System, System-Theoretic Accident Modelling and Processes and Accident Modelling. Each resulting output was compared using the model characteristic criteria developed by Underwood and Waterson. Safety recommendations developed based on the outputs from the models/methods were also compared for any similar findings. It was acknowledged that while accident models incorporating ‘systems thinking’ can prove to be beneficial for healthcare in providing insight on systemic factors, there is a need for improving the reliability and validity of these models. This particularly applies to Rasmussen’s Accident Modelling approach to be considered useful in the healthcare domain.

Keywords

accident models health information technology patient safety risk assessment risk management systemic models

Introduction

The healthcare system can be regarded both as a safety critical and a sociotechnical system consisting of the interconnection of people (users of health information technology (health IT) systems), technology (software/hardware), processes, organisation and the external environment (where policies are developed and enforced)¹ as shown in Figure 1.

Figure 1.

Sociotechnical system underlying health-IT related adverse events (adapted from Sittig and Singh¹ and Harrington et al., 2011).

Health IT broadly comprises of ‘all computer software used by health professionals and patients to support care’.² Its implementation has helped to reduce medical errors leading to patient harm and improve clinical processes, workflow and communication between clinicians for increased efficiency.^3,4 Unfortunately, unintended consequences and new forms of errors resulting from using IT systems can adversely affect patient safety.² These consequences occur from interactions between operators and clinical IT systems contributing to the occurrence of such outcomes. Reasons behind any occurrences should not only focus on proximate causes but also existing latent conditions including design, use of health IT systems and existing policies within and outside the organisation. The Insititute of Medicine (IOM) also highlighted the need for improving patient safety by ensuring the safe use of health IT in the delivery of effective healthcare.³

Different accident analytical approaches have been used in different safety-critical domains for investigating ‘why’ accidents occurred and developing effective countermeasures to improve safety.^5,6 While approaches incorporating ‘systems thinking’ are considered the dominant paradigm for accident analysis, there has been a research gap in practically applying these approaches for incident analyses in healthcare systems.⁶ Due to the growing awareness of the limitation of root cause analysis (RCA) techniques, there is a need for incorporating systemic approaches for accident analysis. This is especially very critical in being to investigate further into contributing factors relating to the use of health IT within a complex system like healthcare.

Objectives of study

The aim of this study was to determine which of the selected accident analytical approaches is the most practically suitable using a health IT–related case example. To achieve this, study objectives carried out include the following:

Comparing the resulting outputs from each accident causation model based on the usage characteristics criteria.⁷

Comparing causes/contributing factors identified from the application of these approaches.

Comparing safety recommendations developed based on analyses.

Research methodology

A case study approach⁸ was considered the appropriate approach in applying accident causation approaches and comparing findings (contributing factors and safety recommendations). Details of the incident were drawn from multiple sources.^3,9 The aim of exploring this case incident was in understanding the role of Computerised Order Provider Entry System (CPOE) application in addition to other contributing factors that led to the patient receiving a high dose of potassium chloride (KCI).

Each accident causation approach was selected based on their theoretical underpinnings and methodology in analysing adverse events. The Events and Causal Factors (ECF) approach was selected based on its ability to provide ‘linear’ and simplistic evidence-based view of the accident by graphically presenting a chain of events leading to a failure.¹⁰ The Human Factors and Classification System (HFACS) allows causes/contributing factors to be classified according to its taxonomy of failure categories which is based on Reason’s Swiss Cheese model. AcciMap (Accident Modelling) and STAMP (System-Theoretic Accident Modelling and Processes) were selected for being among the most utilised systemic approaches in safety-critical domains.⁶ While the former allows contributing factors to be causally mapped according to the model’s abstract levels,¹¹ the latter allows for control flaws and inadequate constraints to be identified.¹²

The application of the selected approaches was carried out by the principal author, and respective outcomes were reviewed by human factors specialists and experienced users of these techniques. The safety recommendations were also reviewed and validated. A usage evaluation framework used in previous comparative studies⁷ forms the basis for discussing for comparing the application of the selected approaches.

Description of accident approaches

This section provides a brief description of the selected accident causation approaches.

ECF

ECF charting provides a means of graphically presenting chains of events that lead to failure.^13,14 It allows investigators to chart the sequence of events in a chronological or linear manner.¹³ This method can be utilised as a standalone technique for representing events using different symbols from the ‘initiating event’ to the ‘accident’ and further into amelioration processes. Counterfactual reasoning is also applied by investigators to differentiate causal factors from influences relating to the incident.¹⁵

HFACS

The HFACS method not only was initially developed for analysing accidents in the aviation industry but has also been adapted to the healthcare domain (Wiegmann and Shappell, 2003).¹⁶ It is based on Reason’s Swiss Cheese Model comprising of unsafe acts, pre-conditions for unsafe acts, unsafe supervision, and organisational influences^17,18 (see Figure 2). This model also provides qualitative and quantitative means of analysing multiple cases to determine patterns and trends to accident causations in the system as well as root causes.

Figure 2.

The Human Factors and Classification System (HFACS).¹⁷

AcciMap

The AcciMap model provides a graphical multi-causal diagram for determining factors (events and decisions) that contributed to the accident^19,20 (see Figure 3). This approach is also based on Rasmussen’s²¹ theory of accident causation. Based on the AcciMap analysis, safety recommendations are formulated to prevent their future occurrence.^22,23 The model can be used as a standalone and as part of a broader risk management framework (RMF) involving Actor maps, Info maps and Conflict maps.¹⁹ This process is achieved by linking causal connections within and between different system levels.

Figure 3.

Skeletal AcciMap outline (adapted from Svedung and Rasmussen¹¹).

STAMP

The STAMP model is a systemic-based model developed by Leveson¹² for analysing adverse events in complex systems. It is based on systems and control theory where safety constraints between components are analysed for any disturbances that can affect system safety^24,25 (see Figure 4).

Figure 4.

Generic complex sociotechnical safety control structure.²⁴

The STAMP model also comprises of two types of analysis: Systems Theoretic Process Analysis (STPA) and Causal Analysis using System Theory (CAST). The STPA is a STAMP-based hazard analysis on system engineering in defining accidents, system hazards and control structure.²⁴ The model’s architecture consists of a generic complex sociotechnical system safety control structure and a high-level taxonomy of safety constraints utilised for specific system hazards defined at the beginning of the analysis (see Table 1).

Table 1.

STAMP’s control failure taxonomy of control flaws leading to hazards.¹²

1. Inadequate enforcements of constraints (control actions)

1.1 Unidentified hazards

1.2 Inappropriate, ineffective or missing control actions for identified hazards

1.2.1 Design of control algorithm (process) does not enforce constraints

Flaws in the creation process

Process changes without an appropriate change in control algorithm (asynchronous evolution)

Incorrect modification or adaptation

1.2.2 Process models inconsistent, incomplete or incorrect (lack of linkup)

Flaws in the creation process

Flaws in updating process (asynchronous evolution)

Time lags and measurement inaccuracies not accounted for

1.2.3 Inadequate coordination among controllers and decision makers

2. Inadequate execution of control action

2.1 Communication flaw

2.2 Inadequate actuator operation

2.3 Time lag

3. Inadequate or missing feedback

3.1 Not provided in system design

3.2 Communication flow

3.3 Time lag

3.4 Inadequate sensor operation (incorrect or no information provided)

STAMP: system-theoretic accident modelling and processes.

Case incident synopsis

This incident involved a patient who was initially hypokalemic when admitted. Two providers (A and B) attended to the patient with the first administering KCI.^3,9 Events leading up to the patient receiving a high dose of KCI took place within the space of 3 days (see Table 2). This incident was classified as an example of a ‘software-related’ or ‘health IT–related’ incident where the combination of factors relating to the technological product and how it was operated can increase the likelihood of the accident occurring.

Table 2.

Orders and actions by providers A and B over 2 days (adapted from Horsky et al.⁹).

Time	Provider	Action	Type	Description	Notes/findings	Order No.
Saturday 13.30 (7 min)	A	ACT	IV injection	40 mEq KCI IV injection over 4 h decision	Correct order The provider wants to change IV injection of KCI to a medicated drip to avoid pain administration	1
		DC	Drip	D5 W non-medicated fluid	Discontinues an older standing order (not in table)	2
		ACT	Drip	D5 W with 40 mEq KCI 1000 mL at 75 mL/h	Intended for 1 L of fluid only; free text volume limit, auto stop in 7 days	3
		DC	Drip	Preceding order discontinued	Realises the preceding order [3] was incorrect and discontinues	4
		ACT	Drip	D5 W non medicated fluid	Enters order identical to the one just discontinued [2]	5
		ACT	Drip	D5 W with 100 mEq KCI 1000 mL at 75 mL/h	Second attempt to enter drip order, similar to order [3]; now with a higher dose (100 mEq)	6
		DC	IV injection	KCI 20 mEq	Meant to discontinue order [1] but discontinued an expired order from 2 days before (not in table)	7
49 min time lag					Pharmacy calls to warn about order [6], which has dose over limit (100 mEq, max allowed 80 mEq)
Saturday 14:26 (16 min)	A	DC	Drip	D5 W non-medicated fluid	Discontinues non-medicated fluid order [5] in response to the call from pharmacy	8
		DC	Drip	D5 W with 100 mEq KCI 1000 mL at 75 mL/h	Discontinues erroneous drip order [6] in response to the call from pharmacy	9
		ACT	Drip	D5 W with 80 mEq KCI 1000 mL at 75 mL/h	Enters recommended 80 mEq. Intended for 1 L only, but no stop time entered; auto stop in 7 days	10
52 min time lag
Saturday 15:34	A	DC	Drip	D5 W with 80 mEq KCI 1000 mL at 75 mL/h	The preceding order [10] discontinued	11
Saturday 15:34	A	ACT	Drip	D5 W with 80 mEq KCI 1000 mL at 75 mL/h	The same order [cf. 10, 11] re-entered, runs for 36 h and delivers 216 mEq KCI	12
27 h time lag change of providers
Sunday 18:36	B	ACT	IV injection	40 mEq KCI IV injection	Misperceived older potassium laboratory values as current; did not notice running KCI drip [12]	13
34 min time lag
Sunday 19:10	B	DC	IV injection	40 mEq KCI IV injection	The preceding order [13] discontinued	14
Sunday 19:10	B	ACT	IV injection	60 mEq KCI IV injection	Increased IV injection dose to 60 mEq	15
27 min time lag
Sunday 19:37	B	ACT	IV injection	40 mEq KCI IV injection	Another IV injection of KCI ordered; however, no clear evidence that it was in fact administered	16

ACT: activate; DC: discontinue; IV: intravenous; KCI: potassium chloride.

Results of analysis

Each of the results obtained from the use of ECF, HFACS, STAMP and AcciMap are highlighted in the following sub-sections.

ECF result

The output of the ECF analysis assumes linear sequences of events and associated conditions leading to the adverse event (patient becoming hyperkalemic) (see Figure 5). The event that ended with ‘the patient becoming hyperkalemic’ can be further analysed to include amelioration processes of ensuring that this adverse event does not occur again. In identifying root causes, there was evidence that there was miscommunication between providers A and B regarding the patient’s KCI levels which contributed to the provider B unintentionally increasing the KCI dose. Another root cause includes the interface design of the CPOE system which made it difficult for both the first provider and the attending nurse to place in the order.

Figure 5.

Events and causal factor (ECF) analysis of the case study over the course of 3 days.

HFACS result

The HFACS analysis allows the classification of failures into four levels relating to the medication and CPOE incident (Figure 6).

Figure 6.

The application of the HFACS on the health-IT adverse event case study.

Unsafe acts

Under unsafe acts category, skill-based errors consist of provider A and the attending nurse incorrectly entering the order in the CPOE system. Decision errors consist of actions of provider A in discontinuing and reentering a similar order which ran for 36 h but with a higher dose. In the perceptual errors subcategory, there was evidence suggesting misjudgement when reading KCI of the patient particularly by provider B. In the violations subcategory (routine and exceptional), violation of the procedures regarding rechecking of the patient’s KCI levels during clinical handover between providers A and B increased the likelihood of the patient becoming more hyperkalemic.

Preconditions for unsafe acts

This level helps in understanding ‘latent conditions’ that facilitated unsafe acts by the staff involved in the incident. In the condition of operators subcategory (adverse mental state), provider A had to put in an amount of cognitive effort when operating with the CPOE system due to how the interface was designed which can cause stress. The personnel factors indicate inadequate communication between both providers during the changeover process. Provider A and the attending nurse regarded the misinterpretation of the order and between providers in communicating the actual result of the patient’s KCI level. The environmental factors, specifically ‘technological environment’, relate to CPOE system’s interfaces creating difficulties for providers due to the way it was designed (i.e. lacking an interface for allowing instructions to be written in comment boxes), and thus have usability issues. There was no additional evidence to suggest any contributing factors from the ‘physical environment’.

Unsafe supervision

Failures under the ‘inadequate supervision’ include not providing adequate technical training and guidelines on the use of the existing CPOE system. Failures can also potentially include not addressing issues regarding the use of the CPOE system as well as change-over processes. Also, under the subcategories ‘failure to correct problem’ and ‘supervisory violations’, failures here emanate from not identifying initial risks to the patient becoming hyperkalemic. In addition, issues relating to the usability of the CPOE system and failure to initiate corrective plans due to possible lack of investigation on ‘Health IT’-related events were not reported. However, there was not enough evidence that this will be considered a valid contributing factor.

Organisational influences level

Not all categories in this level could be used for classification. At the ‘organisational process’ category, procedures relating to changeover between providers was identified (procedures). Under the ‘resource management’ category, purchase of an unsuitable technological product (equipment/facility resources) and inadequate training in using the CPOE system (human resources) can only be inferred based on the unsafe events that took place and why it happened. However, more evidence will be needed to validate these factors.

STAMP/STPA result

In applying causal analysis based on STAMP, a high-level safety control structure (see Figure 7) and a more focused control structure where the incident occurred were developed (see Figure 8). Using the STAMP taxonomy, interactions between the providers and the CPOE were analysed for identifying safety requirements and constraints, inadequate/missing feedback and inadequate/unsafe control actions (see Table 3). Some of the unsafe control actions relating to the providers include, for instance, the first provider not entering the dosage order in the system correctly. This is also in addition to provider B not confirming previous orders before ordering additional KCI dose. Some of the unsafe control actions relating to the CPOE system include the system not having controls for calculating dosage values automatically and the system only being able to specify drip orders with the default stop time of 7 days (see Table 3). There was also inadequate/missing feedback relating to changeover between providers and also for the CPOE system not differentiating different screen entry forms for drugs and IV injection.

Figure 7.

The STAMP control structure of the CPOE medication error case study.

Figure 8.

STAMP analysis of lower level system components (interaction between the providers and the clinical information system).

Table 3.

Control failure taxonomy of control flaws leading to hazards from the case study.

Accident
Patient severely became hyperkalemic (overdose of potassium chloride (KCI))
Hazard(s)
The patient received multiple doses of KCI IV and drip over a 42 h period (total of 316 mEq).
Provider A and Provider B
Inadequate/unsafe control actions	Safety requirements and constraints
● Provider A did not confirm if any similar prior order had been cancelled before entering a duplicate dosage order.● Provider A did not correctly enter the dosage order into the CPOE system.● Provider B did not confirm any previous orders entered in the system before administering additional KCI.	● Provider A must confirm if previous orders were given to the patient to avoid entering duplicate orders.● Provider A must refer to guidelines in administering dosage correctly.● Provider B must be able to confirm previously entered orders to be within safe levels before administering the additional dose.
Inadequate or Missing Feedback
● Inadequate feedback between providers A and B regarding patient’s KCI levels during changeover (shift change).● Provider B did not correctly interpret the actual KCI value, mistakenly an earlier result for the current value.
Computerised Order Provider Entry System (CPOE)
Unsafe Control Actions		Safety Requirements and Constraints
● The system can only specify drip orders by duration with a default stop time of 7 days.● The system did not have adequate controls for distinguishing between time-limited (drips) and amount-limited (boluses) dosing.● The system did not have controls for automatically calculating dosage values.● CPOE did not provide inbuilt safety alert regarding duplication of similar orders previously entered or discontinued.		● CPOE system must allow providers to set flexible start and stop times when administering medicated drips.● CPOE system must include clear controls for distinguishing both drips and boluses.● CPOE system must be able to automatically calculate dosage calculations for all medicated drips.● CPOE system must alert providers on duplicate or discontinued dosage orders.
Inadequate or missing feedback
● The system did not clearly distinguish different screen entry forms for IV injection and drips.● Inadequate feedback from the CPOE system due to ambiguity of terms (total volume) and alerting providers on the latest KCI readings at real-time.● CPOE did not provide safety alert for provider B regarding the total KCI amount administered to the patient.

AcciMap result

Branford’s AcciMap model was applied on the case incident from the original documentation⁹ as well as the additional abridged information from other sources³ (see Figure 9).

Figure 9.

AcciMap analysis of the medication dosing error related to CPOE system (based on Branford’s standardised AcciMap format).

Based on the result obtained, the contributing factors identified are discussed below:

Outcome(s) of the case incident

This indicates the adverse outcome on the patient where he suffers severe hypokalaemia because of multiple doses of KCI.

Physical processes and actor activities

This level describes a combination of events carried out by different ‘actors’ that directly affected the patient including how the conditions may have affected them. The first provider (A) had given the patient a high dose of KCI than was originally intended because the order was not entered correctly into the CPOE system. Provider A also discontinued the wrong KCI order and mistakenly allowed an initial KCI dosage to be administered. During the changeover, provider A did not adequately inform provider B regarding the KCI dosage administered. This was due to the first provider assuming that the situation was resolved. This then led to provider B mistakenly taking an earlier result of the KCI as the current and then administering an additional KCI. Also, the nurse that was initially involved did not also notice the duplication of the dosage orders entered.

Organisational level

Contributing factors at this level include inadequate training in using the CPOE system and usability issues. These include the design of the interface including small fonts and clarity of safety-critical information. Another contributing factor was that neither the CPOE application nor the pharmacy system was programmed to notify excessive dosage or duplicate order entries. These factors stem due to inadequate human factors integration during the design and testing of the CPOE application.

External level

There was no explicit evidence to suggest that there were any external contributing factors that created the conditions for the accident to occur.

Usage characteristics of selected accident analytical approaches

The usage characteristics criteria were used to summarise the application of the selected approaches (see Table 4). However, three important characteristics, usability, validity and reliability, are further discussed.

Table 4.

Summary of usage characteristics comparison (adapted from Underwood and Waterson⁷).

Model characteristics	Events and causal factor (ECF) analysis	Human factors and classification systems (HFACS) analysis	System-theoretic accident model and processes (STAMP) analysis	AcciMap model analysis
Graphical representation of the accident	● Analysing events leading to the accident in a chronological and linear way.● Uses different symbols for denoting specific meanings thus providing effective visual communication.	● Utilises taxonomy of failures for analysing and classifying causes.● Does not provide any diagrammatic tools for visual communication.	● Its taxonomy provides a platform for analysing deviations (in terms of component and system behaviour) between different components for both lower and higher levels.● Complex visual presentation in depicting dysfunctional relationships.	● Provides a very effective way of visual communication regarding the accident (adverse event) that took place.● Depicts causal relationships within the different hierarchical levels of the model.
Data requirements	Data required based on different formats including incident reports, interview documentation and observation. However, this is applicable when the method is used in a reactive way rather than proactively.
Usability/ease of learning	● Requires substantial knowledge, understanding and representing the sequence of events leading up to the failure.	● Requires substantial knowledge, understanding and application of the model in analysing incidents and classifying the failures into its different subcategories.	● Requires substantial knowledge, understanding of the concept of control theory as well as its resultant system taxonomy for analysing deviations between component interactions.	● Requires substantial knowledge in understanding but simple to utilise for representing the causal relationships (distal and proximal).
Usability/ease of learning	● Easy to learn and use for analysing causal factors.	● Does not require too much guidance in its analysis.	● The level of usage of the method is dependent on the knowledge, skills and experience of the user as a standalone or within a multidisciplinary team setup.	● Lack of a standard guide for conducting the analysis.
Validity of analysis	Useful in multiple safety-critical domains.External validity is required.
Validity of analysis	● Utilised with other techniques like Management Oversight and Risk Tree (MORT) based accident investigation procedures, barrier and change analysis.	● Based on Reason’s theory of accident causation which provides a systematic classification for latent and active failures.	● Based on system and control theory regarding accident causation for analysing system complexity.	● Based originally on Rasmussen’s theory of accident causation for analysing structure and dynamics within complex sociotechnical systems.
Reliability of analysis	● A qualitative approach for analysing case incidents.● The analysis of an accident is open to interpretation by different investigators and so reliability is not high.	● Provides qualitative and quantitative means of analysing both single and multiple cases.● Its taxonomy allows investigators to analyse active and latent failures and the contributing factors allowing for higher reliability to be achieved.	● Provides a qualitative approach for analysing dysfunctions in the system. Its taxonomy is not very detailed for classifying failures and contributory factors.● Based on control theory which allows for a considerable level of reliability.	● Provides a qualitative approach for analysing multiple proximal, distal causes and contributory factors across different levels.● Its reliability is low due to its subjective nature (open interpretation to different analysts).
Sociotechnical analysis	● Its theoretical approach does not make the method the most suitable for analysing incidents/accidents from complex interactions in a sociotechnical system.	● Its theoretical approach allows for the classification of multiple contributing factors from the lower to higher levels of failure within the system.● Its current taxonomy does not consider other higher levels of systemic failures (e.g. regulatory bodies, government)	● Its theoretical approach allows for system flaws and causal factors of hazards to be analysed from interactions between system components existing in both the lower levels to the higher levels of the sociotechnical system.● Suitable for sociotechnical analysis of (possible) accidents in sociotechnical systems.	● Its theoretical approach allows for a sociotechnical analysis of failures and contributing factors across hierarchical levels within the sociotechnical system.

AcciMap: accident mapping.

Usability/ease of learning

Substantial knowledge, especially for first-time users, is required to model in using these approaches. From this experience, each of the models presented contrasting challenges. The utilisation of ECF requires following guidelines in representing the chronological sequence of events. An understanding of human error, as well as Reason’s Swiss Cheese Model²⁶ which the taxonomy is built on, is at the very least a basic requirement. While using all the approaches require a considerable amount of time, cognitive effort and knowledge of the domain, ECF, HFACS and AcciMap were less challenging in its application and more intuitive than STAMP.

Modelling the accident using the STAMP model requires a thorough understanding of system and control theory concepts to be able to effectively analyse accidents. This in addition to understanding how to apply the model’s taxonomy for identifying unsafe controls and safety constraints within a complex sociotechnical system took a great deal of time. Despite AcciMap being relatively easier to apply than STAMP, it still requires following a set of guidelines in correctly identifying valid contributing factors, placing them in the right level and setting logical causal relationships between them. Of the approaches, ECF provides a simplified way of modelling and describing the events leading to the accident. While this may be the case, counterfactual reasoning was required in determining the necessary events needed to have caused the accident.

Validity of analysis

Formal validity measurements were not carried out in this study, but validation with experienced analysts (human factors specialists) was required for outputs of the analyses. Each of them is based on a recognised theory of accident causation and has been applied across multiple safety-critical domains for accident analysis (Salmon et al., 2012).⁷ To formally measure the validity of each approach used, a study will need to be taken that involves comparing outcomes and safety recommendations taken by multiple analysts and experienced/expert opinion.²²

While each model presents different methodological approaches for analysing accidents, AcciMap and STAMP allows users to determine why accidents occurred using systems thinking and looking at the entire sociotechnical system. The classification scheme of the HFACS model provides guidance in data collection and analysis of singular and multiple incidents. However, HFACS overall is limited because of the restrictive nature of its classification scheme in addition to not considering contributing factors at the external level.

Reliability of analysis

No formal reliability measurements were carried out in this study. However, HFACS’s taxonomy can potentially enable multiple analysts to produce similar results from the case study. This is because of its reliability when classifying human errors^27,28 and is considered moderately higher in comparison to AcciMap and STAMP models.^6,7 While the methodological approach regarding the use of AcciMap allows analysts to construct multiple causal relationships, the reliability of the AcciMap model has been mixed with past studies considering it to be low.⁷ Its outcomes can vary between multiple investigators due to lack of a classification scheme.²⁸

A similar conclusion on potential variations if undertaken by multiple investigators can be applied to the ECF charting approach. The STAMP’s generic taxonomy should allow for some considerable levels of reliability. Ideally, results from the use of these approaches would need to be re-evaluated from other analysts using both intra- and inter-reliability measurements to determine if similar outcomes and recommendations will be produced.

Discussion

Contributing factors from the analyses

Based on the results, each approach produced a set of contributing factors/control flaws based on its underlying methodology (see Table 5). This can potentially influence the development of safety recommendations for mitigating and preventing its reoccurrence. Common themes regarding contributing factors identified from these approaches include ‘inadequate communication’ between providers (A and B) and usability of the CPOE application. In the case of the ECF output, ‘lack of/or poor communication’ between providers A and B played a role in the latter not knowing the patient’s true KCI levels. However, the question that will be asked will be ‘why’ there was a miscommunication. In this case, provider A may have assumed that the CPOE system would emphasise the current KCI value for provider B to access.

Table 5.

Contributing factors (ECF, HFACS and AcciMap) and control flaws (STAMP) from case incident.

System category	Events and causal factors (ECF) – contributing factors	Human factors and classification systems (HFACS) – contributing factors	Accident mapping (AcciMap) – contributing factors	System-theoretic accident model and processes (STAMP) – control flaws
Physical actors involved providers A and B	● Provider A did not enter the new order (80 mEq/KCL) correctly into the CPOE system.● Communication lapse between providers A and B regarding patient’s KCI levels during the changeover.● Provider B did not realise the laboratory results were from before the last potassium depletion.	● Provider A made several errors including ° Medicated drip order not entered correctly (skill-based error), ° Re-enters same order which ran for 36 h (decision error) and ° Misjudging KCI levels by both providers (A and B) (perceptual error).● Exception violation relating to patient’s KCI levels not checked between providers during clinical handover.● Inadequate communication between providers A and B regarding patient’s KCI values.	● Provider A discontinued a wrong order and thereby allowing the initial KCI dosage order to be administered.● Provider A entered the dosage incorrectly into the CPOE system resulting in a higher KCI dosage than intended.● Miscommunication between providers A and B during the changeover.● Provider B did not take note of the latest KCI results preceded potassium repletion.● Provider B ordered an additional KCI to be administered to the patient.● Nursing staff did not notice and report KCI dosage duplication.	● Unsafe Control Action: ° Provider A not confirming if any similar prior order had been cancelled before proceeding with a duplicate order. ° Provider A not correctly entering dosage order in the CPOE system. ° Provider B not confirming prior orders entered before administering additional KCI order.● Missing feedback: ° Issues in communication and coordination between providers A and B on the patient’s KCI levels. ° Provider B incorrectly interpreted earlier KCI value as the current one.
Technological component health IT system (computerised order entry system)	● The CPOE system’s user interface design was not intuitive and made entering dosage orders challenging.	● Stress and fatigue in using the CPOE system due to how the interface was designed (e.g. small fonts, similar windows).● The CPOE system was not adequately designed (usability issues, e.g. start and stop time calculations).	● CPOE system did not visually emphasise the latest KCI results.● Misidentification of order entries due to user interface issues (small font sizes, high-density content).● CPOE system containing ambiguous labels and not providing automatic dosage calculations.Neither CPOE application nor pharmacy system was programmed in notifying users of excessive or duplicate orders.● KCI IV drips not displayed on CPOE’s medication list.	● Unsafe Control Actions: ° Only specified drip orders by duration with a default stop time of 7 days. ° Did not have adequate controls for differentiating between amount-limited (boluses) and time-limited (drips) dosing. ° Did not have controls for automatically calculating dosage values. ° No inbuilt safety alert regarding similar orders duplicated that were previously entered or stopped. ° Missing feedback: ° No clear distinction on different screen entry forms for both drips and IV injections. ° None/inadequate safety alert regarding total KCI amount administered to the patient. ° Inadequate feedback due to the ambiguity of terms (total volume) ° Lack of safety alerts on patient’s KCI levels at real-time.
Organisational hospital	● No contributing factors	● Failure in not reporting about issues regarding the operation of the CPOE system by providers.● Inadequate training on how to use the current CPOE system in operation.● Purchase of an unsuitable technological product.● Inadequate procedures regarding changeover between providers.	● Inadequate training in CPOE system usage.● Inadequate human factors integration during CPOE interface design and testing.	● No control flaws identified due to lack of information.
External	● No contributing factors identified.			● No control flaws identified due to lack of information.

KCI: pottasium choloride; CPOE: computerised order provider entry system.

On further examining the ECF result, it can be deduced that the dosage order was not entered correctly due to design issues of the CPOE system. These design flaws would be considered another necessary cause for provider A’s difficulty in entering the dosage order. Applying counterfactual reasoning to the ECF analysis, it can be argued that if provider A had properly informed provider B about the patient’s KCI status, then the likelihood of the patient receiving a higher dosage of KCI would have been reduced or prevented. Another counterfactual claim would be that if the CPOE interface was designed to have clarity in the labels indicated on the interface and a comment box to allow the first provider and nurse to easily enter the dosage order, then the chance of an error in inputting the dosage order would have been reduced. In the case of the incident between the nurse and the first physician (provider A), if the nurse had reconfirmed the order that was given by the provider, the value would have been entered incorrectly into the CPOE interface.

From the HFACS result, the categories defined allow for contributing factors to be classified but more importantly to determine the nature of unsafe acts, pre-conditions to these acts and any organisational influences. Several factors similarly identified in ECF include errors (skill, decision and perception) committed by both providers. Other factors include communication flow between both providers and the CPOE system having usability issues due to design flaws. Other contributing factors classified included issues relating to inadequate technical training on the use of the current CPOE application (unsafe supervision) as well as factors relating to procedures (handover) and purchase of the system not very suitable for this clinical purpose.

Contributing factors identified using the AcciMap approach produced similar factors with additional ones relating to the CPOE system. These include both CPOE and pharmacy systems not being programmed to notify providers on duplicate orders and drips not being displayed on the medication list. All these stem from the lack of human factors integration into the design of the CPOE which was identified at the organisational level. The STAMP model differs from AcciMap, HFACS and ECF due to its focus on violations in safety from interacting system components rather than from an initiating event. From the STAMP/STPA analysis, unsafe control actions and missing/inadequate feedbacks regarding providers (A and B) and the clinical IT system (CPOE) are identified as well as safety constraints. These include provider A not confirming if similar orders were cancelled, and provider B not confirming orders entered previously by provider A. Unsafe control actions relating to the CPOE include not having safety controls for alerting on similar or duplicate orders previously entered, and only having a default setting of 7 days duration for drip orders.

Comparison of safety recommendations

Safety recommendations formulated from the outcomes also produced some similar measures (see Table 6). These include redesigning of user interfaces incorporating best practices for system usability (ECF, AcciMap, HFACS, STAMP) and implementation of safety alerts on the CPOE (STAMP). Also included is the application of human factors in designing IT systems (AcciMap) and improving existing feedback protocols during the changeover between providers (ECF, HFACS, AcciMap, STAMP). Other safety measures not considered from the analyses include, for example, creating policies of simulated testing of health IT systems which can also serve as a way of training staff in using them. Also, the amount and type of alerts will need to be considered to avoid scenarios where providers will be bombarded with may alerts which could lead to alert fatigue. This can potentially pose a risk of ignoring important alerts that could act as a safety barrier.

Table 6.

Safety recommendations from the accident analytical approaches.

ECF-based safety recommendations	HFACS-based safety recommendations	AcciMap-based safety recommendations	STAMP-based safety recommendations
1. Policies regarding changeover between providers must be reviewed to ensure that the patient’s status is fully known to avoid miscalculations.2. Providers must be trained to be able to ascertain if prior similar orders were given or discontinued before proceeding with entering new dosage orders.3. The interface of the CPOE system must be designed to differentiate between different screens clearly as well as ensuring that safety critical information is not high in density containing relevant information.4. Providers need to be provided with training especially regarding confirming prior orders and entering them correctly in the CPOE system provided (based on if point 3 is not implemented).	1. Review of existing procedures regarding clinical changeover between providers on the condition of the patient and decisions made during clinical shifts.2. Review of the CPOE system focusing on improving the intuitiveness, clarity of labels and setting appropriate alerts to avoid cognitive overload during operation.3. Review the training of providers in using the CPOE system and to ensure that dosage orders are entered correctly. This also includes encouraging a culture of confirmation of orders and following the hospital’s policy of maximum dosage limit.	1. Comprehensive human factors review and interface design evaluation of the CPOE system to be undertaken and action taken to facilitate error reduction, detection and recovery.2. The CPOE interface design should be reviewed and revised to ensure that:a. The currency of test results is clearly evidentb. Medications provided by IV drips are included in medication listsc. Human–computer interaction design principles are followed to facilitate easy identification and interpretation of order entries, andd. IV dosage input options are clear, unambiguous, meet requirements (expectations) and provide automatic dosage calculations to aid error prevention.3. The CPOE application should be programmed to notify clinicians of excessive dosage orders and duplicate therapy.	1. Creating protocols for providers to confirm prior orders given to patients to avoid entering duplicate orders.2. Creating or improving safety guidelines for clinical changeover between providers to avoid lapses in communication and feedback.3. Reviewing current CPOE application and incorporating appropriate additional controls including:4. Alerting providers on duplicate orders or if any similar orders were given or discontinued.a. Automated calculation of dosage orders for medicated drips.b. Providing flexibility for providers in setting ‘start’ and ‘stop’ times when administering dosage orders.c. Providing a real-time update on the KCI amount given to ensure levels are within an acceptable range to reduce the risk of overdose.

ECF: events and causal factors; HFACS: human factors and classification systems; AcciMap: accident mapping; STAMP: system-theoretic accident model and processes; KCI: pottasium choloride; CPOE: computerised order provider entry system.

Experiences in the application of selected accident analytical approaches

Applying each approach in analysing the incident presents different ways of analysing why the adverse event occurred. While ECF, HFACS and AcciMap generally employ a traditional approach of analysing accidents by extracting contributing factors, the STAMP approach focuses on determining unsafe control actions and constraints that were violated. Based on the usage characteristics criteria used to summarise differences of each accident approach, it is also very important to determine which of these approaches provide an overall benefit when carrying incident analysis in healthcare systems. Based on experience using these approaches, the ECF would be considered to be the most straightforward, intuitive and less time-consuming approach compared to the others. However, due to its linear approach in plotting sequences of events, it will be considered limited when modelling multiple causal interactions that typify complex sociotechnical systems. When considering the HFACS approach, with its classification scheme built upon a recognised theory of accident causation, categories defined will need to be specific to considering causal factors related to healthcare to make it more practically useful. In using this approach, some of the categories could not be used due in part to lack or no information to support evidence. The HFACS approach also does not have categories to consider external factors.

Similar to ECF, applying the AcciMap model did not require much effort in understanding how it is applied especially when following Branford’s guidelines. Key aspects include identifying valid contributing factors, placing them at the appropriate system level and inserting causal links between these factors. This gives an overall picture of how these multiple interactions proceeded to the accident. However, this ‘freedom’ of analysis introduces issues relating to subjective results based on analysts’ perception and understanding of the incident. A major criticism of the AcciMap approach is its reliability ranging from low to mixed.⁷ The STAMP model certainly provides an alternative way of analysing accidents and can be used in conjunction with other approaches to provide a deeper understanding of the loss of control. STAMP’s theoretical underpinning can create challenges in classifying human and organisational failures within its taxonomy.²⁸ As powerful as the STAMP approach is, it is the least intuitive and the most time-consuming approach compared to the others.

Overall, in terms of being able to identify and graphically present multiple causal relationships in complex sociotechnical systems, AcciMap would be considered the most suitable of the approaches used in this study. However, its reliability and validity will need to be further improved to be considered a valuable tool for accident investigation and analysis in healthcare systems.

Limitation of study

The approaches applied to this type of incident not only presented different ways of analysing what went wrong but also presented challenges. There is the challenge of having different forms of bias when investigating and analysing why the accident occurred including hindsight and outcome bias. This was reduced by trying to analyse the case with the information extracted from multiple sources without using the original outcomes and recommendations.

It would have been more ideal to have access to additional information particularly from sources like interview transcripts of clinicians who were involved. This would have allowed for comparison with the previous author’s findings for similarities or differences. In addition, analyses and results from the application of these approaches were carried out by the principal author and reviewed by experienced practitioners. This approach was taken to ensure that results obtained were valid in the absence of no formal reliability and validity measurements that could be used for this study.

These issues can potentially determine how effective and practical safety recommendations developed can prevent reoccurrence. Another limitation in this study was that it was not extended to compare with the original recommendations from the incident. If this study was to be repeated, it will need to be applied to major cases with substantial supporting information to gain further insight into systemic issues. Also, for reliability and validity testing, each of the approaches can be analysed by multiple experienced investigators. Their findings can then be assessed for the reliability of the model as well as validating safety recommendations formulated from their analyses.

Contribution to knowledge

This study demonstrated the application of different accident analytical approaches in an example case incident. It was also important to determine which of them will be considered the most practically suitable for determining contributing factors including systemic factors in healthcare systems. However, while there is an acknowledgement of the strengths and limitations of each approach, it is important to consider crucial characteristics in determining which accident analytical approach to use. These include its ability to depict complex interactions, how intuitive and easy it is to apply and how much time it takes to learn and understand how they are applied.

Conclusion

The comparison of different accident approaches for investigating health IT–related safety risks demonstrates the strengths and limitations of their respective methodological approach. Despite each of their respective benefits, AcciMap and STAMP approaches generally provide a ‘systems thinking’ perspective when investigating and analysing incidents in complex systems. However, to be able to effectively apply these approaches, their theoretical foundations and guidelines need to be understood and applied correctly to obtain reliable and valid results. This will allow effective safety recommendations to be developed.

There is also the current challenge of how existing accident analytical approaches like the AcciMap’s reliability and validity can be further improved and hence be more practically useful in healthcare organisations. According to Salmon et al’.s²⁹ study, a domain-specific classification scheme like the HFACS approach can be developed and synthesised within the AcciMap methodology. This methodology can be similarly applied within the healthcare context for analysing health IT–related accidents. To achieve this, grounded taxonomies will need to be examined to develop a health-specific classification scheme of contributing factors incorporated within the AcciMap model.

Footnotes

Acknowledgements

Analyses and results obtained from the case study were reviewed by Dr Kate Branford, a human factors specialist with the railway industry, Australia and Clear Thinking consultant Dr Suzanne Shale and HFACS taxonomy specialist. Further acknowledgement also goes to Dr Maria Mikela Chatzimichailidou, Systems Engineer of the Imperial College, London, and Mr Iain Bishop, an e-Health Pharmacy Adviser, Information Technology of the National Services Scotland (NSS).

Declaration of conflicting interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) received no financial support for the research, authorship, and/or publication of this article.

ORCID iD

Oseghale Osezua Igene

References

Sittig

Singh

A new socio-technical model for studying health information technology in complex adaptive healthcare systems. Qual Saf Health Care 2010; 19(Suppl. 3): i68–i74.

Magrabi

Ong

Coiera

Health IT for patient safety and improving the safety of health IT. Evid-Based Heal Inform 2016; 222: 25–36.

IOM. Health IT and patient safety: building safer systems for better care, 2012, https://iom.nationalacademies.org/~/media/Files/Report%20Files/2011/Health-IT/HIT%20and%20Patient%20Safety.pdf

Koppel

Metley

Cohen

, et al. Role of computerized physician order entry systems in facilitating medication errors. JAMA 2005; 293(10): 1197–1203.

Qureshi

. A review of accident modelling approaches for complex critical socio-technical systems. Report No. DSTO-TR-2094. Department of Science and Technology Organisation, Department of Defence, Australian Government, 2008, https://pdfs.semanticscholar.org/0a98/e16b80777a40f206f3a7f942c2fb17b43173.pdf

Underwood

Waterson

Systemic accident analysis: examining the gap between research and practice. J Accid Anal Prev 2013; 55: 154–164.

Underwood

Waterson

Systems thinking, the Swiss cheese model and accident analysis: a comparative systemic analysis of the Grayrigg train derailment using the ATSB, AcciMap and STAMP models. J Accid Anal Prev 2014; 68: 75–94.

Yin

RK.

Case study research: design and methods. 2nd ed. Newbury Park, CA: SAGE, 1994.

Horsky

Kuperman

Patel

VL.

Comprehensive analysis of a medication dosing error related to CPOE. J Am Med Inform Assoc 2005; 12(4): 378–381.

10.

Kingston

Koornneef

Events and conditional factors analysis manual. 2nd ed. Delft: The Noordwijk Risk Initiative Foundation, 2014, https://www.nri.eu.com/NRI4.pdf

11.

Svedung

Rasmussen

Graphic representation of accident scenarios: mapping system structure and the causation of accidents. J Safety Sci 2002; 40(5): 397–417.

12.

Leveson

NG.

System safety engineering: back to the future. Cambridge, MA; Department of Aeronautics and Astronautics, Massachusetts Institute of Technology, 2002, http://sunnyday.mit.edu/book2.pdf

13.

Buys

Clark

JL.

Events and causal factors analysis. Idaho Falls, ID: Technical Research and Analysis, SCIENTECH, 1995, https://exclusive.iosh.com/media/2053/events-and-casual-factors-chiltern-january-2017.pdf

14.

Johnson

CW.

Failure in safety-critical systems: a handbook of accident and incident reporting. Glasgow: University of Glasgow Press, 2003, http://www.dcs.gla.ac.uk/~johnson/book/parts/chap10.pdf

15.

Johnson

WG.

The management oversight and risk tree analysis. Technical Report No. SAN 8212, 12 February 1973. Washington, DC: Atomic Energy Commission.

16.

Diller

Helmrich

Dunning

, et al. The human factors analysis classification system (HFACS) applied to health care. Am J Med Qual 2013; 29(3): 181–190.

17.

Shappell

Wiegmann

DA.

The human factors analysis and classification system-HFACS. 2000, https://www.nifc.gov/fireInfo/fireInfo_documents/humanfactors_classAnly.pdf

18.

Wiegmann

Shappell

A human error approach to aviation accident analysis: the human factors analysis and classification system. Burlington, VT: Ashgate, 2003.

19.

Rasmussen

Svedung

Proactive risk management in a dynamic society. Karlstad: Swedish Rescue Services Agency, 2000, https://www.msb.se/RibData/Filer/pdf/16252.pdf

20.

Branford

Naikar

Hopkins

Guidelines for AcciMap analysis. In Hopkins

(Ed.), Learning from High Reliability Organisations. Sydney: CCH. 2009.

21.

Rasmussen

Risk management in a dynamic society: a modelling problem. J Safety Sci 1997; 27(2/3): 183–213.

22.

Branford

An investigation into the reliability and validity of the AcciMap approach. Unpublished PhD Thesis, The Australian National University, Canberra, ACT, Australia, 2007.

23.

Branford

Seeing the big picture of mishaps: applying the AcciMap approach to analyse system accidents. Aviat Psychol Appl Hum Fact 2011; 1: 31–37.

24.

Leveson

Daouk

Dulac

, et al. Applying STAMP in accident analysis. 2004, http://shemesh.larc.nasa.gov/iria03/p13-leveson.pdf

25.

Johnson

CW.

An introduction to root cause analysis in healthcare. 2004, http://www.dcs.gla.ac.uk/~johnson/papers/Pascale_book/incident_analysis.PDF

26.

Reason

Understanding adverse events: human factors. Qual Health Care. 1995; 4(2): 80–89.

27.

Ergai

Assessment of the human factors analysis and classification system (HFACS): intra-rater and inter-rater reliability. 2013, https://tigerprints.clemson.edu/all_dissertations/1231

28.

Salmon

Cornelissen

Trotter

MJ.

Systems-based accident analysis methods: a comparison of AcciMap, HFACS, and STAMP. J Safety Sci 2012; 50(4): 1158–1170.

29.

Salmon

Goode

Taylor

, et al. Rasmussen’s legacy in the great outdoors: a new incident reporting and learning system for led outdoor activities. Appl Ergon 2017. 59(Pt. B): 637–648.