Abstract
Console operators in process plants have to maintain a high level of situation awareness to operate the plant safely, effectively, and efficiently. An overview display is one of the primary displays in a control room that operators monitor to gain and maintain an understanding of the plant. In this study, the authors evaluated operator performance using two overview display formats. The first format, characterized as a functional design, included qualitative, graphical indicators for process parameters and organized the position of the indicators on the basis of functional relations of the process equipment. The second format, characterized as a traditional schematic display, showed connecting process lines between equipment and numerical fields to present process information. Both displays contained the same indicator values. Eighteen plant operators used both display formats to monitor a crude unit process for process parameters that deviated from normal values. We measured operators’ situation awareness using think-aloud protocols and situation awareness global assessment technique, subjective workload, and usability ratings. Results indicated that operators’ situation awareness was significantly higher when they monitored the process on a functional display compared with a schematic display. Their subjective workload and usability ratings also favored the functional overview display format. Implications of the findings for continuous process control and overview display design are discussed.
Introduction
C
An overview display is one potential tool that can support operators in maintaining awareness of the plant’s status. Because process monitoring is a complex and cognitively demanding task, it is critical to design overview displays that effectively represent the most critical information to operators. Advanced visualization and cognitive engineering techniques have shown that there are effective methods to represent process information to a console operator (Burns & Hajdukiewicz, 2004; Jamieson & Vicente, 2001; Vicente & Rasmussen, 1990). However, most operator consoles in the hydrocarbon-processing industry still use traditional schematic displays based on piping-and-instrumentation diagrams (P&ID) to present data and information via numerical indicators. Hence, the Abnormal Situation Management® (ASM) Consortium undertook an effort to design and develop effective indicator shapes and displays that can qualitatively represent process data and information to console operators to support at-a-glace awareness of plant status (Reising & Bullemer, 2008).
A prevailing philosophy in the hydrocarbon processing industry is that an effective human-machine interface for the console operator will simultaneously distribute information across a multilevel display hierarchy (Bullemer, Reising, Burns, Hajdukiewicz, & Andrzejewski, 2008). At the top of such a hierarchy, a span-of-control overview display is designed to support operator monitoring for the existence, severity, location, and direction of process deviations. The subsequent “children” displays in the hierarchy then provide increasing levels of detail for equipment areas or specific equipment that supports operators’ troubleshooting and response actions after they recognize that a process deviation exists (for a detailed review, see Bullemer et al., 2008).
To recognize that a problem exists, it is essential for operators to maintain good situation awareness (SA) during process monitoring. SA is defined as the “perception of the elements in the environment within a volume of time and space, the comprehension of their meaning, and the projection of their status in the near future” (Endsley, 1988, p. 97). More specifically, the perception of the elements in the environment within a volume of time and space has been labeled as Level 1 SA, the comprehension of their meaning has been labeled Level 2 SA, and the projection of their status in the near future has been labeled Level 3 SA. Problems with SA were found to be the leading causal factor for several industrial accidents and aviation mishaps (e.g., Durso, Rawson, & Girotto, 2007). In addition, the lack of an overview display has been identified as a contributing factor in several significant hydrocarbon processing industry accidents (Health and Safety Executive, 1997; U.S. Chemical Safety and Hazard Investigation Board, 2007). Therefore, it is essential to design displays that can help operators to maintain good SA.
The ASM Consortium initiated several research projects to develop effective qualitative indicators and displays for an operator console so as to improve operator SA while monitoring a process (e.g., Reising & Bullemer, 2008). Although the new qualitative indicators were developed for use in overview displays, no prior studies had evaluated how a display that uses the qualitative indicators would affect an operator’s SA while monitoring a process. Therefore, the primary objective of this study was to evaluate how functionally organized qualitative indicators on overview displays support operator SA compared with schematic overview displays with numeric indicators.
Functional Versus Schematic Overview Displays
There are two key differences between the functional overview display and the traditional schematic overview display. The first difference relates to how individual process parameters are visualized. The second difference relates to whether the visualization supports perceptual recognition of qualitative process deviations (Flach & Bennett, 1992; Jessa & Burns, 2000) or whether it demands mental operation to identify deviations.
In the functional overview display, dynamic data are visually represented in a qualitative manner. In addition, the parameters are organized according to a predetermined functional dimension that is based on process flows and interrelations of the operating equipment. In this study, the overview displays were designed for a crude unit at a refinery that consisted of a crude distillation unit and a naphtha hydrotreater unit. As a result, the functional organization for the functional overview display used in this study was eight major subsystems of equipment, namely, crude feed, crude feed heater, vacuum heater, debutanizer, crude tower, vacuum tower, hydrotreater, and stripper. The major subsystems were organized on the display (i.e., left-to-right, top-to-bottom organization) consistent with the overall process flow in the plant. Within each major equipment subsystem, the individual display indicators were organized on the basis of spatial relations of the equipment or process flow, to the extent that it was feasible. The functional overview display used in this study is shown in Figure 1.
Schematic representation of the functional overview display.
The functional overview display included three types of qualitative indicators—analog gauges, qualitative objects, and controller output objects—to display the “dynamic” process parameter information. Qualitative indicators show actual values relative to important reference values to enable the operator to directly perceive the qualitative status of the parameter, such as abnormal, off spec, in alarm, or fully open. A detailed description of the appearance and behavior of these indicators has been reported elsewhere (Reising & Bullemer, 2008). As a guiding principle, the new indicators were designed to support the “direct perception” (Flach & Vicente, 1989) of the state of the parameter being displayed.
In this study, four analog gauges were used to represent temperature, pressure, level, and flow. As represented in Figure 2, when an underlying process parameter value exceeded its normal or alarm limit, the visual coding of the object changed to draw the viewer’s attention (i.e., increased in salience). The shapes and sizes of the gauges were designed to make the most effective use of real estate on an overview screen while simultaneously providing critical information effectively. In addition, three types of generic qualitative objects were used in this study: quality indicator, change indicator, and deviation indicator. Qualitative objects are used for functional parameters, such as conversion rate, opacity, or ratios. These objects are generic in the sense that they can be used in conjunction with any parameter type (flow, temperature, pressure, and so on). Finally, the controller output object is an indicator that shows the status of the controller output signal with reference to a valve. For the controller output object, the visual features illustrate the controller output percentage (OP %) relative to the normal, expected value and alarm limits.
A schematic representation of change in visual coding to attract operator’s attention. The magenta highlighting indicates an abnormal condition, light blue highlighting indicates a low-alarm condition, and red highlighting indicates a high-alarm condition.
In contrast to a functional overview display, a traditional schematic overview display uses graphic symbols to represent “static” information, such as equipment types and connecting process lines between the equipment to show material flows. In addition, dynamic data are visually represented in a quantitative manner with numerical indicators. These types of display are also referred to as “mimic” displays in the process industry (Moore & Corbridge, 1996). One potential advantage of the schematic overview display is that mimics have been used to design console operator displays for a long time. However, a critical disadvantage with such a representation is that operators have to execute more cognitive operations to understand the health of a parameter, unlike with the functional displays. Moreover, typically, static information requires more space in the display, which could otherwise be used to show additional, more critical dynamic information. The schematic overview display used in this study is shown in Figure 3.
Schematic representation of the schematic overview display.
In summary, a functional overview has at least three potential advantages. First, compared with a schematic display that shows changing numbers, a functional overview display shows shapes that can better capture an operator’s attention when a process deviates from a normal state to an abnormal or alarm state. A normal state is when a process parameter is within the normal variability, indicating that the plant is stable. An abnormal state is when the process parameter is outside the normal variability range but not yet in an alarm. An alarm state is when the process parameter has exceeded the predefined upper or lower alarm limit. Second, a functional overview display provides an operator with qualitative information (e.g., normal to abnormal to alarm) rather than quantitative information (e.g., current value, how much changed). The benefit in such a representation is that the quantitative information, which otherwise requires an operator to carry out cognitive operations to comprehend the health of a parameter, is shown directly. As a result, an otherwise more cognitively demanding task becomes a perceptual task (e.g., Jessa & Burns, 2000; Sanderson, Flach, Buttigieg, & Casey, 1989). Finally, the layout of the equipment areas in a functional overview display is consistent with the functional relations between the equipment areas in a plant. The functional overview display is not cluttered with static information, such as vessel shapes and process connection lines.
Purpose
The primary research question in this study was, “Are functional overview displays that are designed for detecting qualitative process deviations (i.e., normal to abnormal to alarm) more effective in supporting operator SA than schematic overview displays that use numerical indicators?” More specifically, because overview displays are intended to help operators maintain an overall awareness of the plant state, SA was measured from a perspective of whether operators were able to maintain awareness of whether the conditions are (qualitatively) normal, abnormal, or in alarm. We investigated the impact of the displays on operators’ Level 1 (perception) and Level 2 (comprehension) SA in a dual-task context using a within-subjects, repeated-measures experimental design. Finally, because this was the first step, we were primarily interested in discerning the differences in the ability of the displays to support Level 1 and Level 2 SA before measuring Level 3 (projection) SA. In addition, it was impossible for the research team to know ahead of time about the level of experience the operators would have with crude distillation processes. Hence, Level 3 SA queries were not included in the study.
Method
Participants
Eighteen professional console operators volunteered to participate in the study. The had following demographics were recorded: age (M = 42.56, SD = 8.48), current process unit experience in years (M = 10.11, SD = 8.35), other process unit experience in years (M = 6.11, SD = 7.30), field operations experience in years (M = 6.03, SD = 7.04), and computer usage in hours per day (M = 4.28, SD = 4.17).
Operators were recruited on the basis of their availability at the discretion of their shift supervisor. All operators were working their normal shift schedule, and their console was covered by another operator while they participated in the simulator evaluation. All operators were naive to the hypothesis of the experiment.
Apparatus
The experimental apparatus included two computer monitors. A functional or schematic overview was displayed on one computer screen. A Dell Precision M6300 laptop with a 17-in. monitor and screen resolution of 1,280 × 1,024 pixels was used to present the overview display.
To create an environment that had a good degree of overlap with the realistic work context, operators were asked to perform a flag-matching task while simultaneously monitoring the overview display. An elaborate description of this task and the underlying rationale is provided in a later section. A Dell Latitude D630 laptop with a 15-in. monitor and screen resolution of 1,280 × 800 pixels was used to present the flag-matching display. In short, the operators viewed the overview display on one computer screen and the flag-matching display on a separate computer screen.
Experiment Setup
First, we will define the steps adopted to develop the scenarios for the overview displays. Second, we will elaborate on the flag-matching task.
Scenario development for overview display
Both overview displays were designed for the span of control of a simulated crude unit, which consisted of a crude distillation unit and a naphtha hydrotreater unit. The functional and schematic overview displays were designed according to the same list of key process parameters. A display designer consulted with a process control engineer and a plant operations specialist to identify the key parameters for each of the major equipment areas in the unit’s span of control. It is important to note that both overview displays contained the same process parameters. However, the manner in which each parameter was represented in each display was different (functional = qualitative vs. schematic = numeric).
With regard to the overview displays, using the advice of a process expert, we identified several upset scenarios that were typical of upset events in actual crude units. The location of the initiating condition of an upset event was distributed across the major equipment areas to avoid expectation biases. The upset scenarios were created with the use of a high-fidelity commercial process simulator.
Four scenarios (namely, 1, 2, 3 and 4) were finally selected to be included in the experiment. Each scenario was presented to the operators as a prerecorded video via Windows Media Player. It is important to note that operators could only passively monitor the displays and, hence, could not intervene with the process in the scenario or change the outcome of a scenario. The primary rationale for such a setup was to assure that all operators were exposed to the same scenario conditions and events.
Flag-matching task
While simultaneously monitoring the overview display for process parameter deviations in the scenario videos, operators were asked to complete a flag-matching task. In an actual control room setting, operators often perform other work activities while monitoring their overview displays for process parameter deviations. Moreover, if operators are actually involved in responding to an abnormal situation, they would typically interact with many operating displays in the display hierarchy to evaluate the plant conditions, execute control actions, or coordinate activities with other plant personnel. Hence, monitoring an overview display may be seen as only one of many simultaneous tasks that operators are responsible for completing. However, without the workload from these other activities in this study, the monitoring task scenarios would potentially place too low of a mental workload on operators to evaluate the effects of the overview display in a realistic work context.
Given such study objectives, there were several important requirements for the additional task. First, the task needed to demand the use of similar cognitive resources as performing typical console operations activities. Second, the primary task needed to minimize the training time that would be required of the participants in the study. Third, the primary task needed to be measurable and quantifiable. On the basis of these requirements, the study team identified and selected a flag-matching task, which is a variant of the Concentration game (Pringle, 2000). Importantly, this task met all the aforementioned task requirements.
Operators were presented with a 6 × 8 grid of numbered tiles on a computer screen. When an operator selected a tile using the left mouse button on an optical mouse, a country flag would appear. Only the two most recently selected flags could be in view at a time. When the two flags in view matched, the tiles disappeared from the grid. In the event that all the flags were matched (i.e., all tiles disappeared), the grid was manually restarted to ensure a continuous task. The flag-matching application is represented in Figure 4.
Schematic representation of the primary flag-matching task interface.
Experimental Design
We adopted a repeated-measures experimental design. There were two main independent variables and four dependent variables in the study.
Independent variables
The two main independent variables in the experiment were display type and scenario complexity. More specifically, there were two levels of display type. Operators monitored the simulated upset scenarios using either a functional overview display or a schematic overview display.
In addition, there were two levels of scenario complexity. More specifically, out of the four scenarios that were selected for the experiment (1, 2, 3, and 4), two of them were low-complexity scenarios (1 and 3), and the other two were high-complexity scenarios (2 and 4). The level of complexity for the scenarios was determined on the basis of the number of process parameter deviations that occurred. A process deviation was defined as a condition whereby a process parameter changes either from normal to abnormal or from abnormal to alarm condition, or vice versa.
Abnormal and alarm conditions were based on the underlying process simulation dynamics. Any change in process state was included in the total number of process deviations that actually occurred. Therefore, a parameter that went in and out of alarm repeatedly during the course of the scenario was counted as multiple process deviations. The total number of process deviations during each scenario ranged from 15 to 33 for low-complexity scenarios and 146 to 169 for the high-complexity scenarios. It is important to note that the total number of process deviations could not be kept exactly the same for each level of complexity because of constraints in the underlying process simulation dynamics. However, such a manipulation has a good degree of ecological validity.
To control for order effects in the repeated-measures experimental design, we defined two scenario sequences to determine which upset scenario an operator would receive on each of the four evaluation trials. Scenario complexity alternated between low and high in the sequence and was counterbalanced such that half the operators completed a low-complexity scenario on the first trial and half completed a high-complexity scenario on the first trial (Scenario Sequence A: 1, 2, 3, 4; Scenario Sequence B: 4, 3, 2, 1).
The pairing of each display type with the two alternative scenario sequences was also counterbalanced, such that half of the participants assigned to Scenario Sequence A used the schematic overview for the first two scenarios of this sequence and used the functional overview for the last two scenarios of this sequence. The remaining half of the participants assigned to Scenario Sequence A used the first two scenarios with the functional display and finished with the schematic display. The same applied to Sequence B as well.
Dependent variables
There were four dependent variables in addition to posttest (usability) ratings of the different scenarios. The first dependent variable was Level 1 SA (perception). The analysis of display and scenario complexity treatment effects with respect to operator Level 1 SA was measured by the percentage of actual process deviations identified via the operator speak-aloud protocol. From the recorded data, it would have been possible to measure the time taken by operators to detect the changes. However, to maintain focus on the primary research question, we limited our dependent variables to those directly related to SA accuracy. The second dependent variable was Level 2 SA (comprehension). The analysis of display and scenario complexity treatment effects with respect to the operator Level 2 SA was measured by the percentage accuracy of response to the multiple-choice SA probes asked at each pause. Using multiple-choice probes is a recommended and widely accepted method (for a detailed review, see Endsley, 2000). Also, we averaged across pauses to get a scenario-level measure. Third, the number of flag matches as an effect of display and scenario complexity was measured as a dependent variable. Fourth, the effect of the treatment conditions on operators’ subjective workload was measured. Operators were asked to provide subjective workload ratings on a 10-point scale (1 = low workload, 10 = high workload) at each pause. They provided a rating of workload after they responded to the SA queries. When providing rating of workload, operators were instructed to consider the mental and physical effort required, plus their sense of time pressure, stress, and how accurate they felt while reviewing the scenarios. Hence, the process was similar to a NASA Task Load Index workload rating. However, we used a 10-point scale in this study to keep it consistent with another study that was being conducted within the ASM Consortium. We averaged the workload ratings across pauses to get a scenario-level measure of workload.
Procedure
A detailed sequential flow diagram of the procedure is shown in Figure 5. Before starting with the experimental scenarios, the operators completed practice scenarios for each overview display, and they were also given screen captures (printouts) of both overview displays in steady state as a reference sheet. This sheet was available to the operators until the end of the evaluation session.
Flow diagram representing the experimental procedure.
We measured operator SA using two techniques. Level 1 SA (perception) was measured by the number of significant process changes identified with the use of a speak-aloud protocol. This variable was considered a measure of how well operators could perceive significant changes in the process. A significant process change was defined as a condition whereby a parameter changes from normal to abnormal (beyond normal variability), abnormal to alarm condition, a low alarm to a low-low alarm, or a high alarm to a high-high alarm, and vice versa. Operators were instructed to say aloud the name of the process parameter that changed, the equipment area on the display where the parameter was located (e.g., crude feed, vacuum heater), and the type of change that occurred (e.g., normal to abnormal). Any change in process state was included in the total number of significant changes that actually occurred. Therefore, a parameter that went in and out of alarm repeatedly during the course of the scenario was counted as multiple process changes. We scored the number of changes detected from the verbal report recordings for each scenario and divided it by the total changes that occurred during each scenario to calculate the percentage of changes detected.
Level 2 SA (comprehension) was measured by the accuracy of responses to multiple-choice SA probes asked at each pause. To answer every multiple-choice probe correctly, operators had to accurately comprehend multiple sources of information. Hence, the accuracy of a response to the probes was considered as a measure of how well operators were able to accurately comprehend the situation (e.g., Endsley, 1995). An example of a Level 2 SA probe that was included in the study was “Since the beginning of the scenario, controller output of the reboiler flow in the stripper has (a) significantly increased, (b) significantly decreased, (c) stayed constant with no significant deviation from normal state.”
Level 2 SA was assessed twice during each scenario. Operators were informed that a pause would occur at a random time during the middle of the scenario and also at the end of each scenario. During the pause, the experimenter minimized the overview display screen and instructed the operators to respond as accurately as possible to multiple-choice questions designed to gauge their comprehension of the situation. This is similar to the situation awareness global assessment technique (SAGAT), which has been used widely to measure SA (e.g., Endsley, 1995). During each pause, operators were asked eight multiple-choice questions.
No feedback was provided regarding their performance on the overview display monitoring task. However, they did receive visual feedback on their flag-matching task performance because any matched flags were immediately removed from the monitor screen. Operators were instructed to complete as many matches as possible. They were also instructed to pay equal importance to the flag-matching task and the monitoring task as much as possible.
Predictions
This study was designed to test two hypotheses. First, we hypothesized that operator SA as indicated by Level 1 and Level 2 SA would be higher while operators monitored the scenarios on the functional overview display as compared to the schematic overview display. Second, we hypothesized that operator subjective workload would be lower while operators monitored the scenarios on the functional overview display as compared to the schematic overview display.
Results
Data Screening
Each operator was asked to complete a demographic questionnaire. The demographic data were examined for potential outliers in the experience profiles that might justify additional exclusions from the performance analysis. The results are shown in Table 1. We also examined the dependent performance measures for outliers using box plots. One operator reported abnormal vision but was not an outlier for any of the performance measures. None of the other operators was an outlier for any of the measures. Thus, all operator data were included in the subsequent analyses.
Summary of Descriptive Statistics on Demographic Data From the 18 Operators
Level 1 SA
Operators were asked to verbalize all significant process changes that they detected while monitoring the overview displays. Therefore, the percentage of changes detected provided a measure of Level 1 SA. A 2 (display type) × 2 (scenario complexity) repeated-measures ANOVA was conducted on the percentage of changes detected.
Main effects indicated that Level 1 SA was significantly higher when operators monitored scenarios on the functional display when compared with the schematic display, F(1, 17) = 94.45, p < .0001, η2 = .85, and when they monitored the low-complexity scenarios compared with the high-complexity scenarios, F(1, 17) = 81.41, p < .0001, η2 = .83. There was also a significant interaction between display type and complexity, F(1, 17) = 14.77, p < .002, η2 = .95. As represented in Figure 6, analysis of the simple main effects indicated that Level 1 SA was higher when operators monitored the videos on functional displays compared with schematic displays. This was true for both low complexity, t(17) = 6.72, p ≤ .001, and high-complexity scenarios, t(17) = 14.38, p ≤ .001. Means of the statistically significant findings are reported in Table 2.
Level 1 situation awareness as a function of display type and scenario complexity. Error bars indicate +1 standard error of the mean.
Summary of Means and Standard Deviations for Statistically Significant Findings
Note. SA = situation awareness.
Level 2 SA
We measured Level 2 SA using multiple-choice probes administered at each pause. Responses were scored in terms of percentage correct responses and were averaged across pauses for each scenario. The probe response accuracy provided a measure of operators’ Level 2 SA. A 2 (display type) × 2 (scenario complexity) repeated-measures ANOVA was conducted on the percentage accuracy of responses.
As represented in Figure 7, main effects indicated that Level 2 SA was significantly higher when operators monitored the scenarios on the functional display as compared with the schematic display, F(1, 17) = 4.74, p < .05, η2 = .22. In addition, Level 2 SA was significantly higher when the operators monitored the low-complexity scenarios as compared with the high-complexity scenarios, F(1, 17) = 12.26, p < .004, η2 = .42. The interaction was not significant. Means of the statistically significant findings are reported in Table 2.
Level 2 situation awareness as a function of display type and scenario complexity. Error bars indicate +1 standard error of the mean.
Flag-Matching Task Performance
The total number of correct flags matched by each operator was recorded as a measure of flag-matching task performance. A 2 (display type) × 2 (scenario complexity) repeated-measures ANOVA was conducted on the total number of correct flags matched. As represented in Figure 8, main effects indicated that the operators made significantly fewer matches when monitoring the high-complexity scenarios as compared with the low-complexity scenarios, F(1, 17) = 10.92, p < .005, η2 = .39. There was no main effect for display condition. Means of the statistically significant findings are reported in Table 2.
Primary task performance as a function of scenario complexity. Error bars indicate +1 standard error of the mean.
Overall Workload Score
Operators were asked to provide subjective workload ratings on a 10-point scale (1 = low workload, 10 = high workload) at each pause. Workload ratings were averaged across pauses to get an overall workload score for each scenario. A 2 (display type) × 2 (scenario complexity) repeated-measures ANOVA was conducted on the overall workload score. As represented in Figure 9, main effects indicated that workload score was significantly higher when operators monitored the high-complexity scenarios as compared with the low-complexity scenarios, F(1, 17) = 23.94, p < .0001, η2 = .59. There was no main effect of display condition. Means of the statistically significant findings are reported in Table 2.
Overall subjective workload rating as a function of display type and scenario complexity. Error bars indicate +1 standard error of the mean.
Posttest Usability Questionnaires
After the completion of the four display evaluation scenarios, operators were asked to indicate their ratings of each of the displays on an 11-statement usability questionnaire that was a modified version of the Software Usability Scale (Brooke, 1996). Operators were asked to rate the extent to which they agreed with each usability statement on a 7-point scale on which 1 meant strongly agree, 4 indicated neutral, and 7 meant strongly agree. All statements were framed in a positive manner with respect to usability such that a high score indicates effective usability on that item.
Table 3 shows the mean rating for each display on each of the 11 items as well as an average usability score across all 11 statements. Paired-comparisons t tests were conducted on the usability ratings. Results indicated that the average usability score for the functional display was significantly higher than that of the schematic display, t(17) = 2.46, p ≤ .025.
Mean Usability Ratings for Each Display Type on an 11-Item Usability Questionnaire
Discussion
In this study, we tested two hypotheses. First, we hypothesized that operator SA as indicated by Level 1 and Level 2 SA would be higher when participants monitored the scenarios on the functional overview display as compared with the schematic overview display. Second, we hypothesized that operators’ subjective workload would be lower when participants monitored the scenarios on the functional overview display as compared with the schematic overview display. The results supported the first hypothesis pertaining to operator SA but failed to support the second hypothesis pertaining to subjective workload.
Operators were able to detect significantly more changes (Level 1 SA) when monitoring the functional overview display as compared with the schematic overview display. This result applied to scenarios with both levels of complexity. Importantly, the overall percentage accuracy for the schematic overview displays compared with functional overview displays was quite low. Considering that operators who participated in this study were familiar with schematic displays, such low-percentage accuracy with such displays is interesting. This could partially be attributable to the demands created by the whole experimental setup. However, the experimental setup was the same for both types of displays, and hence, the results are generally supportive of the functional overview displays.
Operators were also able to comprehend the status of processes (Level 2 SA) much better when monitoring the functional display as compared with the schematic display. Also, operators’ Level 2 SA was higher when monitoring scenarios of low complexity as compared with those of high complexity. Taken together, such findings clearly indicate the effectiveness of qualitative displays (such as functional displays) in supporting proactive monitoring.
To make the monitoring session more realistic relative to how an overview display might be used in a control room environment, operators in this study were asked to perform a visuospatial flag-matching task. First, the results for primary task performance indicated that there were no differences in total number of flags matched when operators monitored the scenarios on the functional display when compared with the schematic display. However, their SA scores were higher with the functional displays. This finding indicates that although operators were able to maintain equivalent performance levels on the flag matching, their SA was higher when they monitored the scenarios on functional display compared with the schematic display. Second, the number of flags matched was significantly larger when the operators monitored the low-complexity scenarios compared with the high-complexity scenarios. The same pattern was reflected in the SA scores. That is, operators had lower SA with high-complexity scenarios. This finding indicates that regardless of the type of overview display, operators’ overall performance is relatively poor with high-complexity scenarios compared with low-complexity scenarios.
Workload ratings were significantly higher for the high-complexity scenarios compared with the low-complexity scenarios. This finding validated our experimental manipulation of scenario complexity. However, counter to the expectations stated in the study hypothesis, the subjective workload ratings did not significantly differ for the two display conditions. There are two possible explanations for this finding, although it is based on the experimenters’ observation of the test sessions.
First, the experimenters observed that the operators verbalized relatively more while using the functional displays compared with the schematic displays. Such a difference in the amount of verbalization could indicate that operators were expending more effort locating and speaking about the changes in the functional display compared with the schematic display. Also, there was a significant increase in percentage of process changes detected when using the functional display. Because operators in the functional display condition were more aware of the significant process changes, they could have experienced an increase in mental workload while trying to keep track of these changes in short-term memory. Hence, the reduced mental workload to perceive significant changes in the functional display would probably have been offset by the increased workload to keep track of the changes in short-term memory. Second, when more changes are explicitly represented, operators may have been more aware of what they do not know. In other words, they may have had the experience that they are not keeping up with the pace of changes. In such conditions, one might expect the perceived workload ratings to be higher with the functional display. However, given the number of process parameter deviations that were occurring with the high-complexity scenarios, an absence of a difference in workload rating between the two display types might be considered an indication of the effectiveness of the functional display. Although from a practical perspective, these explanations seem accurate, they should be accepted with caution, because the study did not include an objective measure of the amount of verbalization.
Finally, although we did not have any specific predictions regarding operator usability ratings for the two displays, ratings for the functional display were significantly higher than those for the schematic display. This finding also overlapped with higher SA when operators monitored the scenarios on the functional displays as compared with the schematic displays. Taken together, such findings clearly indicate the positive effectiveness of qualitative, functional overview displays in enhancing operator SA and overall performance during proactive monitoring in control room operations.
Implications for Decision Making and Cognitive Engineering
It has been asserted that an integrated model of naturalistic decision making incorporates concepts of situation awareness, recognition-primed decision making, and considerations about levels of expertise (e.g., Greitzer, Podmore, Robinson, & Ey, 2010). Importantly, our research led to the design of a functional overview display that can support such concepts. For example, the results indicated that operators viewing the functional overview display had higher levels of Level 1 and Level 2 SA.
After detection of a process deviation that warrants further investigation, the decision hierarchy that console operators follow is to drill deeper in to the problem. To do so, they navigate from the overview displays to the unit summary display, to the equipment-level display, and finally, to the group and point detail (e.g., Errington et al., 2005). To navigate effectively through this hierarchy, operators should be provided with the right amount of information at each level to quickly make appropriate decisions and follow-on actions. For example, if there is a process value moving to an abnormal state, the functional overview display quickly highlights the right unit and equipment so that operators can immediately drill down to the next detailed level of display. In essence, operators can effectively move directly from perceiving an abnormally increasing process value (e.g., flow) to deciding which display to navigate to. Therefore, the functional overview display effectively supports recognition primed decision making (RPDM) as well, indicating that compared with the traditional schematic overview display, functional overview display better supports the critical components associated with the naturalistic decision-making hierarchy that operators typically adopt.
Our design approach also incorporated critical steps involved in effective cognitive engineering. For example, we worked with an expert operator to understand and define the goals of a console operator. In addition, we functionally organized the crude distillation process to help operators better maintain an accurate “big picture” of the process. On the basis of the task constraints and goals of the operators, we designed the overview display to show the most pertinent information to help operators detect qualitative changes in the process. Therefore, we represented the semantics, user goals, and mental representations on the overview display in such a way so as to reduce the overall cognitive load on operators (e.g., Stary & Peschl, 1998).
The practical implication of our design is that the functional overview display can help in freeing more of the operators’ internal memory resources. The availability of more internal memory resources can be advantageous when operators have to multitask, which is typical in control room environments. It has been suggested that when an external memory aid is available, humans make less use of their internal memory (e.g., Anderson & Douglas, 2001). One example is that in functional overview displays, an abnormal situation is directly indicated by color. In contrast, in a traditional schematic overview display, operators have to make mathematical calculations of process values to gauge whether a situation is moving out of normal range.
In summary, with increasingly automated technologies in control rooms, operators have primarily changed their role to a decision maker (e.g., Yang & Hwang, 2001). This is typically true for any domain in which there is an increase in the use of automation to support decision making and action execution. Therefore, in the development of decision support systems (e.g., overview displays) in any complex, dynamic domain, it is important to use effective cognitive engineering techniques that help to meet at least the following four requirements. First, the decision support system should help to specify and represent the apt semantics of the domain. Second, it should help to address the tasks, constraints, goals, and mental representations of the operator. Third, it should help support the (potentially dynamic) decision-making hierarchy of operators. Finally, by effectively externalizing the cognitive resources, the decision support system should help in enhancing the availability of internal cognitive resources (e.g., Stary & Peschl, 1998).
Limitations and Future Directions
There are limitations in this study. First, the operators were able only to monitor the display without an opportunity for actual intervention. More specifically, when operators detect a significant change in an overview display, regardless of whether it is a functional or schematic display, operators typically navigate through the levels (Level 1 through Level 4) of display for more information. Level 1 is the overview display, and Level 4 provides the most detailed level of information. Importantly, operators in this study were not provided with an option for actual intervention so that we could maintain a level of experimental control such that all operators were exposed to the same process plant conditions. Whether the level of information provided on the overview displays was effective for further navigation through the additional levels (i.e., Levels 2, 3, and 4) was beyond the scope of this study. Second, the flag-matching task was only a simple representation of the cognitively demanding activities that operators typically perform in a control room in addition to monitoring the overview displays. We adopted the flag-matching task to mimic the workload associated with real operator activities while at the same time creating a setup that was measurable. Third, although the overall process information represented in the two types of overview displays was exactly the same, the way the information was represented differed along multiple dimensions, such as the display layout, type of shape, overall organization, location of shapes, and so on. If we had designed experimental conditions to control for each of these dimensions, we would have had to recruit significantly more operators for the study. However, finding operators who are willing to participate in such studies is challenging, and the experiment must be simple and flexible enough to adapt to their schedules. Therefore, we tried to keep the most critical variable the same across both displays, that is, the process information available on both displays.
In summary, there were limitations in this study. Such limitations were primarily attributable to the constraints involved in conducting a controlled experiment in the field with actual operators. However, it is important to note that our experiment was controlled to a large extent, and our results and inferences should have a good degree of external validity. Nevertheless, future studies should try to address the aforementioned limitations.
Conclusions
Recent research indicates that qualitative indicators help operators to detect changes more quickly (e.g., Jessa & Burns, 2000). With this study, we validated the understanding of the effectiveness of qualitative, functional shapes. Operators monitored for process parameter deviations associated with simulated upset scenarios using two alternative overview displays (comprising different shapes) that showed the same process parameters but with different display objects and formats. Taken together, the results indicate that the use of a functional display format and qualitative display shapes, designed for detecting qualitative changes in the state of the process (e.g., normal to abnormal or abnormal to alarm), is more effective for supporting and enhancing operators’ Level 1 and Level 2 SA than the use of a schematic display format that includes the schematic details and quantitative indicators that show process values as numeric indicators.
In conclusion, the results indicate that a functional overview display is a promising way to represent overview information to continuous-process industry control room operators. It is important to note that even though the operators were experienced with traditional schematic formatted displays, they performed much better with the functional overview displays, suggesting that a transition from a traditional display to a functional display may not be a negative one. Future studies should continue to investigate valuable mechanisms for displaying information much more effectively to operators and on mechanisms that transition away from traditional schematic overview displays. Finally, the cognitive engineering approach we adopted in this study can be an effective way to understand the needs of humans functioning within complex environments (e.g., operators, pilots, soldiers, air traffic controllers), more specifically, the interaction requirements, the hierarchy of information needed for their tasks, and types of displays that can help them in maintaining a high level of SA.
Footnotes
Acknowledgements
A subset of the results from the experiment was reported at the 54th annual meeting of the Human Factors and Ergonomics Society and was published in the conference proceedings (Tharanathan, Bullemer, Laberge, Reising, & Mclain, 2010). This study was funded by the Abnormal Situation Management Consortium (ASM), a Honeywell-led research and development consortium. We would like to acknowledge the contribution of operators at the ASM member refineries who volunteered to participate in this study. In addition, we would like to thank George Gabaldon for his assistance in the information requirements definition for the overview displays. Finally, we would like to thank the reviewers for their insightful comments.