The Effects of Level of Automation on the Out-of-the-Loop Unfamiliarity in a Complex Dynamic Fault-Management Task during Simulated Spaceflight Operations

Abstract

Out-of-the-loop unfamiliarity (OOTLUF) and information sampling strategies were examined after operator use of intelligent fault management (FM) support in interaction with a generic, autonomous, atmospheric control system. A simulated model-based reasoning agent provided fault diagnosis and recovery at three different levels of automation (LOA). In about 10% of the experimental trials FM support was withdrawn simulating a catastrophic failure. This allowed investigating OOTLUF potentially developed during reliable FM automation and its variation as a function of LOA. Dependent measures were the percentage of time the system was out-of-target, time elapsed until a repair was initiated, alarm reaction time, accuracy of manual tank level recordings, information sampling behavior, and subjective ratings of workload, tension, fatigue, self-confidence and trust in automation. It was found that automated FM support improved overall system performance and reduced subjective operator workload. OOTLUF occurred in terms of longer times until repair initiation only at medium LOA and not as expected at high LOA. A differential effect of LOA on information sampling strategies was observed suggesting that offloading operators from recovery implementation at high LOA during reliable automation enabled engagement in fault assessment activities, thereby maintaining situation awareness. It is concluded that for avoiding OOTLUF in complex, dynamic tasks, involving intelligent FM support, the important factor is not simply the LOA, but the ability of a particular LOA to support the human operator's information sampling and higher-level reasoning activities.

References

Billings

C.S.

(1997). Aviation automation: the search for a human-centered approach. Hillsdale, NJ: Erlbaum.

Caldwell

B.S.

(2000). Information and communication technology needs for distributed communication and coordination during expedition-class spaceflight. Aviation, Space, and Environmental Medicine, 71, Supplement, A6–A10.

Christoffersen

Woods

D.D.

(in press). How to make automated systems team players. In Salas

(Ed.), Advances in Human Performance and Cognitive Engineering Research, Volume 2. JAI Press/Elsevier.

Dorais

G.A.

Bonasso

R.P.

Kortenkamp

Pell

Schreckenghost

(1998). Adjustable autonomy for human-centered autonomous system on Mars. 1^st International Mars society Convention Proceedings..

Endsley

M.R.

Kiris

E.O.

(1995). The out-of-the-loop performance problem and level of control in automation Human Factors, 37, 381–394.

Endsley

M.R.

Kaber

D.B.

(1999). Level of automation effects on performance, situation awareness and workload in a dynamic control task Ergonomics, 42, 462–492.

Hart

S.G.

Staveland

L.E.

(1988). Development of NASA-TLX (Task Load Index): Results of empirical and theoretical research In Hancock

P.A.

Meshkati

(Eds.), Human mental workload. (pp. 143–154). New York: Plenum.

Hockey

G.R.J.

Wastell

Sauer

(1998). Effects of sleep deprivation and user interface on complex performance: A multilevel analysis of compensatory control Human Factors, 40, 233–253.

Lee

Moray

(1992). Trust, control strategies and allocation of function in human-machine systems Ergonomics, 35, 1243–1270.

10.

Parasuraman

Sheridan

T.B.

Wickens

CD.

(2000). A model of types and levels of human interaction with automation IEEE Transactions on Systems, Man, and Cybernetics - Part A: Systems and Humans, 30, 286–297.

11.

Sauer

Hockey

G.R.J.

Wastell

(1999a). Maintenance of complex performance during a 135-day spaceflight simulation Aviation, Space, and Environmental Medicine, 70, 236–244.

12.

Sauer

Hockey

G.R.J.

Wastell

(1999b). Performance evaluation in analogue environments: Adaptation during an 8-month Antarctic over-wintering expedition Aviation, Space, and Environmental Medicine, 70, 230–235.

13.

Sauer

Hockey

G.R.J.

Wastell

(2000). The effects of training on short- and long-term skill retention in a complex multiple-task environment Ergonomics, 43, 2043–2064.

14.

Wickens

C.D.

(1992). Engineering psychology and human performance (2nd ed.). New York: Harper Collins.

15.

Wickens

C.D.

Kessel

(1981). Failure detection in dynamic systems In Rasmussen

Rouse

(Eds.) Human Detection and Diagnosis of System Failures (pp. 155–169). New York: Plenum Press.

16.

Wiener

E.L.

Curry

R.E.

(1980). Flight deck automation: Promises and problems Ergonomics, 23, 995–1011.