A Cross-Domain Approach to Designing an Unobtrusive System to Assess Human State and Predict Upcoming Performance Deficits

Abstract

Many work environments are fraught with highly variable demands on cognitive workload, fluctuating between periods of high operational demand to the point of cognitive overload, to long periods of low workload bordering on boredom. When cognitive workload is not in an optimal range at either end of the spectrum, it can be detrimental to situational awareness and operational readiness, resulting in impaired cognitive functioning (Yerkes and Dodson, 1908). An unobtrusive system to assess the state of the human operator (e.g., stress, cognitive workload) and predict upcoming performance deficits could warn operators when steps should be taken to augment cognitive readiness. This system would also be useful during testing and evaluation (T&E) when new tools and systems are being evaluated for operational use. T&E researchers could accurately evaluate the cognitive and physical demands of these new tools and systems, and the effects they will have on task performance and accuracy. In this paper, we describe an approach to designing such a system that is applicable across environments. First, a suite of sensors is used to perform real-time synchronous data collection in a robust and unobtrusive fashion, and provide a holistic assessment of operators. Second, the best combination of indicators of operator state is extracted, fused, and interpreted. Third, performance deficits are comprehensively predicted, optimizing the likelihood of mission success. Finally, the data are displayed in such a way that supports the information requirements of any user. The approach described here is one we have successfully used in several projects, including modeling cognitive workload in the context of high-tempo, physically demanding environments, and modeling individual and team workload, stress, engagement, and performance while working together on a computerized task. We believe this approach is widely applicable and useful across domains to dramatically improve the mission readiness of human operators, and will improve the design and development of tools available to assist the operator in carrying out mission objectives. A system designed using this approach could enable crew to be aware of impending deficits to aid in augmenting mission performance, and will enable more effective T&E by measuring workload in response to new tools and systems while they are being designed and developed, rather than once they are deployed.

References

Bengio

(2009). Learning deep architectures for AI. Foundations and Trends-® in Machine Learning, 2, 1-127.

Berka

Levendowski

D. J.

Lumicao

M. N.

Yau

Davis

Zivkovic

V. T.

Olmstead

R. E.

Tremoulet

P. D.

Craven

P. L.

(2007). EEG correlates of task engagement and mental workload in vigilance, learning, and memory tasks. Aviation, space, and environmental medicine, 78, B231-B244.

Bock

(1998). Problems of sensorimotor coordination in weightlessness. Brain research reviews, 28, 155-160.

Bock

Fowler

Comfort

(2001). Human sensorimotor coordination during spaceflight: an analysis of pointing and tracking responses during the “Neurolab” Space Shuttle mission. Aviation, space, and environmental medicine, 72, 877-883.

Bonner

M. A.

Wilson

G. F.

(2002). Heart rate measures of flight test and evaluation. The International journal of aviation psychology, 12, 63-77.

Bracken

B. K.

Guarino

Dorin

Romero

DiZio

Lackner

Pfautz

(2014). Interpreting Cognitive States from Biometric Sensors to Predict Performance Degradation. Body Sensor Networks (BSN, 2014 IEEE International Conference on, Submitted.

Bracken

B. K.

Palmon

Kellogg

Elkin-Frankston

(2015). Cognitive Assessment and Prediction to Promote Individualized Capability Augmentation and Reduce Decrement (CAPT PICARD). NASA.

Cao

Chintamani

K. K.

Pandya

A. K.

Ellis

R. D.

(2009). NASA TLX: Software for assessing subjective mental workload. Behavior research methods, 41, 113-117.

Deng

(2014). A tutorial survey of architectures, algorithms, and applications for deep learning. APSIPA Transactions on Signal and Information Processing, 3, e2.

10.

D’Mello

Picard

Graesser

. (2007). Toward an Affect-Sensitive AutoTutor. IEEE Intelligent Systems, Special issue on Int.

11.

Eva

K. W.

Cunnington

J. P.

Reiter

H. I.

Keane

D. R.

Norman

G. R.

(2004). How can I know what I don’t know? Poor self assessment in a well-defined domain. Advances in Health Sciences Education, 9, 211-224.

12.

Frank

G. R.

(2007). Monitoring seated postural responses to assess cognitive state.

13.

French

Bisson

R. U.

Neville

K. J.

Mitcha

Storm

W. F.

(1994). Crew fatigue during simulated, long duration B-1B bomber missions. DTIC Document.

14.

Gevins

A. S.

Durousseau

Libove

(1990). Dry electrode brain wave recording system.

15.

Hart

S. G.

(2006). NASA-task load index (NASA-TLX); 20 years later. In Proceedings of the Human Factors and Ergonomics Society Annual Meeting, 50, 904-908.

16.

Hart

S. G.

Staveland

L. E.

(1988). Development of NASA-TLX (Task Load Index): Results of Empirical and Theoretical Research. In Human Mental Workload, 139-183.

17.

Karavidas

M. K.

Lehrer

P. M.

S. E.

Vaschillo

Cheng

(2010). The effects of workload on respiratory variables in simulated flight: a preliminary study. Biological psychology, 84, 157-160.

18.

Kerick

S. E.

Oie

K. S.

McDowell

(2009). Assessment of EEG signal quality in motion environments. DTIC Document.

19.

Kirchner

W. K.

(1958). Age differences in short-term retention of rapidly changing information. Journal of experimental psychology, 55, 352.

20.

Lackner

DiZio

(2011). Evaluating the Influence of Ship-Like Motion on Fatigue Models of Human Performance. Office of Naval Research.

21.

Lackner

J. R.

DiZio

(2005). Motor control and learning in altered dynamic environments. Current opinion in neurobiology, 15, 653-659.

22.

Lackner

J. R.

DiZio

(2006). Space motion sickness. Experimental brain research, 175, 377-399.

23.

Leslie

K. R.

Stickgold

Dizio

Lackner

J. R.

Hobson

J. A.

(1997). The effect of optokinetic stimulation on daytime sleepiness. Archives Italiennes de Biologie, 135, 219-228.

24.

Marques

A. H.

Silverman

M. N.

Sternberg

E. M.

(2010). Evaluation of stress systems by applying noninvasive methodologies: measurements of neuroimmune biomarkers in the sweat, heart rate variability and salivary cortisol. Neuroimmunomodulation, 17, 205-208.

25.

Mathan

Whitlow

Dorneich

Ververs

Davis

(2007). Neurophysiological estimation of interruptibility: Demonstrating feasibility in a field context. In In Proceedings of the 4th International Conference of the Augmented Cognition Society.

26.

Mehler

Reimer

Coughlin

J. F.

(2012). Sensitivity of Physiological Measures for Detecting Systematic Variations in Cognitive Demand From a Working Memory Task An On-Road Study Across Three Age Groups. Human Factors: The Journal of the Human Factors and Ergonomics Society, 54, 396-412.

27.

Mota

Picard

R. W.

(2003). Automated posture analysis for detecting learner’s interest level. In Computer Vision and Pattern Recognition Workshop, 2003. CVPRW’03. Conference on, 5, 49-49.

28.

Murai

Okazaki

Hayashi

(2004). Measurement for mental workload of bridge team on leaving/entering port. In Position Location and Navigation Symposium, 2004. PLANS 2004, 746-751.

29.

Nalishiti

Reucher

Vejvoda

Zulley

(1993). Sleep and circadian rhythm during a short space mission. The clinical investigator, 71, 718-724.

30.

Pearl

Russell

(2000). Bayesian Networks.

31.

Pfautz

Cox

Koelle

Catto

Campolongo

Roth

(2007). User-Centered Methods for Rapid Creation and Validation of Bayesian Networks. In 5th Bayesian Applications Workshop at Uncertainty in Artificial Intelligence (UAI ’07).

32.

Poon

Domingos

(2011). Sum-Product Networks: A New Deep Architecture. In Uncertainty in Artificial Intelligence.

33.

Reynolds

Picard

R. W.

(2001). The Bayes Point Machine for computer-user frustration detection via pressuremouse. In Proceedings of the 2001 workshop on Perceptive user interfaces, 1-5.

34.

Russell

Koren

Rieder

Van Uum

S. H.

(2014). The detection of cortisol in human sweat: implications for measurement of cortisol in hair. Therapeutic drug monitoring, 36, 30-34.

35.

Sackett

D. L.

(1979). Bias in analytic research. Journal of chronic diseases, 32, 51-63.

36.

Schmidhuber

(2014). Deep Learning in Neural Networks: An Overview. arXiv preprint arXiv:1404.7828.

37.

Yerkes

R. M.

Dodson

J. D.

(1908). The Relation of Strength of Stimulus to Rapidity of Habit-Formation. Journal of Comparative Neurology and Psychology, 18, 459-482.

38.

Zeng

Pantic

Roisman

G. I.

Huang

T. S.

(2009). A survey of affect recognition methods: Audio, visual, and spontaneous expressions. Pattern Analysis and Machine Intelligence, IEEE Transactions on, 31, 39-58.