Aviation Human-in-the-Loop Simulation: Best Practices for Subjective Performance Measurement

Abstract

Subjective performance measurements are a useful tool for researchers and instructors to evaluate tasks that are difficult to quantify with objective data pulled from a simulator or the physiological data of pilots. Subjective performance measurements are non-intrusive measures typically conducted by human raters. Some recommendations for utilizing subjective measures include reducing the workload of the human raters, concealing the aim of the experiment from participants, utilizing multiple raters to evaluate each participant, providing raters with proper training, developing error-resistant rater forms and, including both subjective and objective measures when possible.

References

Aamodt

M. G.

(2007). Evaluating employee performance. In Kim

Moneypenny

(Eds.), Industrial/organizational psychology: An applied approach (5th ed., pp. 244–246). Thomson-Belmont Wadsworth.

Atkinson

Tindall

Sheehy

Bailey

(2016). Answering the call for analytics within the maritime patrol community. Interservice/Industry Training, Simulation, and Education Conference (I/ITSEC) Proceedings.

Atkinson

Tindall

M.J.

Killilea

J.P.

Anania

E.C.

(2018, July) Advancing performance assessment for aviation training. Proceedings of the AHFE 2018 International Conference on Human Factors in Training, Education, and Learning Sciences, Orlando, FL, United States.

Barrouillet

Portrat

Vergauwe

Diependaele

Camos

(2011). Further evidence for temporal decay in working memory: Reply to Lewandowsky and Oberauer (2009). Journal of Experimental Psychology: Learning, Memory, and Cognition, 37(5), 1302–1317.

Bernardin

H. J.

Buckley

R. M.

(1981). Strategies in rater training. Academy of Management Review, 6, 205–212.

Bretz

R. D.

Milkovich

G. T.

Read

(1992). The current state of performance appraisal research and practice: Concerns, directions, and implications. Journal of Management, 18(2), 321–352.

Gawron

V. J.

(2019). Human Performance. Human performance and situation awareness measures (3rd ed., pp. 13). CRC Press. https://doiorg.ezproxy.libproxy.db.erau.edu/10.1201/9 780429019562

Hall

Brannick

M. T.

(2009). Performance assessment in simulation. Human factors in simulation and training, 149–168.

Hebbar

P. A.

Pashilkar

A. A.

(2017). Pilot performance evaluation of simulated flight approach and landing manoeuvres using quantitative assessment tools. Sādhanā, 42(3), 405–415.

10.

Kahneman

(1973). Attention and effort (p. 246). Englewood Cliffs, NJ: Prentice-Hall.

11.

King

(2020). An aviation weather preflight decision support tool to improve GA pilots preflight and inflight performance. [Doctoral dissertation, Embry-Riddle Aeronautical University]. ERAU Campus Repository. https://commons.erau.edu/edt/538/

12.

Landman

Groen

E. L.

Van Paassen

M. M.

Bronkhorst

A. W.

Mulder

(2017). The influence of surprise on upset recovery performance in airline pilots. The International Journal of Aerospace Psychology, 27(1-2), 2–14.

13.

Marlow

S. L.

Lacerenza

C. N.

Paoletti

Burke

C. S.

Salas

(2018). Does team communication represent a one-size-fits-all approach?: A meta-analysis of team communication and performance. Organizational Behavior and Human Decision Processes, 144, 145–170. https://doi.org/10.1016/j.obhdp.2017.08.001

14.

McCambridge

Witton

Elbourne

D. R.

(2014). Systematic review of the Hawthorne effect: New concepts are needed to study research participation effects. Journal of clinical epidemiology, 67(3), 267–277.

15.

Müller

Giesa

H. G.

(2002). Effects of airborne data link communication on demands, workload and situation awareness. Cognition, Technology & Work, 4(4), 211–228.

16.

Roch

S. G.

Woehr

D. J.

Mishra

Kieszczynska

(2012). Rater training revisited: An updated meta-analytic review of frame-of-reference training. Journal of Occupational and Organizational Psychology, 85(2), 370–395. https://doi.org/10.1111/j.20448325.2011.02045.x

17.

Ryu

Myung

(2005). Evaluation of mental workload with a combined measure based on physiological indices during a dual task of tracking and mental arithmetic. International Journal of Industrial Ergonomics, 35(11), 991–1009. https://doi.org/10.1016/j.ergon.2005.04.005

18.

Taylor

J. L.

Kennedy

Noda

Yesavage

J. A.

(2007). Pilot age and expertise predict flight simulator performance: A 3-year longitudinal study. Neurology, 68(9), 648–654.