Insights From Pupil Size to Mental Workload of Surgical Residents: Feasibility of an Educational Computer-Based Surgical Simulation Environment (ECE) Considering the Hand Condition

Abstract

The advantage of simulation environments is that they present various insights into real situations, where experimental research opportunities are very limited—for example, in endoscopic surgery. These operations require simultaneous use of both hands. For this reason, surgical residents need to develop several motor skills, such as eye-hand coordination and left-right hand coordination. While performing these tasks, the hand condition (dominant, nondominant, both hands) creates different degrees of mental workload, which can be assessed through mental physiological measures—namely, pupil size. Studies show that pupil size grows in direct proportion to mental workload. However, in the literature, there are very limited studies exploring this workload through the pupil sizes of the surgical residents under different hand conditions. Therefore, in this study, we present a computer-based simulation of a surgical task using eye-tracking technology to better understand the influence of the hand condition on the performance of skill-based surgical tasks in a computer-based simulated environment. The results show that under the both-hand condition, the pupil size of the surgical residents is larger than the one under the dominant and nondominant hand conditions. This indicates that when the computer-simulated surgical task is performed with both hands, it is considered more difficult than in the dominant and nondominant hand conditions. In conclusion, this study shows that pupil size measurements are sufficiently feasible to estimate the mental workload of the participants while performing surgical tasks. The results of this study can be used as a guide by instructional system designers of skill-based training programs.

Keywords

eye tracking simulation modeling training mental workload pupil size

Introduction

When compared with traditional methods, endoscopic surgery provides several benefits. It has been reported that people who underwent endoscopic surgery instead of traditional open procedures experienced a faster recovery period and less pain.¹ In these operations, smaller-sized incisions are used unlike in traditional methods. To perform surgical operations, surgeons need to use both hands effectively. For this reason, the surgeons’ motor skills in performing tasks with their dominant, nondominant, and both hands are extremely critical for patients’ safety and the quality of operations. Thus, the training of surgeons needs to address how to gain these critical skills in an appropriate manner.

According to the mental workload theory, there is a correlation between the rate at which knowledge is handled by the human operator and the rate at which decisions are taken.² In the literature, it is stated that mental workload is related to several physiological measures—namely, heart rate, blink frequency and duration, pupil size, electrodermal activity, respiratory frequency, and other variables derived from the electroencephalogram.³ Factors such as hand condition most probably have an effect on the mental workload of surgical residents. Parallel to the mental workload theory, psychophysiological studies broadly use eye tracking to collect response parameters from the eye-movement data to analyze cognitive processes underlying visual behavior.⁴ Additionally, the cognitive load theory describes how the mental effort of learners is influenced by the design of the learning material.^5-8

Studies show that the human pupillary response is satisfactory evidence for the relationship between pupil size and a wide range of important cognitive variables, including mental workload^9-18; hence, there exists a relationship between cognitive load and pupil size.¹⁹ This is supported by other studies that report that pupil size is an important indicator of brain function.^20-23

Present day eye-tracking technology provides various physiological data to allow a better understanding of the consumption of resources through ocular activity, which is closely related to the neural networks in the brain.^9,24 It has also been reported that this technology can be used for high-temporal-resolution tracking of cognitive workload.²⁵ Therefore, current advances in the consistency of the eye-tracking methodology in addition to the increasing accessibility of affordable simulation and modeling technology have widened research prospects in a range of areas and applications.^9,26,27 In addition, pupil size changes have been suggested as a metric for evaluating mental workload while complex visual tasks are performed.⁹ For instance, studies report that changes in pupil size might reflect neuronal activity and cognitive functions throughout some parts of the brain.^17,18,20,28 In another study, pupil size changes have been recommended as a biomarker in early-phase detection of Alzheimer’s.⁹ Additionally, the relationship between pupil size and information processing load in a variety of cognitive tasks is also well proven,^28,29 and pupil size dynamics have been shown to be a reliable measure to investigate the cognitive processes involved in sentence processing and memory functioning.³⁰ Because human mental capacity is a finite resource, the extent of the achievement of a complex task relies on the requirements of the task to be performed.³¹ It is reported that the pupil response pattern is distinguishable depending on different levels of difficulty of the task because the pupil size increases in harder tasks.²⁹ Based on psychomotor evidence regarding the performance of surgeons, studies show that task-specific training curricula can be designed to improve the related skills.³² In particular, eye-tracking applications are being increasingly used in different areas. The accessibility of affordable devices from monitor screens to goggles and computer peripherals has extended this application area in a variety of disciplines, such as medicine, commerce, and education.⁹ Several eye-tracking studies conducted on physiological signs have been used to evaluate the mental workloads of surgeons.³² The results of past research suggest that when surgical tasks become difficult, the mental workload increases, as evident from pupil size changes.¹⁰ The outcomes of earlier studies concerning surgeons also show that in laparoscopic surgery, because of the long tool shaft and the lack of depth information when projecting the surgical scene inside the body onto a 2-dimensional screen, the task requirements are more demanding than in open surgery.²⁹ Also, when the task difficulty level increases in surgical laparoscopic procedures, the surgeon’s peak pupil size also increases.³² Additionally, studies show that multitask performance entails a higher demand compared with a single task.³³ In the domain of endoneurosurgery, surgical procedures performed under the both-hand condition can be considered as a multitask, whereas those undertaken by the dominant or nondominant hand can be regarded as a single task. Xie and Salvendy³³ report that it is critical to assess and predict the workload under different conditions. However, in the literature, there are very limited studies that have been conducted to better understand how the pupil size changes correlate with the level of task difficulty in an eye-hand coordination movement.²⁹ Therefore, in this study, the changes in surgical residents’ physiological signals are examined in relation to the mental workload by considering their pupil size for different hand conditions.

Today, modern software is becoming capable of supporting autonomic behaviors by monitoring the relevant phenomena of the environment and analyzing the collected data to better understand the possible consequences of changes in the environment. Hence, this type of software, called context-aware adaptive software systems (CAASS), aim to adjust themselves in response to the changes in the operating environment.^34,35 However, to create better adaptive software structures, it is necessary to acquire an enhanced comprehension of the user behaviors to achieve the adaptation in a successful manner.

Accordingly, this study aims to better understand the behaviors of surgeons through their eye data to create CAASS tools for surgical education programs. This study is descriptive and was conducted on a computer-simulated surgical task with 23 surgical residents. In detail, this study analyzed the relation between the hand condition (dominant, nondominant, and both hands) and the eye movements of surgical residents while they performed a skill-based surgical task in a computer-simulated environment. It is problematic to provide a real context in which to implement these complex activities for resident surgeons; however, computer-based simulation environments provide very similar context for performing the training tasks.^8,36,37 The results of this study can guide instructional system designers to better address the skill development requirements and, furthermore, can be used as a basis for self-adaptive education systems.³⁸

Sequencing and selecting appropriate learning resources for adaptive learning environments is a critical issue.³⁹ Accordingly, by regularly assessing surgeons’ skill levels and evaluating the difficulty levels of each computer-based simulation scenario through the eye movement events of the trainees, the sequence of these scenarios in the curriculum can be adapted to the user skill levels and behaviors under different hand conditions. This will help create a specific curriculum for each trainee that is adapted dynamically to their skill and knowledge.

Method

To fully grasp the influence of the hand condition (dominant, nondominant, and both hands) on the mental workload of skill-based surgical tasks in a computer-based simulated environment, right-eye pupil sizes, left-eye pupil sizes, and average both-eye pupil sizes of 23 surgical residents were examined.

Participants

The surgical residents performed a computer-based simulated endoscopic surgery task, during which their pupil sizes were recorded. As shown in Table 1, a total of 28 surgical residents from Neurosurgery and Ear-Nose-Throat (ENT) surgery departments voluntarily participated in this study. Because of eye tracker recording failures, 5 participants were excluded from the analysis, and the recordings of the remaining 23 participants were analyzed. Because of the difficulty of accessing surgeons of a specific field willing to volunteer, several previous studies have also been conducted with a limited number of participants: 14 surgeons,⁴⁰ 9 neurosurgeons,⁴¹ and 22 surgeons.⁴² Accordingly, this study can be considered to be a feasibility trial in this field.

Table 1.

Participant Information.

		n	Percentage
Gender	Female	3	13
Gender	Male	20	87
Eye glass	No	17	74
Eye glass	Yes	6	26
Dominant hand	Right	20	87
Dominant hand	Left	3	13

As seen in Table 1, the majority of participants were male (87%) and did not use eyeglasses (74%). Additionally, the majority of the participants were right-hand dominant (87%). The average age of the 23 participants was 28.35 (SD = 5.48) and all surgical residents were at the Neurosurgery or ENT surgery departments of the medical school. Table 2 shows that on average, the participants observed 24.74 (SD = 29.20) and assisted in 14.70 (SD = 22.03) endoscopic surgical operations and performed 6.48 (SD = 13.16) endoscopic surgical operations as surgeons.

Table 2.

Participant Endoscopic Surgery Experience.

	Mean	SD
Age	28.35	5.48
Observed	24.74	29.20
Assisted	14.70	22.03
Performed	6.48	13.16

Procedure

Endoscopic surgery is performed with an endoscope and several long, thin instruments through small incisions. According to the medical requirements, in this study, a simulation scenario of an endoscopic procedure was modeled in an educational computer-based simulation environment. In this scenario, the participants were asked to clear tumor-like objects from a 3D human nose model environment in a fixed period of time. There were 10 objects (10 tasks) located in different places in the model. As seen in Figure 1, the participants, using a haptic device, could move back and forth in this simulated environment. Haptic devices enable interaction between users and systems,⁴³ providing true 3D navigation and force feedback. In this study, the haptic devices gave force feedback when the participants took an object from its location or on collision with any surface. The Geomagic Touch haptic devices were used to perform the tasks in this study.

The pupil sizes of the surgical residents were collected with an eye tracker during the task performed by haptic devices. The Eye Tribe eye tracker with a sampling rate of 60 Hz was used in this study to record the pupil sizes of the participants. The Eye Tribe provides binocular eye-tracking technology with a sleek appearance and a portable structure. In previous studies, it was stated that the pupil size differences can be significantly differentiated by the Eye Tribe eye tracker in various levels of workload trials.⁴⁴ It is also reported that the results prove to be promising for human factors researchers.⁴⁴

The volunteers were seated and centered in front of the monitor at a distance of 70 cm. An instruction describing the procedure was also given to each individual. The pupil size responses of the participants while performing the scenario were recorded by the eye tracker device, which was located below the monitor, as can be seen from Figure 2. The experimental setting was controlled as much as possible for luminance and head position, which may have affected the pupil sizes of the participants. Ambient light and screen brightness were standardized, and the participants were required to continuously look at the screen during the experiment. Shutters and constant ceiling lights ensured the same lighting conditions in the room. Before the surgical simulation tasks began, 9 calibration points were presented. All participants performed the surgical simulation scenario under 3 conditions (dominant hand, nondominant hand, and both hands) consecutively on the same day. There was a fixed time duration (20 s) for completing each of the 10 tasks, but there were no feedback elements on the screen that could lead to stress (such as task score or error rate).

The participants (5 female, 23 male) performed the simulated scenario with the dominant hand (Figure 2A) and the nondominant hand (Figure 2B). To eliminate order effects, half of the participants were asked to start the experiment with their dominant hand, and the other half started the experiment with their nondominant hand. After the completion of the one-handed tasks, the same task was performed with both hands simultaneously (Figure 2C), which was intended to increase the mental workload of the participants. In this case, participants controlled the operational tool with their dominant hand (through a haptic device) and the endoscope with the nondominant hand in parallel. Because of eye tracker recording failure, 5 participants were excluded, and a total of 690 trials were analyzed (23 participants, each performing 10 tasks in 3 different hand conditions). The Consolidated Standards of Reporting Trial (CONSORT) flow diagram is presented in Figure 3.

Results

The analysis of all the data was performed using SPSS for Windows software package (version 23; IBM Corporation, New York, NY) at a 95% confidence level. Because the normality condition was violated, in this study, the Friedman nonparametric test technique was used as an alternative to bidirectional variance analysis to determine the differences between 2 major masses.⁴⁵

A nonparametric Friedman test of differences between the repeated measures was conducted for the hand condition effect on the left-eye, right-eye, and the average of the left- and right-eye pupil sizes, and the rendered χ² values were 11.57, 16.44, and 11.57, respectively. The descriptive statistics are given in Table 3, and as shown in Table 4, the results were significant (P < .01). The P values are given for orientation because this study was considered as a feasibility trial in this field.⁴⁶

Table 3.

Descriptive Statistics.

	Left Eye			Right Eye			Both Eyes
	n	Mean	SD	n	Mean	SD	n	Mean	SD
Dominant hand	23	18.37	3.23	23	18.57	2.74	23	18.47	2.87
Nondominant hand	23	19.27	2.13	23	19.18	2.10	23	19.23	2.02
Both hands	23	19.90	2.77	23	20.34	2.61	23	20.12	2.63

Table 4.

Friedman Test Statistics for Pupil Size.

	Left Eye	Right Eye	Both Eyes
n	23	23	23
χ²	11.57	16.44	11.57
df	2	2	2
P	.003	.000	.003
	Mean Rank
Dominant hand	1.52	1.48	1.48
Nondominant hand	1.96	1.87	2.04
Both hands	2.52	2.65	2.48

The results indicate that when the task was performed under different hand conditions, there was a significant difference in the left, right, and the average of both eye pupil sizes. As seen in Figures 4 and 5, under the both-hand condition, the left-eye and the right-eye pupil sizes as well as the average pupil sizes of both eyes were larger when compared with the dominant hand and nondominant hand conditions. Additionally, the results showed a slight difference between the left- and right-eye pupil sizes under all conditions. Earlier studies also report such differences between the right- and left-eye pupil sizes.⁴⁷ However, the main reason for this difference has not yet been clarified; thus, further research is necessary.

Discussions and Conclusion

Surgical residents performed the same tasks under the dominant hand, nondominant hand, and finally, both-hand conditions. In this study, the Friedman statistical analysis method was used to better understand the effect of different hand conditions on pupil sizes. The results of this study show that while performing computer-simulated surgical tasks, the hand condition (dominant, nondominant, and both hands) significantly affects the pupil sizes (left-eye, right-eye, and average of both eyes) of the participants. The main results of this study are highlighted below:

Under different hand conditions, there is a significant difference in the pupil sizes of the left eye, right eye, and average of both eyes

Under the both-hands condition, the left-eye and right-eye pupil sizes as well as the average pupil sizes of both eyes were larger when compared with the dominant and nondominant hand conditions.

Hence, under the both-hand condition, the pupil sizes of the surgical residents become larger than in other conditions (dominant and nondominant hands). Earlier studies report that pupil sizes grow in direct proportion to the mental workload.^15,19,32 Also, it is noted that mental workload is generally defined as the ratio between the capacity of a person and the task demands, and that mental workload is high when the task demands exceed capacity.⁴⁸ Accordingly, the practical implementation of mental workload theory in the field of endoneurosurgery shows that the mental workload of surgeons significantly increases under the both-hand condition compared with the dominant or nondominant hand conditions. Mental workload has been described as a subjective perception of the association between mental processing ability and the amount of processing required to perform a task.⁴⁸ Knowledge of a person’s mental workload is useful in assessing and designing systems or working conditions, such as monitoring and helping people at work.⁴⁹ Hence, under the both-hand condition, the simulated surgical tasks are regarded as harder than under other hand conditions, indicating that under the both-hand condition, the mental workload increases.

According to the results, it is concluded that surgical residents have more difficulty when using both hands during a procedure. Because both-handed tasks require a multitasking process, this may cause a higher mental workload.³³ Because in real operations it is necessary to use both hands in a coordinated fashion, surgical residents’ skills under the both-hand condition need to be assessed and improved systematically during surgical education programs. Because the pupil sizes appear to provide an objective assessment of the mental workload among surgical residents, this information can be used to better evaluate and guide skill improvements during simulation-based surgical training programs. This finding is critical to guide the instructional system designers to provide skill-based training curricula and appropriate CAASS tools based on the surgeons’ both-hand skills until an acceptable mental workload is assessed through their eye data. In other words, by continuously assessing their skill improvements through their eye data, adaptive tools to the trainees’ individual performances can be developed. Hence, this study provides an approach that can better assess the mental workload under different conditions (single task: dominant and nondominant hand conditions; multitask: both-hand condition), which is reported as a requirement to achieve an improved comprehension of the mental workload.³³

The field of endoneurosurgery education programs has several issues, with the main problem being the skill-based training opportunities. Because the training and skill development needs to be provided in the operating theater, there are several drawbacks of these education programs, such as the ethical considerations from the patients’ perspective, limited time and cases, and the risk of patient safety. According to the results of this study, the pupil size measures are shown to be as capable of providing instructional system design alternatives for individualized training programs. This study encourages instructional system developers of simulation-based surgical education programs to utilize pupil size data to better guide and assess the trainees’ skill improvements. Currently, there are few alternative training opportunities for surgical training programs. Because skill improvement is highly critical for these programs, individual skill-based training opportunities are required. There are examples of computer-based simulations for supporting surgical training programs, but there are very limited examples of curriculum integrated models. Additionally, there is no instructional model of CAASS for surgical education programs, especially in the endoneurosurgery education programs.

The main aim of the study was to provide insight into the influence of hand conditions on the mental workload of surgical residents to create improved CAASS tools for surgical education programs. For this purpose, the participants’ pupil sizes were recorded and analyzed while performing computer-simulated surgical tasks.

In conclusion, we believe that because of its very nature, the CAASS approach may provide several benefits for the endoneurosurgery education programs. However, the process of creating CAASS for the field of endoneurosurgery education programs is very complex; therefore, the findings of this study aim to help future studies build improved CAASS for endoneurosurgery education programs and to better integrate these systems into the current educational programs.

Limitations of the Study

This study was conducted on surgical residents, the majority of whom used their right hand as the dominant hand. Because there were only 3 participants whose dominant hand was their left hand, a comparison could not be conducted in this study. However, in future attempts, the dominant hand can also be evaluated as a factor. Also, because the tasks under the both-hand condition were performed after the tasks concerning the conditions of dominant and nondominant hands, there might have been a training effect, which can be considered as fatigue increasing the fluctuations of the pupil size and possibly making it larger. Accordingly, in future studies, this training effect could be controlled.

Footnotes

Acknowledgements

This study is conducted for improving the scenario designs of the educational materials that are developed for endoneurosurgery education project (ECE: Tübitak 1001, Project No: 112K287) purposes. The authors would like to acknowledge the support of the TÜBİTAK 1001 program for realizing this research. The researchers would also like to thank the ECE project team and the Hacettepe University Medical School for their valuable support throughout the research.

Author Contributions

Study concept and design: Nergiz Ercil Cagiltay, Gonca Gokce Menekse Dalveren, Erol Ozcelik, Hakan Maras

Acquisition of data: Nergiz Ercil Cagiltay, Gonca Gokce Menekse Dalveren, Erol Ozcelik, Hakan Maras

Analysis and interpretation: Nergiz Ercil Cagiltay, Gonca Gokce Menekse Dalveren, Erol Ozcelik, Hakan Maras

Study supervision: Nergiz Ercil Cagiltay, Gonca Gokce Menekse Dalveren, Erol Ozcelik, Hakan Maras

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) received no financial support for the research, authorship, and/or publication of this article.

ORCID iD

Gonca Gokce Menekse Dalveren .

References

Feng

Rozenblit

Hamilton

AJ.

A hybrid view in a laparoscopic surgery training system. Paper presented at: ECBS’07 14th Annual IEEE International Conference and Workshops on the Engineering of Computer-Based Systems–TOC; March 26-29, 2007; Tucson, AZ.

Moray

Mental Workload: Its Theory and Measurement. Berlin, Germany: Springer Science & Business Media; 2013.

Hogervorst

Brouwer

van Erp

JB.

Combining and comparing EEG, peripheral physiology and eye-related measures for the assessment of mental workload. Front Neurosci. 2014;8:322.

Berger

Winkels

Lischke

Höppner

GazeAlyze: a MATLAB toolbox for the analysis of eye movement data. Behav Res Methods. 2012;44:404-419.

Mayer

Moreno

Nine ways to reduce cognitive load in multimedia learning. Educ Psychol. 2003;38:43-52.

Sweller

Cognitive load during problem solving: effects on learning. Cogn Sci. 1988;12:257-285.

Sweller

Cognitive load theory, learning difficulty, and instructional design. Learn Instr. 1994;4:295-312.

Darabi

Nelson

Meeker

Liang

Boulware

Effect of worked examples on mental model progression in a computer-based simulation learning environment. J Comput High Educ. 2010;22:135-147.

Andreu-Perez

Solnais

Sriskandarajah

EALab (Eye Activity Lab): a MATLAB toolbox for variable extraction, multivariate analysis and classification of eye-movement data. Neuroinformatics. 2016;14:51-67.

10.

Menekse

Dalveren GG

Cagiltay

Ozcelik

Maras

. Simulation-based environments for surgical practice. Paper presented at: 4th International Conference on Control, Decision and Information Technologies; April 5-7, 2017; Barcelona, Spain.

11.

Bradshaw

Pupil size as a measure of arousal during information processing. Nature. 1967;216:515-516.

12.

Simpson

Effects of a task-relevant response on pupil size. Psychophysiology. 1969;6:115-121.

13.

Kahneman

Beatty

Pollack

Perceptual deficit during a mental task. Science. 1967;157:218-219.

14.

Kahneman

Attention and Effort. Englewood Cliffs, NJ: Prentice Hall; 1973.

15.

Hess

Polt

JM.

Pupil size in relation to mental activity during simple problem-solving. Science. 1964;143:1190-1192.

16.

Klingner

Tversky

Hanrahan

Effects of visual and verbal presentation on cognitive load in vigilance, memory, and arithmetic tasks. Psychophysiology. 2011;48:323-332.

17.

Laeng

Ørbo

Holmlund

Miozzo

Pupillary stroop effects. Cogn Process. 2011;12:13-21.

18.

Chen

Liu

Zhang

Visualized awareness support for collaborative software development on mobile devices. Int J Softw Eng Knowl Eng. 2015;25:253-275.

19.

Beatty

Lucero-Wagoner

. The pupillary system. In: Cacioppo

Tassinary

Berntson

eds. Handbook of Psychophysiology. Cambridge, England: Cambridge University Press; 2000:142-162.

20.

Joshi

Kalwani

Gold

JI.

Relationships between pupil diameter and neuronal activity in the locus coeruleus, colliculi, and cingulate cortex. Neuron. 2016;89:221-234.

21.

Murphy

O’Connell

O’Sullivan

Robertson

Balsters

JH.

Pupil diameter covaries with BOLD activity in human locus coeruleus. Hum Brain Mapp. 2014;35:4140-4154.

22.

Rajkowski

Kubiak

Aston-Jones

Correlations between locus coeruleus (LC) neural activity, pupil diameter and behavior in monkey support a role of LC in attention. Soc Neurosci Abstr. 1993;19:974.

23.

Varazzani

San-Galli

Gilardeau

Bouret

Noradrenaline and dopamine neurons in the reward/effort trade-off: a direct electrophysiological comparison in behaving monkeys. J Neurosci. 2015;35:7866-7877.

24.

Jarodzka

Holmqvist

Gruber

Eye tracking in educational science: theoretical frameworks and research agendas. J Eye Mov Res. 2017;10:1-18. doi:10.16910/jemr.10.1.3

25.

Wierda

van Rijn

Taatgen

Martens

Pupil dilation deconvolution reveals the dynamics of attention at high temporal resolution. Proc Natl Acad Sci U S A. 2012;109:8456-8460.

26.

Nyström

Hooge

Andersson

Pupil size influences the eye-tracker signal during saccades. Vision Res. 2016;121:95-103.

27.

Menekse

Dalveren GG

Cagiltay

NE.

Insights from surgeons’ eye-movement data in a virtual simulation surgical training environment: effect of experience level and hand conditions. Behav Inform Technol. 2018;37:517-537.

28.

Zekveld

Heslenfeld

Johnsrude

Versfeld

Kramer

SE.

The eye as a window to the listening brain: neural correlates of pupil size as a measure of cognitive listening load. Neuroimage. 2014;101:76-86.

29.

Jiang

Atkins

Tien

Bednarik

Zheng

Pupil responses during discrete goal-directed movements. Paper presented at: 32nd Annual ACM Conference on Human Factors in Computing Systems; April 26 to May 1, 2014; Ontario, Canada.

30.

Fernández

Biondi

Castro

Agamenonni

Pupil size behavior during online processing of sentences. J Integr Neurosci. 2016;15:485-496.

31.

Cassenti

Kelley

. Towards the shape of mental workload. https://pdfs.semanticscholar.org/a73f/426fd6b147c0515162979befc8a6efebc064.pdf. Accessed August 28, 2018.

32.

Zheng

Jiang

Atkins

MS.

Detection of changes in surgical difficulty evidence from pupil responses. Surg Innov. 2015;22:629-635.

33.

Xie

Salvendy

Review and reappraisal of modelling and predicting mental workload in single- and multi-task environments. Work Stress. 2000;14:74-99.

34.

Jiang

Cai

Huang

Yau

SS.

An intelligent control architecture for adaptive service-based software systems. Int J Softw Eng Knowl Eng. 2009;19:653-678.

35.

Chen

CS.

A proposed architecture for self-adaptive expert systems. Int J Softw Eng Knowl Eng. 2009;19:213-248.

36.

Cagiltay

Berker

. Technology-enhanced surgical education: attitudes and perceptions of the endoscopic surgery community in Turkey [published online March 17, 2018]. BMJ. doi:10.1136/bmjstel-2017-000238

37.

Cagiltay

Ozcelik

Sengul

Berker

Construct and face validity of the educational computer-based environment (ECE) assessment scenarios for basic endoneurosurgery skills. Surg Endosc. 2017;31:4485-4495.

38.

Shehadeh

Felfernig

Stettinger

Jeran

Reiterer

Automated learning content generation from knowledge bases in the STUDYBATTLES environment. Int J Softw Eng Knowl Eng. 2017;27:1387-1408.

39.

Karampiperis

Sampson

Adaptive learning resources sequencing in educational hypermedia systems. J Educ Technol Soc. 2005;8:128-147.

40.

McGrath

Vine

Brewer

Defriend

Masters

Psychomotor control in a virtual laparoscopic surgery training environment: gaze control parameters differentiate novices from experts. Surg Endosc. 2010;24:2458-2464.

41.

Eivazi

Hafez

Fuhl

et al . Optimal eye movement strategies: a comparison of neurosurgeons gaze patterns when using a surgical microscope. Acta Neurochir (Vienna). 2017;159:959-966.

42.

Cope

Mavroveli

Bezemer

Hanna

Kneebone

Making meaning from sensory cues: a qualitative investigation of postgraduate learning in the operating room. Acad Med. 2015;90:1125-1131.

43.

Ucar

Ustunel

Civelek

Umut

Effects of using a force feedback haptic augmented simulation on the attitudes of the gifted students towards studying chemical bonds in virtual reality environment. Behav Inf Technol. 2017;36:540-547.

44.

Coyne

Sibley

Investigating the use of two low cost eye tracking systems for detecting pupillary response to changes in mental workload. Proc Hum Factors Ergon Soc Annu Meet. 2016;60:37-41.

45.

McCrum-Gardner

Which is the correct statistical test to use?

Br J Oral Maxillofac Surg. 2008;46:38-41.

46.

Bowen

Kreuter

Spring

et al . How we design feasibility studies. Am J Prev Med. 2009;36:452-457.

47.

Marcos

Burns

SA.

On the symmetry between eyes of wavefront aberration and cone directionality. Vision Res. 2000;40:2437-2447.

48.

Strang

Best

Funke

GJ.

Heart rate correlates of mental workload in a large-scale air-combat simulation training exercise. Proc Hum Factors Ergon Soc Annu Meet. 2014;58:2325-2329.

49.

Brouwer

Hogervorst

Van Erp

Heffelaar

Zimmerman

Oostenveld

Estimating workload using EEG spectral power and ERPs in the n-back task. J Neural Eng. 2012;9:045008.