Evaluation of Augmented Reality Feedback in Surgical Training Environment

Abstract

Providing computer-based laparoscopic surgical training has several advantages that enhance the training process. Self-evaluation and real-time performance feedback are 2 of these advantages, which avoid dependency of trainees on expert feedback. The goal of this study was to investigate the use of a visual time indicator as real-time feedback correlated with the laparoscopic surgical training. Twenty novices participated in this study working with (and without) different presentations of time indicators. They performed a standard peg transfer task, and their completion times and muscle activity were recorded and compared. Also of interest was whether the use of this type of feedback induced any side effect in terms of motivation or muscle fatigue. Results. Of the 20 participants, 15 (75%) preferred using a time indicator in the training process rather than having no feedback. However, time to task completion showed no significant difference in performance with the time indicator; furthermore, no significant differences in muscle activity or muscle fatigue were detected with/without time feedback. Conclusion. The absence of significant difference between task performance with/without time feedback shows that using visual real-time feedback can be included in surgical training based on user preference. Trainees may benefit from this type of feedback in the form of increased motivation. The extent to which this can influence training frequency leading to performance improvement is a question for further study.

Keywords

biomedical engineering simulation surgical education

Introduction

The complexity of medical tasks makes skills training a critical part of medical education and recertification.^1,2 Laparoscopic surgery is one example of these complex task sets; this type of surgery offers benefits in terms of patient outcomes, but requires more substantial training for novice surgeons to advance to the required degree of mastery. Surgical task proficiency is evaluated by 2 different modalities: the assessment of expert surgeons based on direct observation and computerized analysis.

The need for reliable and consistent evaluation to assess new trainees is evolving training methods from subjective expert surgeons to the more automated and computerized approach. Computer technology has been leveraged to advance the training procedure by developing virtual reality simulators and augmented reality simulators³ (combining real objects with virtual reality). However, the difficulty and cost of creating high-fidelity virtual reality environments are the main drawbacks of using this method.^4-7 The capability of augmented reality to blend some of the advantages of actual and virtual training makes it a promising approach for developing reliable and effective training systems. This type of system has been shown to enhance surgical performance of trainees.^8,9

A goal-based approach to encourage trainees based on meaningful learning experiences increases their interest and curiosity to practice the task.¹⁰ Meaningful learning is:¹¹

Active: It needs to be engaged and involved in the task

Constructive: It needs to use knowledge and apply it in the task

Intentional: It needs to provide a goal for the task

Authentic: It needs to reflect the complexity of the real world

Cooperative: It needs to involve communication with other people

The ability to combine the advantages of feedback to expedite the process of learning has led to the development of surgical simulation and demonstration of its value in surgical training.¹² Therefore, providing more information in the task environment to motivate trainees to practice more extensively is the objective of this study.

The most common standard training system that is used by expert surgeons to evaluate surgical residents is the Fundamentals of Laparoscopic Surgery (FLS) program¹³; this includes a set of representative tasks, a hardware setup, and an evaluation rubric. However, lack of automatic evaluation, which makes it dependent on the expert evaluator, is the main drawback of this approach.¹⁴ In addition, the transfer of skills obtained from the FLS program to the corresponding skills required in the operating room is another challenge for this method.

Several research studies have shown that observed and executed movement are interrelated and observation can help trainees to improve their performance.^15,16 Therefore, real-time visual feedback which provides more information for trainees can influence their performance and would be more helpful compared with summative, postexercise feedback. Judkins et al¹⁷ showed that the physiological demand in terms of muscle effort can be reduced by using augmented visual feedback in surgical training. However, distraction and stress are 2 main negative factors that have to be considered with all the potential advantages of real-time feedback.^18,19

For certain types of augmented visual feedback, increased trainee motivation can be a positive outcome. Dongen et al²⁰ introduced a competitive approach to help subjects improve their skills and task completion time. The motivation coming from the competitive element had an effect to encourage the subjects to do the task more. Therefore, the more frequent use of the simulator helped subjects improve their skills level. However, in some cases feedback can lead to mental or emotional stress. The effect of emotional pressure on muscle activity was presented to show the correlation between these 2 parameters.^21,22 The performance of the subject may be adversely affected in the presence of the mental stressor.¹⁸ Therefore, if feedback leads to mental stress, muscle activity may change, decreasing the training validity or effectiveness of task training involving neuromuscular coordination. The consideration of these different criteria, including motivation and amount of stress, represents an important tradeoff that merits further examination.

Time to task completion is one of the main criteria in existing surgical training. In this article, we examine how inclusion of a visual time indicator as a form of augmented real-time feedback influences surgical training. In the present study, we hypothesized that real-time feedback during the task process would provide more information for the trainees, motivate them to focus on areas for improvement and thereby make more rapid progress compared to the standard approach in which no feedback was provided. Different time indicators were implemented to the FLS peg transfer task to measure the influence of time observation on subject performance. Twenty novice participants used the FLS hardware to complete the peg transfer task with different time indicators overlaid on the video feed as a form of augmented reality. Their muscle activity was also recorded to investigate the presence of any secondary effects related to muscle fatigue, which could be induced by the stress of using this additional feedback.

Material and Methods

Subjects

Twenty novices participated in this study. The population consisted of 11 (55%) men and 9 (45%) women. The subjects were nonmedical students and none of them had previous experience with laparoscopic devices. Subjects were exposed to 4 different cases (3 different time indicators and 1 control without any time indicator) in the laparoscopic training tasks, performed in random order.

Equipment

Fundamentals of Laparoscopic Surgery

In this study, we used the FLS simulator hardware (Figure 1). This simulator consists of a partially enclosed box with light source and camera mounted inside of the box. Two laparoscopic graspers are constrained by the holes placed in the top of the device to simulate the pivot point of a small incision in laparoscopic surgery. The camera is connected to a frame grabber to convert the analog video signal into a digital stream. MATLAB software is used to capture the data from the camera and overlay time indicators onto the video frames. Subjects viewed the workspace on a 19-inch display. The peg board (surgical analog task) was placed in the center of the camera’s field of view.

Figure 1.

Fundamentals of Laparoscopic Surgery (FLS) simulator.

Time Indicator

For the peg transfer task in this study, 3 different time indicators were implemented to observe the dependency of subject performance on the presence or absence of this feedback. For this particular FLS task, the acceptable completion time is 48 seconds.²³ To display the time feedback, MATLAB software is used to track the position of the rings in the peg board using the images from the FLS camera. The subtraction of the basic color channels in the RGB image from the grayscale image provides a binary image indicating the colored rings. The positions of the rings on the peg board, along with the elapsed time, allows the software to calculate and indicate the rate of progress. The time feedback modalities are presented as 3 different overlaid virtual guides on the screen; 2 horizontal countdown time-bars and a digital countdown:

Time-bar graphic with a pointer indicating 48 seconds. The time-bar starts with a green color, decreasing in size and changing to red when the elapsed time reaches 48 seconds (Figure 2). When the task is complete, a numeric indicator displays the completion time for that trial.

Numeric countdown timer indicator (Figure 3). It displays a countdown beginning from 48 seconds. After passing zero, the indicator shows negative numbers until task completion.

Performance-based time-bar graphic. In this method, the allotted time per transferred object is calculated based on the performance from previous session. The time-bar is green when the trainee is ahead of time compared with his or her previous session performance (Figure 4). A red time-bar displays that the trainee is not improving in terms of speed.

Figure 2.

Time-bar graphic with a pointer. (a) Performing before 48 seconds (with green time bar). (b) Performing after 48 seconds (with red time bar).

Figure 3.

Numeric countdown timer indicator.

Figure 4.

Time-bar graphic sensitive to progress. (a) Performance improving (green progress bar). (b) Performance regressing compared with previous trial (red progress bar).

Questionnaire

Participants filled out a questionnaire after all tasks were complete. The mental demand for each variant of the task was evaluated in the first part of the survey. The second section asked subjects to judge the realism, didactic value, and usefulness of the time indicators during the surgical training task. They were also asked to compare their preferred time indicator with the control (absence of any time indicator). An open-ended question solicited feedback on the pros and cons of each task condition and their preference.

Protocol

An introduction was provided to the participants, including information about the FLS task and the four modules in this study. The subjects became acquainted with the task by watching a video of peg transfer. According to FLS program standards, subjects then used the grasper in their nondominant hand to grasp each object and transfer it to their dominant hand, placing it on the opposite side of the board. The process was then reversed for all 6 objects. Each subject performed a total of 16 trials (4 sets of 4), with each set of 4 including 1 trial of each module in random order.

Data Analysis

Electromyography (EMG) data for muscle activities were collected using a Trigno Wireless EMG system (Delsys, Inc, Boston, MA). The data from four selected muscles, biceps brachii (BB), triceps brachii (TB), flexor carpi radialis (FCR), and extensor digitorum (ED), were recorded in each trial to analyze muscle activity and fatigue. The raw EMG data are recorded with a rate of 2000 Hz using EMGworks acquisition software (Delsys, Inc, Boston, MA). The fast Fourier transform (FFT) algorithm was used to calculate the frequency power spectrum. The “fmed” parameter shows the median frequency of each task at half of the total frequency power.

Statistics

Data were analyzed using MATLAB (MathWorks, Natick, MA) with a band-pass filter of 200 to 300 Hz. The raw EMG signals are normalized to be able to compare EMG activity in different individuals. By dividing the raw data during a task by the maximal voluntary contraction (MVC) obtained from the same muscle, we then compared the normalized muscle activity in a repeatable value scale.

Results

Of the 20 participants, 15 (75%) preferred using a time indicator in the training process rather than having no feedback. Nine out of 20 (45%) rated that a visual indicator as feedback causes distraction during the task; nevertheless, most of this group felt that the benefits outweighed this drawback. As shown in Figure 5c, half of the subjects chose the performance-based time-bar, 25% preferred no time feedback, and 25% preferred the countdown time indicator. We assumed 3 attributes of the augmented reality feedback which might influence trainee performance: motivation, competition, and stress. The subjects scored these on a scale from 1 to 100. The strongest factor was found to be motivation (39%), followed by stress (34%) and competition (27%), respectively.

Figure 5.

Participant experience.

The muscle activity is summarized in Table 1 to present the similarity between performance with and without time indicators.

Table 1.

P-values for Comparison of Muscle Performance with and without Aggregated Time Feedback.

Muscles	RMS	Fmed
RBB	0.941	0.75
RTB	0.929	0.630
RFCR	0.998	0.768
RED	.9195	0.876
LBB	0.946	0.97
LTB	0.917	0.858
LFCR	0.78	0.9440
LED	0.935	0.982

Abbreviations: RBB, right biceps brachii; RTB, right triceps brachii; RFCR, right flexor carpi radialis; RED, right extensor digitorum; LBB, left biceps brachii; LTB, left triceps brachii; LFCR, left flexor carpi radialis; LED, left extensor digitorum; RMS, root-mean-square; Fmed, median frequency.

Nonparametric analysis using Kruskal-Wallis Test was applied to compare all 4 feedback conditions for each individual muscle. The P values for all muscles (.63 < P < .99) are large enough (P > .05) to show no significant differences between these different methods. So, we conclude that the time indicators do not affect the stress on subjects’ muscles.

The whole pool of completion time data, combining all cases of time feedback and comparing against the control (no time feedback) are used to test for difference between no time feedback and feedback in general. In addition, each form of time feedback was compared against the control individually.

No significant difference (P = .501) was found for the completion time between all indicators and control. The average time for no time feedback was 137.05 seconds. Times of 141.80, 140.01, and 134.15 seconds were recorded for time bar sensitive to each transfer, time bar with 48 second indicator, and countdown indicator, respectively.

A set of independent t tests with least significant difference (LSD) method was used to measure the difference between each form of time feedback separately against the control.²⁴ The value of LSD is 6.83 seconds, and the difference between each time feedback level and the control is less than LSD. Therefore, there was no significant difference between each individual time feedback and the control.

Discussion

Gaining the basic skills of laparoscopic surgery is the reason to develop training simulators for use outside of the operating room. The simulation-based training should provide enough skills acquisition to adequately prepare trainees for actual surgical tasks.²⁵ Mimicking the realistic surgical environment and improving evaluation output are the main factors for choosing a simulator.

This study showed how visual feedback implemented to the FLS simulator can help trainees improve their performance and efficiency. The outcome of this research indicates the preference of participants to have access to time feedback and have an idea of how far they are in the task. Distraction undoubtedly would be the main drawback of this type of feedback (see Figure 5a); however, the benefits of motivation and competition seem to outweigh this drawback. By examining the muscle activity, there was no significant difference between the 4 feedback conditions. These results show that the participants do not experience higher physiological demand when using augmented visual feedback. Furthermore, the P value of the median frequency between these results demonstrates that muscle fatigue is similar for all conditions.

In our experiment, the P values of the completion time showed no significant improvement when using the various time indicators. Based on this lack of significant difference, time indicators can be used in augmented-reality surgical simulators based on user preference, but should not be expected to directly lead to decrease in task completion time.

One of the main factors that helps trainees to improve their skills is motivation. The rewards that trainees gain by the feedback of their performance can encourage them to work more and improve their operational skill. The time indicators, especially the performance based one, motivate the trainees to reach or improve on their previous time; trainees may implicitly learn from this the most effective and efficient way to interact with objects to improve task completion time. The subjective data collected verifies this assumption that the learning process is more efficient when they have more feedback and motivation to work. The subjects indicated that the interest of finishing the tasks faster than the previous trials encouraged them to work with the task more and more.

The similarity in muscle activity for different feedback settings, and the increase of motivation to finish the task faster, validate some of our hypothesis to use time indicators in the training process. Similar muscle activity and muscle fatigue for all conditions during the task training shows the lack of negative side-effects from real-time visual feedback. The data did not show any improvement in trainee performance compared with the no-feedback condition for a short-term training process. However, the motivation coming from comparison of the current running activity with the previous performance might encourage trainees to devote more time on the task, and this factor requires further investigation.

In addition, some limitations of the study that should be noted include the following:

All participants were novices; further study is merited for including different levels of surgical experience.

The design of augmented visual feedback mainly focused on time. It is possible that different forms of augmented visual feedback, for example, background color changes, might also impact surgical skills learning.

Only 4 muscles over the upper arm and forearm were selected in this study. The upper body, such as shoulder or upper back, could be an area of interest for future study to evaluate the effect of augmented visual feedback on surgical training.

Conclusion

Different time indicators as real-time visual feedback were used to determine their influence on the progress rate of trainees. Twenty subjects worked with variants of a surgical training task and their time completion and muscle activity were recorded to investigate the benefit of real-time visual feedback.

Two performance metrics were defined, one (subjective) for the personal opinion of the subjects, and one (objective) for overall results from muscle activity and completion time. More than two-thirds of the participants preferred having the time indicator, and the EMG data did not show any significant effect on the muscle activity and muscle fatigue compared with having no feedback (control condition). In addition, motivation and competition were the other valuable subjective factors, which may encourage trainees to increase their simulator training frequency.

Future Work

The benefits of real-time visual feedback recorded with non-medical students represent a practical feature to use in medical training simulators. This preliminary evidence supports the validity of this feature to motivate trainees to increase their task performance and its feasibility in terms of lack of detrimental side effects related to muscle effort. Future work should include similar experiments with medical students and experts to see how this augmented reality feedback can help trainees to improve and maintain their skills level.

Footnotes

Author Contributions

Study concept and design: Mohsen Zahiri, Carl A. Nelson,Dmitry Oleynikov, Ka-Chun Siu

Acquisition of data: Mohsen Zahiri

Analysis and interpretation: Mohsen Zahiri, Carl A. Nelson, Ka-Chun Siu

Study supervision: Carl A. Nelson, Dmitry Oleynikov, Ka-Chun Siu

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This study was funded by Nebraska EPSCoR and UNeTECH.

References

Ifenthaler

Eseryel

. Facilitating complex learning by mobile augmented reality learning environments. In: Huang

Spector

Kinshuk

DGS

, eds. Reshaping Learning, the Frontiers of Learning Technologies in a Global Context. New York, NY: Springer; 2013:415-438.

Van Merriënboer

JJG

Kirschner

PA.

Ten Steps to Complex Learning, a Systematic Approach to Four Component Instructional Design. New York, NY: Routledge: 2007.

Zahiri

Booton

Siu

Nelson

CA.

Design and evaluation of a portable laparoscopic training system using virtual reality. J Med Devices. 2017;11:011002.

Aggarwal

Moorthy

Darzi

Laparoscopic skills training and assessment. Br J Surg. 2004;91:1549-1558.

Botden

SMBI

Torab

Buzink

Jakimowicz

. The importance of haptic feedback in laparoscopic suturing training and the additive value of virtual reality simulation. Surg Endosc. 2008;22:1214-1222.

Ashkezari

Mirbagheri

Farahmand

Behzadipour

Firoozbakhsh

Real time simulation of grasping procedure of large internal organs during laparoscopic surgery. Paper presented at: Engineering in Medicine and Biology Society (EMBC), Annual International Conference of the IEEE; August 28, 2012; San Diego, CA.

Bao

. Real-time cutting simulation in virtual reality systems based on the measurement of porcine organs [published online August 21, 2017]. Simulation doi:10.1177/0037549717726144.

Botden

Jacimowicz

JJ.

What is going on in augmented reality simulation in laparoscopic surgery. Surg Endosc. 2009;23:1693-1700.

Kamphuis

Barsom

Schijven

Christoph

Augmented reality in medical education?

Perspect Med Educ. 2014;3:300-311.

10.

Salomon

Perkins

DN.

Rocky roads to transfer: rethinking mechanisms of a neglected phenomenon. Educ Psychol. 1989;24:113-142.

11.

Jonassen

DH.

Meaningful Learning With Technology. 3rd ed. Upper Saddle River, NJ: Pearson; 2008.

12.

Fried

Feldman

Vassiliou

et al . Proving the value of simulation in laparoscopic surgery. Ann Surg. 2004;240:518-528.

13.

Peters

Fried

Swanstrom

et al ; SAGES FLS Committee. Development and validation of a comprehensive program of education and assessment of basic fundamentals of laparoscopic surgery. Surgery. 2004;135:21-27.

14.

McCluney

Vassiliou

Kaneva

et al . FLS simulator performance predicts intraoperative laparoscopic skill. Surg Endosc. 2007;21:1991-1995.

15.

Tia

Mourey

Ballay

Sirandré

Pozzo

Paizis

Improvement of motor performance by observational training in elderly people. Neurosci Lett. 2010;480:138-142.

16.

Jahanbani Fard

. Computational Modeling Approaches for Task Analysis in Robotic-Assisted Surgery [dissertation]. Detroit, MI: Wayne State University; 2016.

17.

Judkins

Oleynikov

Stergiou

Electromyographic response is altered during robotic surgical training with augmented feedback. J Biomech. 2009;42:71-76.

18.

Mehta

Agnew

MJ.

Subjective evaluation of physical and mental workload interactions across different muscle groups. J Occup Environ Hyg. 2015;12:62-68.

19.

Siu

Suh

Mukherjee

Oleynikov

Stergiou

The impact of environmental noise on robot-assisted laparoscopic surgical performance. Surgery. 2010;147:107-113.

20.

van Dongen

van der Wal

Rinkes

Schijven

Broeders

IA.

Virtual reality training for endoscopic surgery: voluntary or obligatory?

Surg Endosc. 2008;22:664-667.

21.

Duffy

The relation between muscular tension and quality of performance. Am J Psychol. 1932;44:535-546.

22.

Cooke

Kavussanu

McIntyre

Ring

Psychological, muscular and kinematic factors mediate performance under pressure. Psychophysiology. 2010;47:1109-1118.

23.

Ritter

Scott

DJ.

Design of a proficiency-based skills training curriculum for the fundamentals of laparoscopic surgery. Surg Innov. 2007;14:107-112.

24.

Montgomery

Runger

Hubele

NF.

Engineering Statistics. Hoboken, NJ: Wiley; 2009.

25.

Fraser

Feldman

Stanbridge

Fried

GM.

Characterizing the learning curve for a basic laparoscopic drill. Surg Endosc. 2005;19:1572-1578.