Effectiveness of Telementoring in Surgery Compared With On-site Mentoring: A Systematic Review

Abstract

Background. Mentorship is important but may not be feasible for distance learning. To bridge this gap, telementoring has emerged. The purpose of this systematic review was to evaluate the effectiveness of telementoring compared with on-site mentoring. Methods. A search was done up to March 2015. Studies were included if they used telementoring between surgeons during a clinical encounter and if they compared on-site mentoring and telementoring. Results. A total of 11 studies were included. All reported no difference in complication rates, and 9 (82%) reported similar operative times; 4 (36%) reported technical issues, which was 3% of the total number of cases in the 11 studies. No study reported on higher levels of evidence for effectiveness of telementoring as an educational intervention. Conclusion. Studies reported that telementoring is associated with similar complication rates and operative times compared with on-site mentoring. However, the level of evidence to support the effectiveness of telementoring as a training tool is limited. There is a need for studies that provide evidence for the equivalence of the effectiveness of telementoring as an educational intervention in comparison with on-site mentoring.

Keywords

telementoring training telemedicine teaching distance learning

Introduction

Adoption of advanced technologies and surgical procedures relies on successful mentoring relationships. Although this model works well for formal training programs, it is not as practical for surgeons who want to integrate new procedures into their practice and who may be geographically distant from appropriate mentors.¹ Learning opportunities such as courses or simulation training exist.² However, they are often not enough for the needs of surgeons to actually start performing new procedures, and they do not provide follow-up and ongoing support. To bridge this gap, telementoring has emerged as an alternative to on-site mentoring.³ By definition, telementoring is where a remote expert surgeon (mentor) guides a less-experienced surgeon (mentee) during real-time surgical procedures.⁴ Even though in the field of medicine communication technologies have been used for years, their use for telementoring is a more recent phenomenon.⁵ With advancements in technology, distance communication has evolved from audiovisual only to video options, including telestration (drawing on the monitor that could be seen by the mentee), robotic arms, electrosurgical control, and so on.⁶ This may provide additional benefits and safety to the mentors, mentees, and patients.

Telementoring has been studied in different fields, and it seems to be a promising approach; however, the implementation, quality, and effectiveness of the experience can vary because of various limitations.^7,8 Telementoring requires specific equipment and software that could be costly, time management of the mentor and the mentee to conduct the sessions, and a preexisting relationship between the mentor and mentee (eg, through pretelementoring sessions). In addition, an important limitation is related to legal and ethical considerations that need to be taken into account.⁹ Several studies have investigated the use of telementoring to support surgeons, but they are quite heterogeneous both in study design and quality.^10-13 There are also some reviews available for surgical telementoring; however, they have limitations in terms of the search strategy and study design.^14,15 The purpose of this systematic review was to evaluate the effectiveness of telementoring compared with on-site mentoring as a way to support surgeons introducing new surgical procedures or techniques.

Materials and Methods

A systematic literature search of bibliographic databases and conference proceedings was performed without date limits up to March 2015. Further studies were identified up to October 2015 by carrying out citation searches. The MEDLINE and PubMed databases were re-searched in May 2016 for any new studies that could be included. The reporting of the review was conducted according to the Preferred Reporting Items for Systematic Reviews and Meta-Analysis (PRISMA) standards.

Eligibility

Full-text articles, in English or French, were considered. Studies were included if they reported on the use of telecommunications platforms for mentoring between surgeons during a clinical encounter. Studies reporting only encounters between surgeons and patients, nonclinical telementoring (eg, simulation), or mentoring for the purposes of assessment were excluded.

Search Strategy

Bibliographic databases (MEDLINE, EMBASE, BIOSISPreviews, ERIC, The Cochrane Library, PubMed, Web of Science, and Scopus), conference proceedings, and gray literature were searched for relevant, full-text articles. The search strategy used keywords and relevant indexing to identify articles discussing the use of telementoring for remote medical education. The full MEDLINE strategy (see the appendix) was applied to all databases, with search term modifications as needed. Citation searches in Web of Science and Scopus were carried out on the included studies to identify further studies for inclusion.

Data Extraction

The initial search results were screened by 2 independent reviewers (EB and ST). Studies were chosen based on their title and abstract according to the inclusion criteria. Any inconsistencies or disagreements were resolved by discussion and consensus. Full-text analysis was completed on articles that were included, and study characteristics will be reported. The effectiveness of telementoring was analyzed in 2 ways: (1) safety in regard to changes in the operative outcomes in telementored versus on-site mentored cases and technical issues, and (2) effectiveness of telementoring as an educational intervention compared with on-site mentoring, which includes reported perceptions of participants and is evaluated using a modified Kirkpatrick Model.¹⁶ The Kirkpatrick Model levels are defined in Table 1. Additionally, we reported on studies that described the various limitations of telementoring for implementation, which include cost, equipment requirements, and legal and ethical issues.

Table 1.

Modified Kirkpatrick Model Used to Evaluate the Effectiveness of Telementoring Educational Interventions.²⁶.

Levels	Description^a
Level 1: Reaction	Are mentors/mentees satisfied with the telementoring sessions?
Level 2: Learning	What did the telementoring sessions teach the mentees?
Level 3: Behavior	Did the mentees change their practice pattern as a result of the newly acquired skills?
Level 4^b: Results	Did the telementoring sessions have an impact on mentee performance and patient outcomes?

Telementoring compared with on-site mentoring.

Level 4 is the highest level of evidence that a study can provide for educational interventions.

Results

Study Characteristics

The primary search identified 40 972 studies; 18 006 duplicates were removed, and the remaining 22 966 titles and abstracts were screened for relevance. Of these, 467 titles underwent full-text review, of which 11 met our inclusion criteria and were included for analysis (Figure 1).^17-27 There were a total of 262 telementored and 191 on-site mentored cases (1 study did not report the case numbers for on-site mentoring), for a total of 453 cases. Study characteristics are shown in Table 2.

Figure 1.

Study identification and selection flowchart.

Table 2.

Characteristics of the 11 Studies.

Characteristics	n^a (%)
Types of surgery
Laparoscopic surgery	6 (55)
Robotic surgery	2 (18)
Open surgery	2 (18)
Endoscopic surgery	1 (9)
Types of systems
Audiovisual systems (AVS) alone ± telestration	9 (82)
AVS + Remotely controllable robotic arm + Telestration	2 (18)
Location
Within a single institution	6 (55)
2 Institutions	3 (27)
Multiple institutions	2 (18)
Study design
Description of telementoring systems	11 (100)
Cohort studies^b
Non–randomized controlled, comparative studies	11 (100)

n Is the number of studies reporting the item out of the 11 studies.

Compared telementoring with on-site mentoring.

Changes in the Operative Outcomes and Technical Issues With Telementoring

There were a total of 453 cases in 11 studies. All studies reported no difference in complication rates between on-site versus telementored cases; 10 studies reported on operative time, and 9 of those (90%) reported that on-site and telementored cases had similar operative times. Table 3 provides more details.

Table 3.

Reported Changes in the Operative Outcomes Comparing On-site With Telementoring.

	n (%)
Operative complications (n = 11)
No difference	11 (100)
Operative times (n = 10)
No difference	9 (90)^a
Longer	1 (10)
Estimated blood loss (n = 4)
No difference	4 (100)^b
Length of stay (n = 5)
No difference	4 (80)^c
Shorter	1 (20)^d

352 Cases out of 453 total cases in the 11 studies (78%).

268 Cases out of 453 total cases in the 11 studies (59%).

97 Cases out of 453 total cases in the 11 studies (21%).

48 Cases out of 453 total cases in the 11 studies (11%).

Out of the 11 studies, 7 (64%) studies reported that they had a backup faculty in the operating room in case there was a need for on-site faculty intervention. In the remaining 4 studies, presence of a back-up faculty was not reported; however, those studies did not acquire any problems. None of the studies reported session disruptions as a result of delays in transmission. Four (36%) studies (8 cases out of 262 telementored cases in 11 studies) reported technical issues, which accounted for 3% of the total number of telementored cases in the 11 studies (Table 4). Among the 8 cases, in 3, the issue was successfully dealt with and did not require an on-site faculty intervention; a wireless system was changed to a wired one, the system had to be restarted, and the video froze, but the connection was reestablished. In 3 cases, an on-site faculty intervention was required because of improper robot arm positioning and video freezing; 2 cases also had an on-site faculty intervention as a result of case difficulty. Figure 2 provides a breakdown of the studies that reported technical issues.

Table 4.

Reported Technical Issues of 11 Studies.

	n^a (%)
Technical issues
No	7 (64)
Yes	4 (36)^b

n Is the number of studies reporting the item out of the 11 studies.

This equals 3% of the telementored cases in the 11 studies.

Figure 2.

Breakdown of the cases that had technical difficulties.

Telementoring as an Educational Intervention

Using the modified Kirkpatrick Model,²⁸ 4 (36%) studies reported level 1 evidence by reporting mentor-mentee satisfaction. Within the 4 studies, 2 studies reported no difference in satisfaction between telementoring versus on-site mentoring, and 2 reported satisfaction with telementoring and did not mention on-site mentoring. One (9%) study reported level 2 evidence by comparing mentee performances in the first and second halves of the sessions for both telementoring and on-site mentoring.²³ They found a significant improvement in operative times for both types of mentoring. No study reported level 3 or 4 evidence.

Limitations of Telementoring for Implementation

There are certain limitations that could create obstacles for the implementation of telementoring.^7,8 First is the cost of the telementoring sessions, which includes the cost of the equipment. Among the 11 studies, 1 study mentioned the cost of implementing telementoring. They reported that the cost to telementor 4 laparoscopic live donor nephrectomies (12 hours use) was around US$18 000. The second involves legal and ethical issues; 8 studies reported that they had ethics approval before starting the sessions, and 6 reported that they obtained informed consent from the patients. None of the studies reported on licensing requirements for the mentors. Third is the need for a preexisting relationship between the mentor and mentee, which could be done through pretelementoring sessions, where the mentor and mentee have meetings in order to become familiar with each other and establish standardized approaches to the procedure that will be telementored. One study included pretelementoring training to help mentees acquire basic skills in laparoscopy and get used to receiving off-site guidance before starting telementoring sessions.

Discussion

This systematic review suggests that telementoring causes minimal/no additional adverse effects, with minimal time delays and technical difficulties. Even though studies reported that there were no differences in the complication rates, the definition of adverse events was not uniform across studies. Also, some studies simply reported “similar complication rates” or “no increase in complication rates” without a more detailed information of what was actually measured. Because of the inconsistency and variability of what studies considered as “complication,” a meta-analysis could not be conducted.

Delay in transmission is a very important concept that should be carefully examined when testing telementoring equipment/sessions. However, in our review, none of the studies reported session disruptions caused by delays in transmission.

Two of the studies investigated telementoring in robotic surgery. In these studies, they used an audiovisual system with telestration abilities. Therefore, the mentors did not have control of the robot, which is traditionally not considered telementoring because of the fact that the mentors have direct control over the procedure, which is more similar to telesurgery (expert operates on a patient in another location) rather than telementoring.

Despite the fact that cost is one of the main concerns in implementing telementoring, only 1 study reported the cost of the telementoring sessions. Furthermore, only 1 study mentioned a pretelementoring training session, regardless of the fact that this may be an essential component of the success of any program. This might be so essential that it is done routinely and taken for granted, but there is no way to ascertain this from the publications. Training and preparation, in addition to appropriate communication parameters and a trusting relationship between mentor and mentee, could potentially have a significant impact on the success of a telementoring program. None of the studies investigated the importance of longitudinal relationships between mentor and mentee, and the impact of this on the effectiveness and efficiency of telementoring.

Most studies reported on having ethics approval and/or obtaining informed consent. At the end of the day, the mentee is responsible for the proper care of the patient. This is why it is important to have a mentee who is competent at managing the patients’ condition if telementoring does not function, probably through using another technique. However, we should remember that telementoring could allow the patient to have access to a type of procedure that might be more effective for their condition, which would improve patient care.⁷ Because ethical issues in telementoring is a big limitation factor, ethics approval and patient consent should be something that is reported in every study.²⁹ Also, studies did not define well-established patient selection criteria, which could be important for the success of the sessions as well as for ethical considerations.

From province to province and country to country, licensing and hospital credentialing varies, which could be a barrier to the implementation of telementoring. Most studies did not mention this limiting factor, even though it might have affected the feasibility of conducting their studies.

We excluded any study that was not a comparison study of on-site versus telementoring. The main reason for this is that we were evaluating whether or not telementored sessions were going to have similar or different profiles. Within noncomparative studies, it was not feasible to draw any conclusions regarding that because only telementoring sessions were evaluated.

There are many case series that show telementoring to be an important educational tool in dissemination of new or advanced procedures.^10,11 However, without comparison with on-site mentoring, evidence for its effectiveness is limited. From the 4 studies that reported on this, we have found that mentors and mentees were satisfied with the sessions as a teaching/learning tool. Nonetheless, studies that evaluate telementoring in regard to the 4 outcomes of the Kirkpatrick model are needed in order to show that telementoring is an effective educational intervention.

In simulation, various metrics such as time and accuracy, assessment tool scores, number of task repetitions, and training hours are used as outcome measures when assessing performance and when comparing the effectiveness of various educational interventions.^30-33 In telesimulation (telementoring done in a simulation setting), some studies have used questionnaires to evaluate satisfaction with the system and the mentoring sessions, recorded tool path length and time of task completion, or assessed performance using an operative performance rating scale.^34-36 In the clinical setting, some telementoring studies (without comparison with on-site mentoring) have used identification of anatomical landmarks, procedure time, length of stay in intensive care unit, length of hospital stay, and number of interactions between mentee and mentor as outcomes of interest.^20,37,38 When investigating the effectiveness of telementoring sessions as an educational intervention in comparison with on-site mentoring, some of these outcomes could be used.

There are many different types of technologies available for telemedicine and quite a few for telementoring. However, implementation of educational programs that use telementoring is still not common. The major reason for this is probably not the availability of the technologies, but rather the various limitations in regard to cost, legal and ethical issues, equipment requirements, and time commitment by the mentee and mentor (for the procedures themselves and for the pretelementoring training sessions). Therefore, studies should be conducted to address and guide surgeons through the limitations.

In conclusion, studies report that telementoring is associated with similar complication rates and operative times compared with on-site mentoring. However, the level of evidence to support the effectiveness of telementoring is limited, and there is no concrete evidence for equivalence of teaching in telementoring versus on-site mentoring. There is a need for studies that provide evidence for the effectiveness of telementoring and address the limitations for implementation of telementoring.

Footnotes

Appendix

Author Contributions

Study concept and design: Elif Bilgic, Yusuke Watanabe, Liane S. Feldman, Melina C. Vassiliou

Acquisition of data: Elif Bilgic, Tara Landry, Daniel Lavigne

Analysis and interpretation: Elif Bilgic, Sena Turkdogan, Yusuke Watanabe, Amin Madani, Liane S. Feldman, Melina C. Vassiliou

Study supervision: Liane S. Feldman, Melina C. Vassiliou

Declaration of Conflicting Interests

The author(s) declared the following potential conflicts of interest with respect to the research, authorship, and/or publication of this article: The Steinberg-Bernstein Centre for Minimally Invasive Surgery and Innovation receives an unrestricted educational grant from Covidien Canada. Elif Bilgic, Sena Turkdogan, Yusuke Watanabe, Amin Madani, Tara Landry, Daniel Lavigne, Liane S. Feldman, and Melina C. Vassiliou have no relevant conflicts of interests or financial ties to disclose.

Funding

The author(s) received no financial support for the research, authorship, and/or publication of this article.

References

Sebajang

Trudeau

Dougall

. Telementoring: an important enabling tool for the community surgeon. Surg Innov. 2005;12:327-331.

Miskovic

Wyles

Darzi

Hanna

GB.

Systematic review on mentoring and simulation in laparoscopic colorectal surgery. Ann Surg. 2010;252:943-951.

Rosser

Young

Klonsky

Telementoring: an application whose time has come. Surg Endosc. 2007;21:1458-1463.

Pande

Patel

Powers

D’ancona

Karamanoukian

HL.

The telecommunication revolution in the medical field: present applications and future perspective. Curr Surg. 2003;60:636-640.

Debakey

ME.

Telemedicine has now come of age. Telemed J E Health. 1995;1:3-5.

Bauer

JJ.

Lee

Bishoff

. International surgical telementoring using a robotic arm: our experience. Telemed J. 2000;6:25-31.

Schlachta

Nguyen

Ponsky

Dunkin

Project 6 Summit: SAGES telementoring initiative. Surg Endosc. 2016;30:3665-3672.

Bogen

Augestad

Patel

Lindsetmo

R-O.

Telementoring in education of laparoscopic surgeons: an emerging technology. World J Gastrointest Endosc. 2014;6:148-155.

Challacombe

Kavoussi

Patriciu

Stoianovici

Dasgupta

Technology insight: telementoring and telesurgery in urology. Nat Clin Pract Urol. 2006;3:611-617.

10.

Ponce

Jennings

Clay

May

Huisingh

Sheppard

. Telementoring: use of augmented reality in orthopaedic education: AAOS exhibit selection. J Bone Joint Surg Am. 2014;96:e84.

11.

Anvari

Remote telepresence surgery: the Canadian experience. Surg Endosc. 2007;21:537-541.

12.

Rodrigues Netto

Mitre

Lima

SVC

. Telementoring between Brazil and the United States: initial experience. J Endourol. 2003;17:217-220.

13.

Adams

Schulam

Moore

Partin

Kavoussi

LR.

New laparoscopic suturing device: initial clinical experience. Urology. 1995;46:242-245.

14.

Augestad

Bellika

Budrionis

. Surgical telementoring in knowledge translation—clinical outcomes and educational benefits: a comprehensive review. Surg Innov. 2013;20:273-281.

15.

Antoniou

Franzen

. A comprehensive review of telementoring applications in laparoscopic general surgery. Surg Endosc. 2012;26:2111-2116.

16.

Love

Yarris

Santen

. A novel specialty-specific, collaborative faculty development opportunity in education research: program evaluation at five years. Acad Med. 2016;91:548-555.

17.

Deaton

Balch

Kesler

Bogey

Powell

CS.

Telemedicine and endovascular aortic grafting. Am J Surg. 1999;177:75-77.

18.

Schulam

Docimo

Saleh

Breitenbach

Moore

Kavoussi

Telesurgical mentoring: initial clinical experience. Surg Endosc. 1997;11:1001-1005.

19.

Burgess

LPA

Syms

Holtel

Birkmire-Peters

Johnson

Ramsey

. Telemedicine: teleproctored endoscopic sinus surgery. Laryngoscope. 2002;112:216-219.

20.

Di Valentino

Alerci

Bogen

. Telementoring during endovascular treatment of abdominal aortic aneurysms: a prospective study. J Endovasc Ther. 2005;12:200-205.

21.

Docimo

Moore

Kavoussi

LR.

Telerobotic surgery is clinical reality: current experience with telementoring in adults and children. Presence. 1997;6:173-178.

22.

Shin

Dalag

Azhar

. A novel interface for the telementoring of robotic surgery. BJU Int. 2015;116:302-308.

23.

Hinata

Miyake

Kurahashi

. Novel telementoring system for robot-assisted radical prostatectomy: impact on the learning curve. Urology. 2014;83:1088-1092.

24.

Challacombe

Kandaswamy

Dasgupta

Mamode

Telementoring facilitates independent hand-assisted laparoscopic living donor nephrectomy. Transplant Proc. 2005;37:613-616.

25.

Sawyer

MAJ

Lim

Wong

SYT

Cirangle

Birkmire-Peters

. Telementored laparoscopic cholecystectomy: a pilot study. Stud Health Technol Inform. 2000; 70:302-308.

26.

Lim

Wong

SYT

Cirangle

. Telementored cholecystectomy. In: Proceedings of the 7th World Congress of Endoscopic Surgery. Bologna: Monduzzi Editore; 2000:309-313.

27.

Rosser

Jr Wood

Payne

. Telementoring: pushing the telemedicine envelope. J Assoc Acad Minor Phys. 1997;8(1):11-15.

28.

Cox

Seymour

Stefanidis

Moving the needle: simulation’s impact on patient outcomes. Surg Clin North Am. 2015;95:827-838.

29.

Anvari

Telesurgery: remote knowledge translation in clinical surgery. World J Surg. 2007;31:1545-1550.

30.

Snyder

Vandromme

Tyra

Hawn

MT.

Proficiency-based laparoscopic and endoscopic training with virtual reality simulators: a comparison of proctored and independent approaches. J Surg Educ. 2009;66:201-207.

31.

Watanabe

McKendy

Bilgic

. New models for advanced laparoscopic suturing: taking it to the next level. Surg Endosc. 2016;30:581-587.

32.

Bilgic

Watanabe

Nepomnayshy

. Multicenter proficiency benchmarks for advanced laparoscopic suturing tasks. Am J Surg. 2017;213:217-221.

33.

Moorthy

Munz

Dosis

Bello

Chang

Darzi

Bimodal assessment of laparoscopic suturing skills: construct and concurrent validity. Surg Endosc. 2004;18:1608-1612.

34.

Sereno

Mutter

Dallemagne

Smith

Marescaux

Telementoring for minimally invasive surgical training by wireless robot. Surg Innov. 2007;14:184-191.

35.

Panait

Rafiq

Tomulescu

. Telementoring versus on-site mentoring in virtual reality-based surgical training. Surg Endosc. 2006;20:113-118.

36.

Ereso

Garcia

Tseng

. Live transference of surgical subspecialty skills using telerobotic proctoring to remote general surgeons. J Am Coll Surg. 2010;211:400-411.

37.

Rafiq

Moore

Zhao

Doarn

Merrell

RC.

Digital video capture and synchronous consultation in open surgery. Ann Surg. 2004;239:567-573.

38.

Byrne

Mughal

MM.

Telementoring as an adjunct to training and competence-based assessment in laparoscopic cholecystectomy. Surg Endosc. 2000;14:1159-1161.