Development of a Model for the Acquisition and Assessment of Advanced Laparoscopic Suturing Skills Using an Automated Device

Abstract

Background. Needs assessment identified a gap regarding laparoscopic suturing skills targeted in simulation. This study collected validity evidence for an advanced laparoscopic suturing task using an Endo Stitch^TM device. Methods. Experienced (ES) and novice surgeons (NS) performed continuous suturing after watching an instructional video. Scores were based on time and accuracy, and Global Operative Assessment of Laparoscopic Surgery. Data are shown as medians [25th-75th percentiles] (ES vs NS). Interrater reliability was calculated using intraclass correlation coefficients (confidence interval). Results. Seventeen participants were enrolled. Experienced surgeons had significantly greater task (980 [964-999] vs 666 [391-711], P = .0035) and Global Operative Assessment of Laparoscopic Surgery scores (25 [24-25] vs 14 [12-17], P = .0029). Interrater reliability for time and accuracy were 1.0 and 0.9 (0.74-0.96), respectively. All experienced surgeons agreed that the task was relevant to practice. Conclusion. This study provides validity evidence for the task as a measure of laparoscopic suturing skill using an automated suturing device. It could help trainees acquire the skills they need to better prepare for clinical learning.

Keywords

laparoscopic suturing surgical training competency-based education

Introduction

Suturing is a skill that is required to perform a variety of different laparoscopic procedures across many different surgical specialties. Surgical trainees, however, find that laparoscopic suturing (LS) is a difficult skill to acquire and master. Simulation is often used to help trainees practice difficult skills, or to transfer part of the learning curve for these skills outside of the operating room (OR), but there is a lack of simulation platforms for this advanced skill. The Fundamentals of Laparoscopic Surgery (FLS^®) program provides a platform to teach and assess basic laparoscopic skills, including basic LS training.^1-3 However, in the OR, the conditions are more variable and nuanced and the skills needed for suturing are more complex and advanced.

A previous needs assessment identified a gap between the LS skills needed in the OR compared with what is taught and assessed in the various simulation platforms.^4,5 Based on this, a set of advanced LS models using free needles was developed and tested for validity evidence.^6-8 The models, however, were not adapted to be used with automated suturing devices such as Endo Stitch^TM. The needs assessment also revealed a gap with regard to learning how to use suturing devices, which are also commonly used in the OR.⁹ Simulation platforms that are currently available for this are limited in terms of number, cost, and validity evidence.^10,11 Therefore, there is a need to develop cost-effective tasks and metrics that could be used to teach and assess the skills required for device-assisted LS skills.

The purpose of this study was to design and collect validity evidence for an advanced LS task and metrics using the Endo Stitch device and to develop an assessment tool that could be used for assessment and feedback of Endo Stitch skills.

Materials and Methods

Task Description

The task, based on our previous advanced LS models for free needle, was developed with readily available and low-cost materials, and in consultation with experts. First, we tried the fabric that was used for the free needle tasks; however, due to the differences in the properties of the free needle compared with the needle of the device, the Endo Stitch needle would get stuck and the needle would come off of the device. Therefore, in order to develop the task, various materials were tested such as different types of fabrics and sponges, and the Endo Stitch only worked with one type of sponge (it is developed by the company Richard and is called Professional Foam Rubber Grouting Sponge, made of foam rubber). Therefore, the task was developed with the sponge, and a gap of 7 cm in the middle was created with 5 premarked black dots on each side of the gap (1 cm apart from one another; Figure 1). The task was placed in the FLS trainer box (Limbs & Things, Savannah, GA). The participants were asked to start from the top and perform continuous suturing with intracorporeal knot-tying at the beginning and at the end; they had to go through the premarked black dots.

Figure 1.

The Endo Stitch task.

Study Design

Experienced surgeons (ESs) and novice surgeons (NSs) performed the task once after watching an instructional video (the video can be made available on request). First, determination of participant experience was based on the number of cases they have done using the Endo Stitch. NSs were allowed to warm up for up to 3 minutes to get use to the Endo Stitch device. ESs completed the task a second time, in order to establish proficiency benchmarks. The participants also completed a questionnaire on demographics and perceived educational value. Scores were based on time and accuracy and assigned by direct observation. The cutoff time was 1200 seconds. Accuracy scores were based on knot security (0 secure, 10 slipping, 20 knot comes apart), millimeter gap in the incision, millimeter distance from the premarked dots, and number of skipped dots. The accuracy score itself is calculated by the summation of the 4 components. The task score is calculated as time taken by the participant minus accuracy. In this formula, accuracy is multiplied by 3 in order to emphasize the importance of accuracy along with speed. Participant performances were also recorded and assessed by one rater using the Global Operative Assessment of Laparoscopic Surgery (GOALS) instrument.¹² For analysis, participants who received a GOALS score of ≥20 were considered as experienced surgeons.

Validity evidence was gathered by comparing task scores and GOALS scores between the 2 groups. Data are shown as medians [25th-75th percentiles] (ES vs NS). Interrater reliability was calculated using intraclass correlation coefficients (confidence interval); 2 raters assessed accuracy for all of the completed tasks and assessed time using the recorded performances of 5 participants. Comparison of task scores, accuracy scores alone, and GOALS scores between ESs and NSs were made using Mann-Whitney U test. Pearson’s correlation was used to correlate task scores to GOALS scores. The proficiency benchmarks were determined by using the 75th percentile of the ES performances, using the score from their second task completion.

For the creation of the Endo Stitch Assessment Tool (EAT), ESs who completed the task were interviewed in order to understand the skills that are important when performing LS using Endo Stitch. ESs were asked to comment on the key steps required to perform LS with Endo Stitch in this task and watch a video of a NS performing the task while commenting on how the person could improve their performance. After the creation of the tool, 2 raters assessed the video-recorded performances to assess the interrater reliability. Comparison of EAT scores between ES and NS were made using Mann-Whitney U test. Pearson’s correlation was used to correlate EAT scores to task scores and GOALS scores. All analysis was done using SAS 9.3 and SPSS V20.0.0. A P value ≤.05 was considered statistically significant.

Power Analysis

Pilot performance data for continuous suturing without using an automated suturing device from 18 senior surgical residents and 13 minimally invasive surgery–trained surgeons showed significant differences in performance in both training platforms.⁷ In order to show a similar difference between the 2 groups in performance with an α of .05 and a power of 80%, with 2-sided testing, a total of 8 participants per group is required.

Results

Seventeen participants (9 ESs, 8 NSs; median age = 34 years; 76% male) were enrolled (details on demographics can be found in Table 1). All the ESs completed a minimally invasive/Bariatric surgery fellowship. Table 2 shows comparisons between ESs and NSs performing the new Endo Stitch task. Compared with NSs, ESs had significantly greater scores (980 [964-999] vs 666 [391-711], P = .0035). ESs made errors in “gaps in incision” and “distance from premarked dots”; however, NSs made significantly more errors overall (6 [4-9] vs 20 [12-23], P = .014). ESs also received significantly better scores in GOALS (25 [24-25] vs 14 [12-17], P = .0029). The Pearson correlation between task scores and GOALS scores was 0.89. Interrater reliability for time and accuracy were 1.0 and 0.9 (0.74-0.96), respectively. All ESs agreed or strongly agreed that the task was relevant to practice, and that it could be used to improve device-assisted suturing skills. Two of the 9 ESs only had 1 trial; therefore, the performances of the other 7 participants were used to determine the benchmarks. The proficiency benchmarks can be seen in Table 3.

Table 1.

Participant Characteristics (N = 17)^a.

Characteristics	Novice Surgeons (n = 8)	Experienced Surgeons (n = 9)
Age (years)	33.5 [30.25-38.25]	34 [33.5-39]
Gender (male)	5 (63)	8 (89)
Training level
PGY3	2 (25)	0 (0)
PGY4	0 (0)	0 (0)
PGY5	2 (25)	0 (0)
Fellow	2 (25)	5 (56)
Attending surgeon	2 (25)	4 (44)
Laparoscopic suturing experience with Endo Stitch^b
None	2 (25)	0 (0)
1-5 cases	6 (75)	0 (0)
6-25 cases	0 (0)	2 (22)
26-50 cases	0 (0)	5 (56)
51+ cases	0 (0)	2 (22)

Abbreviation: PGY, postgraduate year.

Results are in n (%) or median [25th-75th percentiles].

Self-reported case experience.

Table 2.

Comparison Between Experienced and Novice Surgeons^a.

	Experienced^b	Novice^b	P *
Task score^c	980 [964-999]	666 [391-711]	.0035
Accuracy^d	6 [4-9]	20 [12-23]	.014
GOALS	25 [24-25]	14 [12-17]	.0029

Abbreviation: GOALS, Global Operative Assessment of Laparoscopic Skills.

Experienced or Novice surgeons in performing suturing with Endo Stitch.

Results are in median [25th-75th percentiles].

Total task scores (time minus accuracy).

Accuracy count only.

Significance when P < .05.

Table 3.

Proficiency Benchmarks Determined Through the Performances of Experienced Surgeons (N = 7).

Time (Seconds)	Accuracy
181	1 mm off from the dots
	4 mm gaps in closure
	0 knot security error
	0 skipped dots

From interviewing 7 ESs, an assessment tool with 13 items was developed, with a rating scale of 0 to 2 (0 = does poorly, 1 = needs some improvement, and 2 = does well) and a total score of 26. The tool structure was based on assessment tools that were developed for the free-needle advanced LS tasks.¹³ It includes generic and specific items regarding steps needed when suturing using an Endo Stitch (Table 4). Compared with NSs, ESs had significantly greater EAT scores (26 [24-26] vs 16 [11-20], P = .0032). The Pearson correlation between EAT scores and the task scores and GOALS scores was 0.95 and 0.94, respectively. The interrater reliability was 0.9 (0.61-0.97).

Table 4.

The Endo Stitch Assessment Tool Items for the Task.

1. Selects an appropriate suture length for the task

2. Knows how to use the Endo Stitch instrument (ie, how to toggle the needle and how the mechanism works)

3. Anticipates the angle of the Endo Stitch when penetrating the target tissue for correct orientation of the needle

4. Understands the relationship between the suture and the Endo Stitch to tie a knot

5. Leaves suture tail an appropriate length when tying the knot

6. Uses the nondominant hand as a post while tying the knot

7. Uses the dominant hand holding the Endo Stitch to tie around the post to make a knot

8. Ties the knots securely (tightly)

9. Coordinates the use of both hands

10. Takes the bites in the right orientation (ie, clockwise fashion to avoid leakage)

11. Keeps the excess suture out of the way while tying the knots and running the suture

12. Keeps the target tissue steady while suturing

13. Tightens the suture line and maintains appropriate tension while running the suture

Discussion

FLS was developed to address basic laparoscopic skills, including suturing. However, there was a need for advanced platforms to teach and assess LS skills, and in particular device-assisted LS. In order to address this need, and to model the skills needed in the OR, this new task was developed. This study provides preliminary validity evidence for an advanced LS task using Endo Stitch; the interrater reliability for the metrics are high, the task is able to differentiate experts from novices, and the task metrics are highly correlated with the video-taped assessment of their task performance using GOALS. ESs agreed on the relevancy of the task to practice, and its potential to improve device-assisted suturing skills. Also, an assessment tool that assesses the Endo Stitch suturing skills in the task was developed, with good preliminary validity evidence.

There are some simulation platforms available for LS. Specifically, for usage with automated devices, the available platforms are a lot less in number, costly, and have limited validity evidence.¹⁰ Our task was developed from readily available, low-cost materials, and it can be used in a variety of bench-top boxes. If the purpose of using the task is for teaching, individuals can tailor it to their needs. Also, since there are different types of automated suturing devices apart from the Endo Stitch, this task could be used with the various devices and validity evidence could be demonstrated.

The EAT was developed based on expert opinions regarding skills needed to perform LS in our task and validity evidence was demonstrated. Therefore, for the assessment of this task, either the tool and/or the time and accuracy scores could be used. However, this tool could also be used for formative assessment, in order to give feedback and improve the skill of the novices. In a previous study, using the previously developed advanced LS tasks for free needle, we compared skill improvement after training on the suturing models using assessment tools that we based our EAT tool on (without a guide) versus guidance from an expert.^13,14 It was found that both groups improved their suturing skills, suggesting that the items of the tools accurately reflected aspects of skill that experts considered to be important. Therefore, further investigations could be conducted in order to understand the value of our assessment tool for formative feedback.

This study has some limitations. First, the study was done at a single institution, which could limit the generalizability of our results to other training programs. Second, we have not investigated how task performance correlates with OR performance and if the skills gained from this task transfer to the OR.

This study provides validity evidence for the task, its metrics, and the assessment tool as measures of LS skill using an automated suturing device. Incorporating this task into the training curricula could help trainees acquire the skills they need to be better prepared for and maximize clinical learning in the operating room.

Footnotes

Author Contributions

Study concept and design: Elif Bilgic, Madoka Takao, Pepa Kaneva, Yusuke Watanabe, Liane S. Feldman, Melina C. Vassiliou

Acquisition of data: Elif Bilgic, Madoka Takao

Analysis and interpretation: Elif Bilgic, Madoka Takao, Pepa Kaneva, Satoshi Endo, Toshitatsu Takao, Yusuke Watanabe, Katherine M. McKendy, Liane S. Feldman, Melina C. Vassiliou

Study supervision: Melina C. Vassiliou

Declaration of Conflicting Interests

The author(s) declared the following potential conflicts of interest with respect to the research, authorship, and/or publication of this article: The Steinberg-Bernstein Centre for Minimally Invasive Surgery and Innovation received an unrestricted educational grant from Medtronic Canada. Dr Melina C. Vassiliou received Investigator Sponsored Research in the form of Endo Stitch devices and sutures from Medtronic. Elif Bilgic, Pepa Kaneva, and Drs Madoka Takao, Toshitatsu Takao, Yusuke Watanabe, Katherine M. McKendy, Satoshi Endo, and Liane S. Feldman have no relevant conflicts of interests to disclose.

Funding

The author(s) received no financial support for the research, authorship, and/or publication of this article.

References

Ritter

Scott

DJ.

Design of a proficiency-based skills training curriculum for the fundamentals of laparoscopic surgery. Surg Innov. 2007;14:107-112.

Vassiliou

Dunkin

Marks

Fried

GM.

FLS and FES: comprehensive models of training and assessment. Surg Clin North Am. 2010;90:535-558.

Fried

Feldman

Vassiliou

et al . Proving the value of simulation in laparoscopic surgery. Ann Surg. 2004;240:518-525.

Mattar

Alseidi

Jones

et al . General surgery residency inadequately prepares trainees for fellowship: results of a survey of fellowship program directors. Ann Surg. 2013;258:440-449.

Nepomnayshy

Alseidi

Fitzgibbons

Stefanidis

Identifying the need for and content of an advanced laparoscopic skills curriculum: results of a national survey. Am J Surg. 2016;211:421-425.

Nepomnayshy

Whitledge

Birkett

et al . Evaluation of advanced laparoscopic skills tasks for validity evidence. Surg Endosc. 2015;29:349-354.

Watanabe

McKendy

Bilgic

et al . New models for advanced laparoscopic suturing: taking it to the next level. Surg Endosc. 2016;30:581-587.

Bilgic

Watanabe

Nepomnayshy

et al ; Simulation Committee of the Association for Surgical Education. Multicenter proficiency benchmarks for advanced laparoscopic suturing tasks. Am J Surg. 2016;213:217-221.

Enani

Watanabe

McKendy

et al . What are the training gaps for acquiring laparoscopic suturing skills? J Surg Educ. 2017;74:656-662.

10.

Kurenov

Punak

Peters

Lee

Cendan

Development and initial validation of a virtual reality haptically augmented surgical knot-tying trainer for the Autosuture ENDOSTITCH instrument. Stud Health Technol Inform. 2009;142:145-147.

11.

Rinewalt

Velasco

JM.

Evaluation of a novel laparoscopic simulation laboratory curriculum. Surgery. 2012;152:550-556.

12.

Vassiliou

Feldman

Andrew

et al . A global assessment tool for evaluation of intraoperative laparoscopic skills. Am J Surg. 2005;190:107-113.

13.

McKendy

Watanabe

Bilgic

et al . Establishing meaningful benchmarks: the development of a formative feedback tool for advanced laparoscopic suturing. Surg Endosc. 2017;31:5057-5065.

14.

Munshi

Watanabe

McKendy

et al . Use of a formative feedback tool in place of an expert coach in laparoscopic suturing training: a randomized non-inferiority trial. Surg Endosc. 2016.