Technical Skill Impacts the Success of Sequential Robotic Suturing Substeps

Abstract

Introduction:

Robotic surgical performance, in particular suturing, has been associated with postoperative clinical outcomes. Suturing can be deconstructed into substep components (needle positioning, needle entry angle, needle driving, and needle withdrawal) allowing for the provision of more specific feedback while teaching suturing and more precision when evaluating suturing technical skill and prediction of clinical outcomes. This study evaluates if the technical skill required for particular substeps of the suturing process is associated with the execution of subsequent substeps in terms of technical skill, accuracy, and efficiency.

Materials and Methods:

Training and expert surgeons completed standardized sutures on the Mimic™ Flex virtual reality robotic simulator. Video recordings were deidentified, time annotated, and provided technical skill scores for each of the four suturing substeps. Hierarchical Poisson regression with generalized estimating equation was used to examine the association of technical skill rating categories between substeps.

Results:

Twenty-two surgeons completed 428 suturing attempts with 1669 individual technical skill assessments made. Technical skill scores between substeps of the suturing process were found to be significantly associated. When needle positioning was ideal, needle entry angle was associated with a significantly greater chance of being ideal (risk ratio [RR] = 1.12, p = 0.05). In addition, ideal needle entry angle and needle driving technical skill scores were each significantly associated with ideal needle withdrawal technical skill scores (RR = 1.27, p = 0.03; RR = 1.3, p = 0.03, respectively). Our study determined that ideal technical skill was associated with increased accuracy and efficiency of select substeps.

Conclusions:

Our study found significant associations in the technical skill required for completing substeps of suturing, demonstrating inter-relationships within the suturing process. Together with the known association between technical skill and clinical outcomes, training surgeons should focus on mastering not just the overall suturing process, but also each substep involved. Future machine learning efforts can better evaluate suturing, knowing that these inter-relationships exist.

Introduction

Intraoperative surgeon performance has been associated with postoperative clinical outcomes and complications.¹ In particular, suturing has been studied within surgical procedures with our own recent work finding that suturing performance is linked to urinary continence recovery following robot-assisted radical prostatectomy (RARP).²

Recently, suturing has been investigated with more granular detail. Suturing can be broken down into its sequential substep components (i.e., needle positioning, needle entry angle, needle driving, and needle withdrawal ³). This decomposition of suturing can allow for the provision of more targeted feedback for training surgeons by identifying specific substeps that require additional practice.⁴ This has been evidenced by the development and validation of assessment tools at the substep level of suturing.³ In addition, studying suturing at a more granular level can identify specific substeps that are discriminative of surgeon skill (i.e., training vs expert surgeon).^5,6 The detail provided by the substeps of suturing has shown clinical relevance with our most recent work finding that the performance of suturing substeps could more accurately predict urinary continence recovery following RARP compared with the performance of suturing maneuvers as a whole.⁷ Taken together, these previous works highlight the relevance of further exploration of suturing at the substep level.

To date, no previous work has assessed associations in the technical skills required in each substep and the impact technical skill has on the efficiency and accuracy of subsequent substeps of the suturing process. Understanding the inter-relationships of successive suturing substeps will help guide how to teach suturing more effectively and which aspects should be elaborated on.

Herein, we evaluate if the technical skill required for substeps of the suturing process is associated with the execution of subsequent substeps in terms of technical skill, accuracy, and efficiency. Furthermore, we will evaluate whether training and expert surgeons show differences in technical skill, accuracy, and efficiency metrics for the four substeps of suturing.

Materials and Methods

Training and expert surgeons completed a standardized suturing exercise on the Mimic™ Flex Virtual Reality simulator. Each participant was asked to complete the exercise four times. A priori, training surgeons were defined as those having robotic surgery caseload ≤100 (junior urologic residents), and expert surgeons were defined as having a robotic surgery caseload >100 (senior urologic residents, urologic robotic surgery fellows, urology faculty). A console caseload cutoff of 100 was based on a meta-analysis and previous study from our laboratory.^8,9 Participants completed a standardized exercise consisting of a simulated sponge with illuminating targets that directed where to throw consecutive suturing stitches (Fig. 1). Video recordings of the completed exercises were deidentified and start and end times (seconds) of the four substeps (needle positioning, needle entry angle, needle driving, and needle withdrawal) were annotated to the thousandth decimal place. Binary technical skill scores (ideal or nonideal) were given to each individually annotated substep of the suturing sequence (needle position, entry angle, driving, and withdrawal). Grading criteria were guided by the robotic anastomosis competency evaluation (RACE) and as described previously.^3,10 Ideal technical skill scoring criteria are depicted in Figure 2 and described below:

FIG. 1.

A representative image of the simulator exercise. Color graphics available online.

FIG. 2.

Schematic representation of scoring of ideal suturing technical skill and associations in technical skill scores of successive suturing substeps (i.e., needle positioning, needle entry angle, needle driving, and needle withdrawal). Technical skill scores between needle positioning and needle entry angle (1), needle entry angle and needle driving (2), and needle driving and needle withdrawal (3) were found to be significantly associated (solid line). An anticipated relationship between needle entry angle and needle driving was not found (a, dashed line). Data reported as RR, p-value. Solid lines represent p < 0.05. RR, risk ratio. Color graphics available online.

- Needle positioning (1): needle held between the ½ and ¾ position from the needle tip.

- Needle entry angle (2): needle enters the tissue at 90° ± 30°.

- Needle driving (3): smooth wrist articulation with no tissue trauma/excessive force.

- Needle withdrawal (4): smooth wrist articulation with no tissue trauma/excessive force.

Technical skill scores were assigned by three blinded and independent graders (D.S., B.D., and R.H.) who were trained in a standardized manner by the senior author (A.H.). Evaluators participated in a training session where sample training videos were reviewed as a group and individually. Absolute group consensus was reached for any discrepant labels. For analysis, we defined accuracy as the effective target hit rate out of total target attempts for the first or second target, a metric relevant to needle entry angle (2) and needle driving (3), respectively. Efficiency was the time required for a substep calculated using the time annotations; this metric was assessed for needle driving (3) and needle withdrawal (4).

Hierarchical Poisson regression with generalized estimating equation was used to examine the association of technical skill rating categories (ideal vs nonideal) between sequential suturing substeps (e.g., needle driving substep to needle withdrawal substep) by adopting the nested data structure of multiple training exercises within each trainee. Model integrity was diagnosed using residual plots. Overdispersion was examined using a negative binomial model's dispersion parameter. We have adjusted potential confounders and tested effect modifiers in the Poisson model. Following a D'Agostino–Pearson normality test, comparisons of efficiency and accuracy were performed with the paired Student's t-test or Wilcoxon signed-rank test. SAS version 9.4 (SAS Institute, Cary, NC) was used for all data analyses.

Our study complied with protocols approved by the University of Southern California's IRB. All procedures performed in studies involving human participants were in accordance with the ethical standards of the institutional and/or national research committee and with the 1964 Helsinki Declaration and its later amendments or comparable ethical standards. This article does not contain any studies with animals performed by any of the authors. Informed consent was obtained from all individuals included in the study.

Results

Twenty-two surgeons including 7 training (median caseload 0, IQR: 0–8) and 15 expert (300: 165–750) surgeons completed a standardized suturing exercise four times. Altogether, surgeons completed 428 individual suturing attempts with 1669 individual technical skill assessments made.

Overall frequencies of ideal technical skill scores, accuracy, and efficiency metrics are presented in Table 1. When stratified by expertise, ideal technical skill scores were seen at greater rates for experts compared with trainees for needle driving (55.5% ± 2.9 vs 41.9% ± 4.1, p = 0.01) and needle withdrawal (54.4% ± 3.2 vs 31.3% ± 4.4, p < 0.01). Other evaluated metrics did not reach significance, although trends generally favored expert surgeons.

Table 1.

Overview of Technical Skill Scores, Accuracy, and Efficiency Metrics of Suturing Substeps

	Parameter	Overall	Training	Expert	p^a
Needle positioning	Technical Skill Score	76.2% ± 3.4	74.3% ± 5.6	77.0% ± 2.6	0.66
Needle entry angle	Technical Skill Score	73.4% ± 2.6	74.4% ± 4.3	72.9% ± 2.4	0.76
Needle entry angle	Entry Target Accuracy	69.6% ± 3.3	62.8% ± 5.8	72.7% ± 3.4	0.15
Needle driving	Technical Skill Score	51.4% ± 3.7	41.9% ± 4.1	55.5% ± 3.0	0.01
	Time (seconds)	2.2 ± 0.1	2.0 ± 0.2	2.3 ± 0.1	0.17
	Exit Target Accuracy	65.9% ± 2.8	62.9% ± 3.7	67.6% ± 2.7	0.23
Needle withdrawal	Technical Skill Score	47.5% ± 4.1	31.3% ± 4.4	54.4% ± 3.2	<0.01
Needle withdrawal	Time (seconds)	2.2 ± 0.2	2.3 ± 0.2	2.2 ± 0.2	0.74

Technical skill scores and accuracy metrics calculated as number of ideal scorings/total scorings. Overall represents the entire cohort of surgeons. Data reported as mean ± SEM.

Significant differences between training and expert surgeons were seen for needle driving (3) and needle withdrawal (4) technical skill. Significant differences are bolded. Training surgeons defined as caseload ≤100, Expert surgeons defined as caseload >100.

Associations in the technical skill of successive substeps of suturing

Technical skill scores between substeps of the suturing process were found to be statistically significantly associated (Fig. 2). For example, ideal needle positioning score (1) was statistically significantly associated with greater chance of being ideal needle entry angle (2) (78.5% ± 2.8 vs 69.9% ± 4.0, risk ratio [RR] = 1.12, p = 0.05). In addition, ideal entry angle (2) and needle driving (3) technical skill scores were each significantly associated with greater chance of ideal needle withdrawal (4) technical skill scores (50.2% ± 3.2 vs 39.4% ± 4.3, RR = 1.27, p = 0.03; 53.2% ± 3.9 vs 40.9% ± 4.1, RR = 1.3, p = 0.03, respectively). An anticipated relationship between needle entry angle (2) and needle driving (3) was not statistically significant ( p = 0.21).

On multivariate analysis of all four suturing substeps, incorrect needle positioning (1) technical skill (when controlled for needle entry angle) was associated with a greater likelihood of correct needle withdrawal (4) technical skill (RR = 1.32 confidence interval [95% CI = 1.06–1.64] p = 0.01), while correct needle entry angle (2) (when controlled for needle positioning) was associated with a greater likelihood of correct needle withdrawal (4) technical skill (RR = 1.39 [95% CI = 1.11–1.75] p < 0.01)) (Fig. 3). When stratified by surgeon experience, the association between incorrect needle positioning (1) and correct needle withdrawal (4) technical skill was only seen for training surgeons (Expert: RR = 1.192 [95% CI = 0.93–1.52], p = 0.16 vs trainee: RR = 1.81 [95% CI = 1.14–12.88], p = 0.01). In contrast, the association between correct needle entry angle (2) and correct needle withdrawal (4) technical skills was only seen for expert surgeons (Expert: RR = 1.39 [95% CI = 1.07–1.8] p = 0.01 vs trainee: RR = 1.41 [95% CI = 0.87–2.31], p = 0.17). Needle positioning (1) and needle entry angle (2) were not significantly associated with needle driving (3) by multivariate analysis, including when stratified by surgeon experience.

FIG. 3.

Multivariate analysis of associations in technical skill scores of successive suturing substeps. Incorrect needle positioning (1) technical skill (when controlled for needle entry angle) was associated with a greater likelihood of correct needle withdrawal (4) technical skill. Correct needle entry angle (2) (when controlled for needle positioning) was associated with a greater likelihood of correct needle withdrawal (4) technical skill. Data reported as mean ± SEM. *Represents p < 0.05. Color graphics available online.

Measures of accuracy and efficiency

Our study also assessed measures of accuracy, efficiency, and technical skill. We found that when needle driving technical skill was ideal, the time required to complete needle driving was less (ideal vs nonideal = 1.74 ± 0.08 seconds vs 2.69 ± 0.16 seconds, p < 0.01) and there was a greater likelihood of hitting the exit needle target (ideal % hit vs nonideal % hit = 70.06% ± 2.91 vs 61.46% ± 3.27, p = 0.05). Similarly, when needle withdrawal technical skill was ideal, the time required to complete the withdrawal driving was less (ideal vs nonideal = 1.43 ± 0.1 seconds vs 2.88 ± 0.21 seconds, p < 0.01).

Discussion

Suturing is an essential component of surgical procedures. Previous work demonstrated that intraoperative suturing performance is linked to urinary continence recovery following RARP.² Recently, suturing has been investigated with greater detail at a substep level (i.e., needle positioning, needle entry angle, needle driving, and needle withdrawal ³). The present study investigates associations in technical skill required for completing these discrete substeps of suturing.

Our study found significant associations in the technical skill used to complete successive suturing substeps. This is the first study to demonstrate inter-relationships between the substeps of suturing and measure the strength of the relationship. Ideal technical skill scores between needle positioning and needle entry angle, needle entry angle and needle withdrawal, as well as between needle driving and needle withdrawal were significantly positively associated. These findings suggest that the technical skill required for completing individual substeps of the suturing process may be related and/or impacted by other suturing substeps. For example, our data suggest that a surgeon with proper technical skill for needle positioning is more likely to also have proper technical skill for needle entry angle. These suturing substeps occur sequentially and having proper suturing technical skill for early steps may improve the surgeon's likelihood of effectively progressing through the suturing throw.

In a simpler form, suturing can be split into “planning” (i.e., needle positioning and needle entry angle) and subsequent “execution” (i.e., needle driving and needle withdrawal) steps. Our data suggest that the technical skill required for completing “planning” steps of the suturing process is associated with a surgeon's ability to perform the subsequent “execution” steps. This finding aligns with our expectation that for the sequential steps of suturing, performance during the initial substeps impacts the performance in subsequent steps. One exception was the relationship between needle positioning and needle withdrawal on multivariate analysis. A potential explanation is that if a suturing needle is held at the extreme distal end (i.e., considered nonideal needle positioning by our accepted criteria of ½ to ¾ from the needle tip), it may be able to be driven further into a tissue making needle withdrawal easier and a subsequent greater likelihood of an ideal needle withdrawal technical skill score being assigned.

Teaching suturing at the substep level can allow for provision of more precise and targeted feedback to training surgeons.⁴ Prior work found that suturing, studied at a more granular level, can identify specific substeps that are discriminative of surgeon skill (i.e., training vs expert surgeon),^5,6 highlighting where emphasis might be placed when teaching suturing. The importance of studying and teaching suturing with this greater detail has been evidenced by the development and validation of assessment tools at the substep level of suturing³ (i.e., RACE). The results of the present study support further refinement of suturing technical skill instruction at this substep level.

We propose a training curriculum that emphasizes learning and mastering the aforementioned “planning” steps of suturing followed by mastery of the “execution” steps and then the suturing process as a whole. One setting in which this could be implemented is while learning the 12 stitches of a vesicourethral anastomosis from RARP. This pivotal surgical procedure involves suturing the bladder and urethra together following prostatic removal. To optimize technical skill acquisition, trainees can initially prioritize practicing needle positioning and then needle entry angle for each entry point. Different stitches may require adaptation of the needle position and/or entry angle; stepwise practice of each entry point allows opportunity for skill acquisition and targeted feedback. Following mastery of needle positioning and needle entry angle, surgeons can incrementally include needle driving and needle withdrawal maneuvers, building upon an existing skill set. Our data suggest that mastery of the “planning” substeps will be associated with higher technical skill scores of the subsequent “execution” substeps. Further evaluation with a randomized-controlled trial can determine whether this curriculum hastens the learning curve for suturing mastery and can have significant impacts on patient's clinical outcomes.

Our study also assessed measures of accuracy, efficiency, and technical skill for our overall cohort and between expert and training surgeons. Our analysis of technical skill in the context of efficiency and accuracy showed that ideal technical skill scores not only displayed associations with more efficient gestures but also resulted in a greater likelihood of effectively achieving the task at hand (hitting the needle target). In addition, we found that expert surgeons, compared with training surgeons, scored higher technical skill scores in the needle driving and needle withdrawal substeps, arguably the more consequential (execution) domains of the suturing process. Other metrics trended toward greater technical skill for our expert cohort, although they did not reach significance. These somewhat logical findings were methodically quantified, helping better our understanding of the suturing process.

The present work furthers current understanding of the suturing process by looking at associations in how well the specific substeps of suturing are performed and the impacts this has on the technical skill, accuracy, and efficiency of other suturing substeps. Our focus on manually derived technical skill assessments allows for more direct and readily understood feedback that can be provided for training surgeons. In the future, we hope to automate suturing technical skill evaluation using machine learning. Machine learning can be a relatively more opaque process; it is therefore beneficial to first explore any relationships between suturing substeps in the most transparent way possible, utilizing a more traditional statistical approach. Our increasing understanding of the suturing process at the substitch level highlights some of the intrinsic relationships present within the suturing process. These data may help in determining which machine learning models may yield the best performance. For example, certain machine learning algorithms can use the sequential (i.e., temporal) nature of the process being automated when training a data set, whereas others analyze a single feature without this additional temporal information. Establishing the inter-relationships between the substeps of the suturing process may help in the selection of more appropriate machine learning models. Overall, the results of this study are foundational to future machine learning studies aimed at automating assessment of suturing technical skill.

Our study is not without limitations. We conducted this study with a relatively small group of surgeons at a single academic institution. In addition, these surgeons completed one standardized suturing exercise. Despite these limitations, future work can expand upon these initial findings with more participants at multiple institutions who are assessed after completing multiple exercises to verify the aforementioned relationships between suturing substeps. In addition, knowing these associations can help in future efforts to better understand the suturing process and in the development of more efficient and effective ways of teaching/evaluating suturing with training surgeons.

Conclusion

Suturing consists of a sequence of interrelated substeps, the success and efficiency of which are impacted by technical skill in other substeps. Together with the known association between technical skill and clinical outcomes, training surgeons should focus on mastering not just the overall suturing process, but each substep involved as well. Future machine learning efforts will better study suturing, now having these inter-relationships between suturing substeps elucidated and quantified.

Footnotes

Authors' Contributions

D.S. was in charge of project development, data collection and preparation, and article writing and editing. B.D. performed data collection and article writing and editing. T.F.H. performed data collection and article editing. R.M. did data collection, data analysis, and article editing. R.H. performed data collection and article editing. J.H.N. did article editing and project management. S.C. performed data analysis, and article editing. A.J.H. was in charge of project development, data management, and article writing and editing.

Funding Information

Research reported in this publication was supported, in part, by the National Institute of Biomedical Imaging and Bioengineering of the National Institutes of Health under Award No. K23EB026493, and by the National Cancer Institute under Award No. 1 R01CA251579-01A1.

Abbreviations Used

References

Birkmeyer

, Finks

, O'Reilly

, et al. Surgical skill and complication rates after bariatric surgery. N Engl J Med, 2013; 369:1434–1442.

Hung

, Chen

, Ghodoussipour

, et al. A deep-learning model using automated performance metrics and clinical features to predict urinary continence recovery after robot-assisted radical prostatectomy. BJU Int, 2019; 124:487–495.

Raza

, Field

, Jay

, et al. Surgical competency for urethrovesical anastomosis during robot-assisted radical prostatectomy: Development and validation of the robotic anastomosis competency evaluation. Urology, 2015; 85:27–32.

Vedula

, Malpani

, Tao

, et al. Analysis of the structure of surgical activity for a suturing and knot-tying task. PLoS One, 2016; 11:e0149174.

Chen

, Liang

, Nguyen

, Liu

, Hung

. Machine learning analyses of automated performance metrics during granular sub-stitch phases predict surgeon experience. Surgery, 2021; 169:1245–1249.

Vedula

, Malpani

, Ahmidi

, Khudanpur

, Hager

, Chen

. Task-level vs. segment-level quantitative metrics for surgical skill assessment. J Surg Educ, 2016; 73:482–489.

Trinh

, Mingo

, Vanstrum

, et al. Survival analysis using surgeon skill metrics and patient factors to predict urinary continence recovery after robot-assisted radical prostatectomy. Eur Urol Focus 2021. PubMed PMID: 33858811.

Hung

, Chen

, Jarc

, Hatcher

, Djaladat

, Gill

. Development and validation of objective performance metrics for robot-assisted radical prostatectomy: A pilot study. J Urol, 2018; 199:296–304.

Abboudi

, Khan

, Guru

, et al. Learning curves for urological procedures: A systematic review. BJU Int, 2014; 114:617–629.

10.

Der

, Sanford

, Hakim

, Vanstrum

, Nguyen

, Hung

. Efficiency and accuracy of robotic surgical performance decayed among urologists during COVID-19 shutdown. J Endourol, 2021; 35:888-890.