Multi-institutional Study of Otolaryngology Resident Intraoperative Experiences for Key Indicator Procedures

Methods

We prospectively collected operative assessments of surgical trainees between December 2019 and December 2020 on the SIMPL smartphone application. Participants at each of 5 centers underwent standardized rater training sessions, either in person or with a prerecorded video. These 1-hour sessions were previously demonstrated to be sufficient to train surgeons to reliably use SIMPL. ⁷ For each procedure, participants were asked to assess the complexity of the case relative to similar cases in tertiles (easiest, average, and hardest). Residents and faculty were also asked to assess the level of autonomy and performance achieved by the resident during each operation. Autonomy was rated on a 4-level Zwisch scale: show and tell, active help, passive help, and supervision only (see Table 1 for descriptions).^5,8 Meaningful autonomy was defined as passive help or supervision only. ³ If autonomy was rated higher than show and tell, performance was evaluated on a 5-level modified Dreyfus scale: unprepared/critical deficiency, inexperienced with procedure, intermediate performance, practice-ready performance, exceptional performance (see Table 1 for descriptions). ⁵ For cases with multiple unrelated components (eg, mastoidectomy performed with a parotidectomy), participants were asked to log only 1 of these procedures as the focus of each assessment. At the end of the evaluation, attendings could provide an audio recording of feedback to the resident about the case. All assessments were completed or expired within 72 hours of the reported procedure time, as prior research has described the degradation of evaluation quality after this time. ⁹ When residents increased their postgraduate year (PGY) standing in July of the study, this was reflected in the application database. This study was deemed exempt from review by the Institutional Review Board at each study site (Massachusetts Eye and Ear, University of California San Francisco, Boston University, University of Michigan, Mount Sinai Hospital).

OPEN IN VIEWER

Table 1.

Levels of Resident Autonomy and Performance Measured by the SIMPL Smartphone Application. ^a

Zwisch scale of operative autonomy
Level		Characteristic
1	Show and tell	• Attending does the majority of key portions of the procedure, narrating the case and the anatomy Resident opens, closes, first assists
2	Active help	• Attending leads the case for >50% of key portions, identifying anatomy, optimizing the surgical field, and coaching technical skills Resident actively assists and practices component skills
3	Passive help	• Attending follows the resident’s lead for >50% of key portions and coaches for refinement Resident recognizes transition points in the procedure and accomplishes next steps
4	Supervision only	• Attending gives no unsolicited advice for >50% of key portions, monitoring for safety Resident mimics independence and recovers from most errors
Modified Dreyfus scale of operative performance
1	Unprepared/critical deficiency	• Resident poorly prepared to perform the procedure
2	Inexperienced with procedure	• Resident is experienced with the procedure with frequent problems in technique, execution, planning
3	Intermediate performance	• Resident is at an intermediate phase of development; performance of procedural elements is variable but acceptable for the amount of experience with the procedure
4	Practice-ready performance	• Resident is able to perform the procedure safely and independently
5	Exceptional performance	• Resident performs above the level expected of graduating residents

Abbreviation: SIMPL, Society for Improving Medical Professional Learning.

a

Adapted from Chen et al. ⁶

Results

A total of 2984 evaluations were logged by 92 residents and 78 attendings, covering 367 types of procedures. Residents logged 1746 evaluations and attendings logged 1238. Each resident performed a median 10.5 ratings (range, 1-86, right skew); each attending performed a median 10 (range, 1-83, right skew). Fifty-four percent of residents and 55% of faculty logged at least 10 cases.

Across all cases, attending ratings of resident operative autonomy were positively correlated with level of training (Spearman’s ρ = 0.36, P < .001; Figure 1A ). Similarly, ratings of resident operative performance were positively correlated with level of training (Spearman’s ρ = 0.25, P < .001; Figure 1B ). Participants rated 11% (n = 339) of all cases as among the easiest one-third in case complexity, 68% (n = 2020) as average, and 21% (n = 625) as hardest. There was a significant correlation between attending ratings of case complexity and level of training (Spearman’s ρ = 0.15, P < .001; Figure 1C ).

OPEN IN VIEWER

Figure 1.

Attending surgeon ratings for (A) resident autonomy, (B) performance, and (C) case complexity across training levels. Postgraduate year 1 (PGY 1) was excluded as the number was small. *Not applicable: performance ratings not solicited if autonomy was rated “show and tell.”

Among all cases performed by senior residents (PGY 4 or 5), attendings reported that residents achieved meaningful autonomy (passive help or supervision only) in 62% of cases and practice-ready or exceptional performance ratings in 61%.

Key Indicator Procedures

Attendings evaluated 517 cases classified by the ACGME as otolaryngology KIPs, of which 276 were performed by senior residents (PGY 4 and 5). Attending ratings of resident operative autonomy and performance in KIPs were positively correlated with training level (autonomy, Spearman’s ρ = 0.46, P < .001; performance, Spearman’s ρ = 0.43, P < .001). Attending ratings of case complexity for KIPs were also correlated with training level (Spearman’s ρ = 0.11, P = .01).

Senior residents (PGY 4 or 5) achieved meaningful autonomy (passive help or supervision only) in 55% of KIP cases ( Figure 2A ); similarly, they received practice-ready or exceptional performance ratings for 55% ( Figure 2B ). Senior residents achieved meaningful autonomy for ≥50% of cases for most KIPs with the exception of flaps and grafts (40%), pediatric/adult airway (39%), and stapedectomy/ossiculoplasty (33%). Senior residents received practice-ready or exceptional performance ratings for ≥50% of cases across KIPs other than pediatric/adult airway (42%) and stapedectomy/ossiculoplasty (33%).

OPEN IN VIEWER

Figure 2.

Percentage of key indicator procedures performed by postgraduate year 4 and 5 (PGY 4 and 5) residents with (A) meaningful autonomy (passive help or supervision only) or (B) at least practice-ready operative performance in attending surgeon ratings. M/M, mandible/midface.

Discussion

This is the largest prospective multi-institutional study tracking otolaryngology surgical training experiences to date. Senior residents (PGY 4 and 5) in this cohort attained meaningful autonomy and achieved practice-ready performance for the majority (61%) of all logged procedures, including ACGME-defined KIPs. However, residents had a lower proportion of cases with meaningful autonomy for specific KIPs, such as flaps and grafts, pediatric/adult airway, and stapedectomy/ossiculoplasty. Larger studies will allow subsets of senior residents to be studied, such as those in their last 3 to 6 months of residency or those going into fellowships versus general practice. The use of a novel assessment strategy like SIMPL to track resident operative experiences facilitates the identification of such areas for possible targeted educational interventions. Such interventions could include changes to case assignments to emphasize specific procedures for certain learners, feedback to faculty on areas in which residents broadly need further experience, and supplemental educational activities in the cadaver or simulation laboratory.

The ability to track operative experiences will be vital to the modernization of surgical training. Currently, residency program directors and accreditation bodies are tasked with overseeing the progression of residents’ surgical skills with limited information about their day-to-day operative experiences. Individual residents also have limited ability to reflect on the longitudinal trajectory of their surgical experiences, making it difficult to set goals and create learning plans. Such information would complement case logs, which are commonly used to evaluate programs yet actually may be inaccurate and uncorrelated with actual competency.^10,11 More detailed assessments, such as Objective Structured Assessments of Technical Skills, are infrequently conducted at programs that use them and may not be directly comparable to the scope of training experiences that SIMPL aims to track. In the field of general surgery, for example, SIMPL is currently used at 68 residency programs, allowing for pooling of data across institutions, which could drive research initiatives and policy decisions.^12,13

By using SIMPL to compare personal assessments with regional and national benchmarks, residents who need remediation may be more readily identified by themselves and their program directors; this is especially important as studies have shown that trainees who perform the worst in multiprofessional and surgical skills assessments are also the least able to identify their weaknesses.^14,15 A recent survey of otolaryngology program directors indicated that while 37.5% of programs have residents who require remediation, ¹⁶ there is currently no consensus on what constitutes an evidence-based remediation strategy. The ability to track assessments over time would allow trainees and program directors to identify specific areas for improvement. Attending surgeons and program directors would additionally receive actionable feedback. For example, attending surgeons may learn that they give significantly less autonomy than other colleagues, or a program may discover that it has a deficit of experience in particular procedures or subspecialties.

In addition to the identification of low-performing outliers, the ability to compare attending and trainee assessments is critical. This study corroborates the existing literature that surgical residents, particularly senior residents, consistently underestimate their own skills as compared with attending surgeons’ assessments,^6,15 which is problematic since self-reflection is critical for personal improvement. Interestingly, the present study found that senior residents also underestimated the difficulty of their cases as compared with paired attending ratings. We suspect that this may be a result of where these residents train, at large academic centers, where the average case complexity is already high. Senior residents are routinely assigned to the most advanced cases and can lose perspective to how complex these cases are relative to the full spectrum of such procedures in the community. The ability to generate reliable self-assessments should be deliberately refined during training, as it has implications for the self-confidence, self-regulation, and long-term skills development of an independent practitioner. Novel educational instruments that longitudinally compare self-assessments with those of trusted surgical mentors are a possible first step toward cultivating this skill.

Conclusion

In this multi-institutional study, senior otolaryngology residents achieved ratings of meaningful surgical autonomy and practice-ready performance in the majority of cases logged for most KIPs. The future development of nationwide benchmarks will help programs and residents set specific educational goals and personalize training plans.