Abstract
Background
Virtual interviews (VIs) for the 2020 residency application season were mandated as a result of the COVID-19 pandemic. We aimed to determine the perspectives of general surgery (GS) program directors (PDs) on the benefits and drawbacks of VIs.
Methods
A 14-item survey was emailed to all GS PDs from programs identified on the American Council for Graduate Medical Education website. Program directors were asked about the cost-time benefit of VIs, its ability to assess candidates, and their thoughts on the future of VIs for evaluating residency applicants.
Results
60 PDs responded corresponding to a response rate of 21%. While 93% agreed/strongly agreed that VIs were less expensive, only 35% found VIs to be less time-consuming. 75% and 67%, respectively, disagreed/strongly disagreed that VIs allowed for an easier assessment of an applicant’s fit, and personality and communication skills. Almost one-half of our survey respondents suggested that VIs made the selection committee rely more heavily on objective applicant data. Almost two-thirds of GS PDs suggested that they would adopt both VI and in-person interview formats for future application cycles. The median [interquartile range] cost saved through the implementation of VIs was US$ 4500 [1625 – 10 000].
Conclusion
Remarkably, VIs have been swiftly imbibed by all residency programs and many aspects of the VI experience were positive. While MATCH 2021 has definitely proved to be one of its kind, the implementation of VIs has been met with overall broad success and a promising future awaits this novel modality of resident selection to GME programs in the United States.
Introduction
Residency interviews and in-person program visits have been a time-honored and vital component of resident selection into graduate medical education (GME) programs in the United States (US). 1 Among applicants to general surgery (GS) residency, quality of life of current residents and anticipated clinical experience have been found to be major determinants influencing applicant choice of a program. 2 Traditionally, the understanding of the program culture, fit with future co-residents, and life in the city where the matched applicant would reside for the next several years, has been accomplished through pre-interview social gatherings at local restaurants, followed by a formal interview and tour of the institution.
The residency application process for MATCH 2021, however, has witnessed a dramatic change in multiple facets of applicant selection. Owing to the social distancing requirements and restrictions imposed on travel stemming from the COVID-19 pandemic, the Association of American Medical Colleges (AAMC) had called for a move to a virtual format for residency interviews in early 2020.3-6 Although some fellowship programs employed virtual interviews (VIs) during the previous application cycle, this format is still in its early stages of inception in medicine and had not been implemented on a wide basis.7-9
As the application season for MATCH 2021 draws to a close, questions loom regarding the feasibility of the virtual interviews for future cycles. Though one cannot turn a blind eye to the obvious benefits of VIs to applicants including saved time and cost, it is imperative to determine if this format allows for thoroughly evaluating applicants, especially considering the high attrition rate from general surgery residencies. 10 Against this background, we sought to analyze the pros and cons of VIs and their practicability for continued utilization in the years to come.
Methods
A 14-question survey was emailed to all program directors (PDs) of American Council for Graduate Medical Education (ACGME)-accredited GS residency programs. The GS residency programs included in our study were identified using the ACGME website. Email addresses were collected from the ACGME and residency program websites.
The survey was designed based on review of existing literature, group discussions, and further refined based on senior author input (the senior author is a female surgeon who has completed MD, followed by residency training in general surgery and fellowship training in plastic surgery, and currently serves as a program director of an integrated plastic surgery residency at a large academic institution in Texas). Program directors were asked about the number of spots offered during the 2020-2021 residency application cycle, the convenience of VIs from a cost and time standpoint, and the ability to assess residency candidates over VIs. Finally, PDs were asked about their overall experience with VIs and whether they would continue to use the virtual format in coming interview cycles. Free-text feedback allowed for respondents to list aspects of their experience that were not included among the survey questions.
The study was deemed exempt by the Institutional Review Board, and emails were sent with the survey. Participants received an individualized survey link; responses were anonymous, and participation was voluntary. Redcap was used as the survey platform.
Continuous data were summarized as median and interquartile range for the 25th and 75th percentile. Categorical data were summarized as percentages and proportions and comparisons were made using the chi-square test. A p-value of <.05 was considered statistically significant. Statistical analysis was performed using JMP Pro 14 software (JMP, Pro 14, SAS Institute Inc, Cary, NC, 1989-2019)
This work has been reported in line with the Consolidated Criteria for Reporting Qualitative Research (COREQ).
Results
First Part of the Survey Administered (Likert-Type Responses).
Second Part of the Survey Administered (Non–Likert-Type Responses).
¥n = 36.
While 93% of our respondents agreed/strongly agreed that VIs were less expensive, only 35% found that they were less time-consuming. 75% and 67%, respectively, disagreed/strongly disagreed that VIs allowed for an easier assessment of an applicant’s fit, and personality and communication skills. Almost one-half of our survey respondents suggested that the introduction of VIs made the selection committee rely more heavily on objective applicant data and a similar mixed response was obtained when querying if VIs were overall better than in-person interviews.
26% of PDs mentioned that attendance at the virtual social sessions held for applicants to gain insight into the residency program culture was considered while selecting candidates to interview and rank, and almost two-thirds of GS PDs suggested that they would adopt both interview formats for future application cycles. The median [IQR] cost saved through the implementation of VIs was US$ 4500 [1625 – 10 000].
Discussion
While recent literature discusses VIs in the setting of the fellowship match, we present the first study evaluating the perspectives of program directors in GS regarding the benefits, disadvantages, and future applicability of VIs in resident selection. The responses received from GS program directors unveiled interesting findings: VIs were found to save cost, without much consensus on saved time. VIs were not found to be a convenient means of assessing an applicant’s fit and commitment, with resulting increase in reliance on objective applicant data. Only a minority of PDs (26%) reported considering attendance at virtual social events in selecting applicants to interview and rank, and 65% suggested that they would offer both VIs and in-person interviews for future application cycles.
Selecting GS residents for training requires meticulous attention to detail in light of the well-documented high attrition rates from this specialty. A meta-analysis by Khoushhal et al 10 in 2017 documented a pooled attrition prevalence of 18% with “uncontrollable lifestyle” of the residents being mentioned as the number one cause for attrition. Questions asked by both interviewers and interviewees often provide some insight into a candidate’s understanding of the GS resident’s lifestyle. Likewise, subtle body language cues observed during in-person interactions afford hints on an applicant’s personality and priorities. The VI setting compounds the difficulty of such an assessment and may contribute to future attrition from GS residency programs.
While a majority of respondents to our survey agreed that VIs saved cost with an average saved expense of US$6462 per program, there was a mixed response as to whether VIs were less time-consuming. The reduction in cost is significant for both interviewees and for host programs. Previous studies based on fellowship interviews have found that majority of GS residents spent over US$4000 per applicant for this purpose. 11 Given the larger proportion of applicants to GS residency compared to individual fellowships, the net reduction in expense by the adoption of VIs is remarkable. Likewise, Gardner et al 12 found that the average hard costs to a GS residency program conducting live interviews was around US$8000—a considerable sum that could be otherwise allocated for different aspects of resident education. In terms of time spent in organizing and conducting VIs, it is understandable that this format was not yet found to be more efficient compared to in-person interviews. The lack of familiarity with the VI platform and the possible need for technical support as a result of unanticipated interruptions could be among the reasons for this finding. A test run of the VI interface with interviewing faculty prior to the interview day has been reported to be a feasible option for facilitating a seamless interview experience. 13
It is not surprising that only a minority of GS PDs reported that VIs simplified assessment of applicant fit, personality, communication skills, and commitment to the specialty. Without prior experience, observing body language cues and expressions portrayed by applicants is a challenge in the virtual setting. Furthermore, technical interruptions, disturbances in video quality, and applicant background can be distracting and may introduce a selection bias. Prior to the commencement of the interview season, concerns had been expressed by PDs when the special committee appointed by the American College of Surgeons Division of Education published their suggested strategies for programs and candidates to navigate VIs. The authors reported that PDs highlighted the inability to accurately observe interpersonal interactions between applicants and their interviewers and endorsed faculty inexperience with virtual assessment of applicant intangibles. 14 It is, however, important to note that experience thus far with VIs has been limited and institutional experience is varied. Vining et al 13 surveyed faculty interviewers for fellowship interviews to the complex general surgical oncology (CGSO) program at the University of Chicago and found that the faculty unanimously agreed that applicants were able to convey themselves “well” or “very well” through VIs. On the contrary, a survey of PDs from the 2020 CGSO MATCH revealed that only 60% of PDs agreed/strongly agreed that VIs allowed for accurate representation of the candidate, but about 83% reported feeling comfortable in creating a rank list following the interview. 15 The discordance between PD and faculty perceptions may be related to the differences in emphasis placed upon specific applicant characteristics and different evaluation patterns adopted by the 2 cohorts while conducting fellowship interviews. While faculty interviewers often tend to evaluate candidates based on their clinical training and as future colleagues, PDs are tasked with stratifying a large number of qualified candidates and comparing how their goals align with that of the training program. The onus to determine the best program to fellow “fit” lies on the PD and this could necessitate greater attention to minor differences and candidate self-expression.
Almost one-half of GS PDs in our survey suggested that the utilization of VIs resulted in a greater reliance on objective applicant metrics for resident selection. An additional possible explanation for this response (other than the decreased ability to thoroughly assess an applicant as previously stated) may be related to the limited feedback from residents based on the pre-interview social in a virtual setting. While senior residents often participate in formal applicant interviews, social etiquette observed during in-person gatherings offers unique perspectives as to how an applicant would blend with the resident cohort, often serving as a surrogate of an applicant’s “fit” with the program. Some programs, particularly several of the plastic surgery residencies, organized virtual social hours between residents and prospective applicants/interviewees prior to and after extending interviews, and these serve to at least partially mitigate the lack of an in-person social interaction. 7 Furthermore, audition rotations are undertaken by applicants eyeing specific programs to showcase their interest and allows programs to see prospective interns work in their own clinical setting.6,16 However, the substitution of these audition rotations with virtual rotations owing to travel restrictions has posed yet another challenge.6,17 Against this setting, PDs are more likely to scrutinize the objective data including scores on the medical licensing exams, letters of recommendations, and research experience among others to stratify applicants and interviewees.
Are Virtual Interviews Here to Stay?
While the benefits of VIs particularly from the applicant standpoint are appealing, consideration for the continued adoption of this format for future application cycles deserves careful deliberation. Most GS PDs in our study did not find VIs to be better than in-person interviews and only a small proportion (8%) suggested that they would offer only VIs in future. There is a lack of consensus overall on the preference for VIs as a supplement or replacement of traditional in-person interviews. This also varies between applicants and residency programs. Hill et al 15 in their PD survey of the CGSO match found that 60% of their respondents would perform VIs if given the option for the next cycle. Among medical students, a large cross-sectional survey that received 1711 medical student responses found most of their participants agreeing with programs offering both options: in-person and VIs, and that travel distance to the program had a significant impact on favoring virtual interviews. 18
While we did not ask our survey respondents on the logistics of implementing both VIs and in-person interviews in the future application cycles, a couple of possibilities can be contemplated. First, programs may offer VIs as part of a screening process for a larger applicant pool and then short-list their highly desired candidates for an in-person visit. Alternatively, VIs could be offered exclusive of in-person interviews based on either applicant choice or program evaluation of an applicant. This could, however, lead to potential selection bias. As time will eventually dictate the future of VIs for resident selection, the option of easy scheduling, relative flexibility, decreased need for time off from clinical activity, and a marked drop in interview expense for the average medical student with an estimated US$200,000 debt are important factors that offer credence to strong future consideration for VIs. 19
Comments From GS PDs
Our study allowed respondents to vocalize their thoughts on additional aspects of VIs that had not been included among the survey items. Mixed responses supporting VIs and advocating for a return to in-person interviews for the next cycle were obtained. Selected commentaries are cited below: “I think we need to continue the option for virtual and not be biased against those who choose it. I believe we had access to a more diverse applicant pool…” “We liked the virtual format and plan to use this in the future. The technical glitches occurred only in the first session…The glitches involved the breakout rooms and the timer clock which shut the sessions out on a defined timer. Once the faculty got accustomed to watching the time close and to our setting up more time for the breakouts, this worked well. This also kept us on-time for the interviews with the ability to have the mandatory shut out electronically.” “There was significantly less interaction between the interviewees and the residents, which has significantly hampered our ability to assess the interviewees and for them to assess our program. How many people would take a five-year job at a place they have never visited based on a few virtual interviews and reviewing a website? How many people will take a job after residency based only on a virtual interview and the review of a website? We are looking forward to in-person interviews next cycle and are concerned that there will be a push to keep the virtual interview process.”
“While virtual may be OK for an initial round, I feel true assessment of individual and their assessment of program requires in-person interaction. I feel there were also fewer interview options open to middle, but historically successful match candidates; however, this year a large percentage of interview spots of has been taken up by a smaller group of applicants.”
Limitations
While we are the first to present a survey reflecting the thoughts of the cohort of GS PDs on VIs, we acknowledge several notable limitations inherent to the survey-based study design, including a limited survey response rate. We did not assess and differentiate responses based on the geographic location, and type of residency (community vs academic), which could have implications on resources available to organize and support VIs. While we sought to focus on surveying PDs in GS, similar questions directed to residency applicants would help unveil differences in perspectives between the two groups and could be a potential focus of future studies.
Conclusions
Virtual interviews are a new and forced adaptation resulting from the COVID-19 pandemic. It is impressive that the new format was quickly imbibed by all residency programs and many aspects of the VI experience were positive. Virtual interviews provide cost savings but are associated with ongoing concerns about the ability to identify applicants that might be a good fit. The surgical education community will need to monitor resident recruitment to identify best practices for conducting VIs and figuring out how to best re-incorporate in-person interviews back into the process.
Footnotes
Author Contributions
AR, MA – Data analysis, draft of preliminary manuscript, revision based on co-author input, approval of final versionRE, AMF – Data collection, critical review of manuscript with revision for incorporating intellectual content, approval of final versionRSM – Principal investigator, design/conception of project, review of manuscript with critical revisions for incorporating intellectual content, approval of final version
Declaration of Conflicting Interests
The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The author(s) received no financial support for the research, authorship, and/or publication of this article.