Perceived Impact of USMLE Step 1 Score Reporting to Pass/Fail on Otolaryngology Applicant Selection

Abstract

Objectives:

Recently, the decision was made to transition the United States Medical Licensing Examination (USMLE) Step 1 score from a three-digit numerical score to a pass/fail system. Historically, Step 1 scores have been important for otolaryngology resident applicant selection. The purpose of this study was to understand and evaluate otolaryngology residency program directors’ (OPDs) opinions on the impact following the change in Step 1 score reporting.

Methods:

A 22-question survey administered through Qualtrics was sent to 113 academic otolaryngology residency program directors in April 2020. Information about demographics, impressions on the new Step 1 score format, anticipated changes in applicant selection, impact on mental health, and importance of various other factors in selecting applicants were queried. Descriptive statistics were used to analyze survey results.

Results:

A total of 41 out of 113 (36.3%) OPDs completed this survey. A majority of surveyed OPDs (80.5%) do not support the decision to change Step 1 to a pass/fail system. In the absence of a three digit numeric USMLE Step 1 score, OPDs indicated prioritization of away rotations, letters of recommendation (LORs), personal prior knowledge of the applicant, grades in required clerkship, and class ranking or quartile. 53.7% of OPDs anticipate requiring USMLE Step 2 Clinical Knowledge for interview consideration following this change.

Conclusion:

OPDs believe a pass/fail Step 1 score will decrease the importance of this exam and that this change will lead to the implementation and evaluation of additional metrics such as a required Step 2 CK score.

Keywords

USMLE step 1 residency program directors otolaryngology medical education

Introduction

In the early 1990s, the National Board of Medical Examiners (NBME) and the Federation of State Medical Boards (FSMB) established a three-part United States Medical Licensing Examination (USMLE) to provide individual medical license authorities a comprehensive standardized examination for medical licensure.¹ The USMLE Step 1, the first of the 3 exams, was developed to evaluate and ensure medical students’ knowledge and competency in applying basic scientific principles, diagnoses, management, and evidence-based medicine.¹ This exam is currently reported as a 3-digit numerical score, with the passing requirement of 194.² However, on February 12, 2020, the decision was made to transition the USMLE Step 1 examination from the previous three-digit numerical score to a pass/fail scoring system.³ Though the change will not take effect until 2022, this decision has already generated discussion among program directors (PDs), medical school faculty, and administrators regarding its impact on residency applicant selection.

Otolaryngology has been perceived as a highly competitive specialty selective for medical students with outstanding academic achievement.^4,5 The mean USMLE Step 1 score for matched otolaryngology-head and neck surgery (OHNS) applicants is 248,⁶ while the national mean among US medical students is 230.⁷ The 2020 National Resident Matching Program (NRMP) Match Data revealed that 505 OHNS applicants received interviews for 350 positions, indicating a match rate of 69.3%.⁸ Due to disproportionate number of applicants and available first year residency positions, many otolaryngology residency program directors (OPDs) utilize the 3 digit Step 1 score as a primary filter for selecting applicants to interview.⁹ In fact, 65% of OPDs stated they require students to reach a target score in order to be considered for their program.¹⁰

As the USMLE Step 1 grading system evolves to a pass/fail score, it is important to evaluate its implication on applicant ranking, residency interview offers, and medical student away rotations. The purpose of this study is to survey OPDs in order to guide future medical students interested in applying to OHNS residency programs.

Methods

Survey Population

Following Institutional Review Board (IRB) exemption, an online survey (Qualtrics Experience Management, Seattle, WA, USA) was distributed to query OPDs in April 2020. The National Residency Matching Program (NRMP) and the Association of American Medical Colleges release a PD survey about perceived importance of various factors used to select applicants for interviews. NRMP’s PD survey released in 2018 (prior to the announced Step 1 score reporting change) was used to provide baseline metrics for our current study.⁷

Survey Content

Our online survey included 22 items consisting of open-ended questions, multiple choice questions, and a visual analog scale (VAS) (Supplemental Questionnaire 1). Questions were designed to categorize programs participating in the survey and understand perceived importance of a numerical Step 1 score prior to score change, perceived importance of additional factors used to evaluate applicants in the absence of a numeric Step 1 score, and overall impact on medical students’ wellbeing.

All survey responses remained anonymous and were aggregated for data analysis. Descriptive statistics were utilized to describe demographic information. Our data was compared to results on mean importance of variables such as USMLE Step 1 score and LORs reported from the 2018 PD survey. This baseline was used to mitigate bias from VAS utilization.

Results

Study Cohort

We received a total of 41 (36.3%) completed surveys. A majority of programs (90.2%) were university-based, located in East North Central (19.5%), and offered 2 to 3 post-graduate year (PGY)-1 positions (51.2%) in 2020 (Table 1).

Table 1.

Demographics of Surveyed Otolaryngology Programs.

Program description		Percentage
University-based	37	90.2
Community-based	2	4.9
Community-based/university affiliated	2	4.9
Region
New England	4	9.8
Mid Atlantic	5	12.2
East North Central	8	19.5
West North Central	3	7.3
South Atlantic	6	14.6
South West	4	9.8
East South Central	3	7.3
Rocky Mountain Region	1	2.4
West Pacific	7	17.1
Total PGY-1* positions
1 position	4	9.8
2-3 positions	21	51.2
4+ positions	16	39.0

PGY- Post Graduate Year; New England- CT, MA,ME,NH,RI,VT; Mid Atlantic- NJ,NY,PA; East North Central-IL,IN,MI,OH,WI; West North Central- IA,KS,MN,MO,ND,NE,SD; South Atlantic- DC,DE,FL,GA,MD,NC,SC,VA,WV; South West- TX,AZ,OK,NM; East South Central- AL,KY,MS,TN,AR,LA,PR; Rocky Mountain Region- MT,ID,WY,UT,CO,NV; West Pacific- WA,OR,CA,HI,AK.

Importance of Specific Factors

In the absence of a three-digit numeric USMLE Step 1 score, the average importance of USMLE Step 1 dropped from the previously reported 4.1 in the 2018 NRMP PD survey to 1.63, a 36.3% decrease in relative importance (Table 2).⁷ Due to the proposed change in scoring, the following factors were predicted to be most important: prior failed attempt in USMLE/COMLEX (4.4), away rotations (4.1), LORs (3.9), personal prior knowledge of the applicant (3.9), grades in required clerkship (3.9), and class ranking or quartile (3.9) (Table 2). Class ranking or quartile (+15.4%) and USMLE Step 2 CK (+13.4%) had the highest increase in relative importance (Table 2).

Table 2.

Mean Importance Rating* for Each Factor in Selecting Applicants to Interview.

Factors	Post-step 1 score change to pass/fail	Change in relative importance from 2018 NRMP Program Director Survey (%)
USMLE Step 1	1.6	−36.6
Letters of recommendation in the specialty	3.9	1.4
Medical student performance evaluation (MSPE/Dean’s letter)	3.1	−0.6
USMLE Step 2 CK/COMLEX level 2 CE score	3.7	13.4
Personal statement	3.0	−0.6
Grades in required clerkships	3.9	7.4
Any failed attempt in USMLE/COMLEX	4.4	7.4
Class ranking/quartile	3.9	15.4
Perceived commitment to specialty	3.8	3.4
Personal prior knowledge of the applicant	3.9	−0.6
Grades in clerkship in desired specialty	3.8	3.4
Alpha Omega Alpha (AOA) membership	3.0	−4.6
Leadership qualities	3.7	9.4
Demonstrated involvement and interest in research	3.7	7.4
Interest in academic career	2.7	−14.6
USMLE Step 2 clinical skills	2.6	−0.6
Graduate of highly regarded U.S. medical school	2.8	−0.6
Away rotation	4.1	1.4
Gold humanism award	2.6	−10.6

Ratings on a scale from 1 (not at all important) to 5 (very important).

USMLE Step 1 and Step 2 CK

Of our survey responders, 34.1% currently use an applicant’s numerical Step 1 score alone as a screening tool prior to offering an interview (Table 3). 24 OPDs (58.5%) consider the 3-digit numerical score for USMLE Step 1 to be either extremely or very important when selecting applicants for interviews, but with the proposed change in scoring system, only 4 OPDs considered the Step 1 score to be very or extremely important (Table 3). A majority of OPDs (80.5%, n = 33) did not support the decision to change Step 1 to a pass/fail system and 53.7% of PDs (n = 22) anticipate requiring USMLE Step 2 CK for interview consideration (Table 3).

Table 3.

Program Director Impressions on USMLE Step 1 and Step 2 Clinical Knowledge.

Question	Not at all important		Slightly important		Moderately important		Very important		Extremely important
Question	n	%	n	%	n	%	n	%	n	%
Importance of USMLE Step 1 for selecting interview applicants when reported as a three-digit score	0	0.0	7	17.1	10	24.4	11	26.8	13	31.7
Anticipated importance of USMLE Step 1 for selecting interview applicants when reported as pass/fail	11	26.8	21	51.2	5	12.2	2	4.9	2	4.9
	USMLE Step 1 alone		USMLE Step 1 and 2
	n	%	n	%
Which USMLE standardized exams did your program utilize in order to select your applicants?	14	34.2	27	65.9
	Definitely not		Probably not		Might or might not		Probably yes		Definitely yes
	n	%	n	%	n	%	n	%	n	%
Anticipated potential for requiring Step 2 CK for interview	2	4.9	6	14.6	11	26.8	12	29.3	10	24.4
Do you support this decision to make USMLE Step 1 pass/fail?	21	51.2	12	29.3	1	2.4	4	9.8	3	7.3

Impact of Step 1 Score Change to Pass/Fail on Applicants

In response to whether the score change would impact a program’s ability to recruit competitive applications, most OPDs responded “might or might not” (63.4%). Answer choices, “might or might not” and “probably not” were most commonly chosen in response to anticipated improvement in medical student well-being. About 31.7% of OPDs believe the new change would not adversely or positively impact applicant diversity and students from lower socioeconomic backgrounds. About 68.3% of OPDs suggested that they may be less likely to interview applicants from “lower ranked” medical schools and international medical graduate students (Table 4).

Table 4.

Impact of Change in USMLE Step 1 Grading on Applicant Selection.

Question	Definitely not		Probably not		Might or might not		Probably yes		Definitely yes
Question	n	%	n	%	n	%	n	%	n	%
Ability to recruit competitive applicants	3	7.3	8	19.5	26	63.4	13	31.71	4	9.8
Improve diversity of applicants	4	9.8	11	26.8	13	31.7	6	14.63	7	17.1
Place applicants from “lower ranked” school at a disadvantage for interview selection	0	0.0	6	14.6	7	17.1	18	43.90	10	24.4
Place applicants from international graduate schools at a disadvantage for interview selection	2	4.9	2	4.9	10	24.4	16	39.02	11	26.8
Place applicants from lower socioeconomic status at a disadvantage for interview selection	3	7.5	13	32.5	15	37.5	5	12.50	4	10.0
Improve mental health or wellness in medical school	6	15.0	13	32.5	13	32.5	4	10.00	3	7.5

Overall OPD Impressions

Most (70.7%) OPDs responded they would not include additional requirements due to changes in score reporting. OPDs that answered “yes” anticipated adding requirements such as personal statements or supplemental questions for students applying to their program. About 61% of OPDs stated this change would not impact selection of medical students for away rotations.

Discussion

Historically, otolaryngology is considered to be a competitive match.⁶ Our analysis reveals that the transition to a pass/fail scoring system for USMLE Step 1 will significantly impact the otolaryngology resident selection process. Prior to the announcement made by NBME and FSMB, residency OPDs ranked Step 1 as an important determinant for interview selection. With the proposed change, our survey reveals that Step 1 significantly decreases in importance. In lieu of a USMLE Step 1 score, any failed attempt in USMLE/COMLEX, away rotations, LORs, personal prior knowledge of the applicant, grades in required clerkship, and class ranking/quartile were considered to be most important when selecting an applicant. This study helps prospective applicants understand and predict the shift in emphasis from USMLE Step 1 to other aspects of the application.

Over the past 20 years, the mean number of otolaryngology residency applications submitted by each medical student has increased by over 250%.¹¹ To mitigate the overwhelming number of applications, OPDs have used USMLE step 1 score as a filter.⁴ Data released by the Association of American Medical Colleges (AAMC) demonstrates a correlation between high USMLE step 1 scores and percent match rate.¹² Medical students with a Step 1 score less than 239 had a 43% otolaryngology residency program entrance rate, while students with a score range of 239-250 or a score above 250 had a 73% and 81% entrance rate, respectively.¹² Puscas et al¹³ and Hauser et al,¹⁴ both demonstrate that a high USMLE score is an independent predictor for receiving additional interviews and matching into an otolaryngology residency program.

With these expected changes in the USMLE reporting system, USMLE Step 2 is anticipated to increase in relative importance. About 53% of surveyed OPDs were inclined to require a USMLE Step 2 CK score prior to completing the ERAS application deadline. However, in previous years, 34.1% of OPDs evaluated applications based on the applicant’s Step 1 score alone, regardless of a reported a Step 2 score. By shifting the importance from USMLE Step 1 to Step 2 CK scores inevitably prolongs applicant anxiety. Though the decision to change the scoring system was intended to decrease emotional anxiety around examinations, many OPDs believe that it simply increases the weight on a single exam, rather than two.

In the absence of a numerical score, our results indicate that away rotations will be prioritized for interview selection. Prior knowledge of the applicant was also found to be an important determinant. These results coincide with the data presented by the 2018 NRMP PD survey and the study performed by Hauser et al^10,14 Indeed, several osteopathic otolaryngology programs only invited applicants who completed away rotations at their program for interviews.¹⁵ In 2019, otomatch.com, an online forum for medical students applying to Otolaryngology, revealed 30.3% of applicants matched into a program where they performed an away rotation, while 21.4% of applicants matched into their home program.¹⁶ With the increasing emphasis placed on away rotations as a result of the expected changes in the scoring system, students may feel pressured to apply to and participate in more programs. Further, due to due to unforeseen circumstances such as the COVID-19 pandemic, future applicants may not have the opportunity to participate in away rotations, and may be at an even greater disadvantage without a numerical Step 1 score.

With the transition of the USMLE Step 1 examination to a pass/fail model, OPDs will prioritize grades in required clerkships and class ranking or quartile. Several studies have previously highlighted that medical students inducted into Alpha Omega Alpha, a society recognizing students for their high academic performance, have an increased odds of matching into otolaryngology.^14,17,18 In addition, with these upcoming changes, OPDs will also place an emphasis on LORs for applicants. Personal knowledge of the letter writer or quality of LOR was also found to be a determinant for applicant selection.

In the present study, 41.5% residency OPDs believe that changes in score reporting will negatively impact the quality of residents. Interestingly, literature has shown no correlation between USMLE Step 1 performance and surgical dexterity/skill in otolaryngology applicants.¹⁹ This present study also revealed that OPDs believed that removal of a three-digit numerical score may negatively impact student wellbeing, negating the FSMB and NBME’s original purpose for shifting to pass/fail reporting for Step 1. Few OPDs also mentioned the future implementation of a personal statement specific to their program to differentiate applicants and to mitigate becoming overburdened with applications to review. This requirement was previously enforced during the 2015 to 2016 otolaryngology application cycle. Coinciding with the addition of this requirement, the number of otolaryngology residency applicants decreased by nearly 20%.²⁰ Thus, the implications of removing a three-digit numerical scoring system extend far beyond the impact of the examination itself but rather affect multiple aspects of the otolaryngology application process.

There are several limitations to this study. Aspects of the survey instrument were developed through an informal literature review and the complete survey did not undergo validation prior to administration. It is important to note that our reported results are predicated on predictions by OPDs, and are not supported by actual match results. In our questionnaire, we did not define “diversity” or “lower ranked medical school.” Additionally, the survey response rate was 36.3%, indicating that these results only represent a minority of OPDs. Despite this limited representation, the response rate presented in this study is equivalent to the 2018 NRMP Match PD survey data response rate (36.1%).⁹ Future prospective studies include investigating the effects of pass/fail step 1 scores on applicant selection and matched applicants.

Conclusion

A majority of otolaryngology program directors do not support the decision to transition USMLE Step 1 score reporting from a numerical score to pass/fail grading. In lieu of a numeric Step 1 score, program directors will prioritize any failed attempt in USMLE/COMLEX, away rotations, letters of recommendation, personal prior knowledge of the applicant, grades in required clerkship, and class ranking or quartile.

Supplemental Material

sj-pdf-1-aop-10.1177_00034894211028436 – Supplemental material for Perceived Impact of USMLE Step 1 Score Reporting to Pass/Fail on Otolaryngology Applicant Selection

Supplemental material, sj-pdf-1-aop-10.1177_00034894211028436 for Perceived Impact of USMLE Step 1 Score Reporting to Pass/Fail on Otolaryngology Applicant Selection by Ishwarya S. Mamidi, Alex Gu, Collin F. Mulcahy, Chapman Wei and Philip E. Zapanta in Annals of Otology, Rhinology & Laryngology

Footnotes

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) received no financial support for the research, authorship, and/or publication of this article.

Institutional Ethics Approval/Exemption

This study was approved by the George Washington University Institutional Review Board.

ORCID iDs

Ishwarya S. Mamidi

Collin F. Mulcahy

Philip E. Zapanta

Supplemental Material

Supplemental material for this article is available online.

References

United States Medical Licensing Examination. Step 1. Accessed May 3, 2020. https://www.usmle.org/step-1/

United States Medical Licensing Examination. Scores & Transcripts. Accessed May 3, 2020. https://www.usmle.org/transcripts/

United States Medical Licensing Examination. Change to pass/fail score reporting for Step 1. Accessed March 17, 2020. https://www.usmle.org/incus/#decision

Kaplan

Riedy

Grundfast

KM.

Increasing competitiveness for an otolaryngology residency: where we are and Wang et al 7 concerns about the future. Otolaryngol Head Neck Surg. 2015;153(5):699-701.

Johnson

Svider

Folbe

, et al An evaluation of geographic trends in the otolaryngology residency match: home is where the heart is. JAMA Otolaryngol Head Neck Surg. 2015;141(5):424-428.

National Resident Matching Program. Charting outcomes in the match: U.S. allopathic Seniors. 2018. Accessed May 2, 2020. https://www.nrmp.org/wp-content/uploads/2018/06/Charting-Outcomes-in-the-Match-2018-Seniors.pdf

USMLE. USMLE score interpretation guidelines. Published October 14, 2020. Accessed May 3, 2020. https://www.usmle.org/pdfs/transcripts/USMLE_Step_Examination_Score_Interpretation_Guidelines.pdf

The match main residency match data and reports. Accessed May 7, 2020. http://www.nrmp.org/main-residency-match-data

Prober

Kolars

First

Melnick

DE.

A plea to reassess the role of United States Medical Licensing Examination Step 1 scores in residency selection. Acad Med. 2016;91(1):12-15. doi:10.1097/acm.0000000000000855

10.

National Resident Matching Program, Data Release and Research Committee. Results of the 2018 NRMP Program Director Survey. Published 2018. Accessed May 2, 2020. https://www.nrmp.org/wp-content/uploads/2018/07/NRMP-2018-Program-Director-Survey-for-WWW.pdf

11.

Naclerio

Pinto

Baroody

FM.

Drowning in applications for residency training: a program’s perspective and simple solutions. JAMA Otolaryngol Head Neck Surg. 2014;140(8):695-696.

12.

Apply smart: data to consider when applying to residency. Accessed May 3, 2020. https://students-residents.aamc.org/applying-residency/filteredresult/apply-smart-data-consider-when-applying-residency/

13.

Puscas

Sharp

Schwab

Lee

WT.

Qualities of residency applicants: comparison of otolaryngology program criteria with applicant expectations. Arch Otolaryngol Head Neck Surg. 2012;138(1):10-14.

14.

Hauser

Gebhard

Blumhagen

Carlson

Cabrera-Muffly

Applicant characteristics associated with successful matching into otolaryngology. Laryngoscope. 2017;127(5):1052-1057. doi:10.1002/lary.26236

15.

AOA. Approved otolaryngology internships and residencies. http://opportunities.osteopathic.org/search/search.cfm

16.

Otomatch. Accessed May 2, 2020. http://www.otomatch.com/

17.

Rinard

Mahabir

RC.

Successfully matching into surgical subspecialties: an analysis of national resident matching program data. J Grad Med Educ. 2010;2(3):316-321.

18.

Bowe

Schmalbach

Laury

AM.

The state of the otolaryngology match: a review of applicant trends, “impossible” qualifications, and implications. Otolaryngol Head Neck Surg. 2017;156(6):985-990. doi:10.1177/0194599817695804

19.

Carlson

Archibald

Sorom

Moore

EJ.

Under the microscope: assessing surgical aptitude of otolaryngology residency applicants. Laryngoscope. 2010;120(6):1109-1113. doi:10.1002/lary.20914

20.

Stepan

Kaul

Raquib

, et al An evaluation of the program-specific paragraph in the otolaryngology residency application. Laryngoscope. 2018;128:2508-2513. doi:10.1002/lary.27553

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.08 MB