Associations between Otolaryngology Applicant Characteristics and Future Performance in Residency or Practice: A Systematic Review

Abstract

Objective

This systematic review aims to evaluate which applicant characteristics available to an otolaryngology selection committee are associated with future performance in residency or practice.

Data Sources

PubMed, Scopus, ERIC, Health Business, Psychology and Behavioral Sciences Collection, and SocINDEX.

Review Methods

Study eligibility was performed by 2 independent investigators in accordance with the PRISMA protocol (Preferred Reporting Items for Systematic Reviews and Meta-analyses). Data obtained from each article included research questions, study design, predictors, outcomes, statistical analysis, and results/findings. Study bias was assessed with the Quality in Prognosis Studies tool.

Results

The initial search identified 439 abstracts. Six articles fulfilled all inclusion and exclusion criteria. All studies were retrospective cohort studies (level 4). Overall, the studies yielded relatively few criteria that correlated with residency success, with generally conflicting results. Most studies were found to have a high risk of bias.

Conclusion

Previous resident selection research has lacked a theoretical background, thus predisposing this work to inconsistent results and high risk of bias. The included studies provide historical insight into the predictors and criteria (eg, outcomes) previously deemed pertinent by the otolaryngology field. Additional research is needed, possibly integrating aspects of personnel selection, to engage in an evidence-based approach to identify highly qualified candidates who will succeed as future otolaryngologists.

Keywords

otolaryngology match selection ranking applicant medical student resident faculty predictor assessment test procedure outcome performance

Given the cognitive homogeneity of most medical school applicants, it no longer suffices to select those intellectually capable of completing their medical education. Almost all are capable. The prime objective of medical education should be to select and train those who will perform best as physicians. Consideration of non-cognitive criteria appears to be essential to further this objective.¹

In 1979, Keck and colleagues noted that a combination of cognitive variables (eg, board scores and medical school grades) and noncognitive variables (eg, personality-based inventories) functioned much better than any individual variable when predicting postgraduate clinical performance. Despite this knowledge, prior academic achievement has generally been and continues to be the primary basis for residency selection, and it is often used during the initial screening stage of the application process.² Approximately 30 years after Keck’s work, members of the ophthalmology community reviewed the traditional resident selection process in their field and noted that it was highly dependent on cognitive metrics, yet these metrics generally assessed only the medical knowledge domain from the competencies of the Accreditation Council for Graduate Medical Education. Additionally, they reported that the current selection process was relatively inadequate for evaluation of noncognitive domains and did not accurately predict future residency performance. Therefore, they recommended performing further research to correlate cognitive and noncognitive variables available at the time of application with measures of resident performance.³

In recent years, other highly competitive specialties have sought to improve their residency selection practices. It has been suggested that competitive specialties, such as dermatology, are faced with greater challenges in resident selection due to the increased number of applicants with outstanding academic records.⁴ Yet, unfortunately, most criteria available at the time of application have shown very inconsistent correlation with regard to predicting residency success.⁴ Similarly, in response to the overwhelming number of applications per position, the American Orthopedic Association Steering Committee on Resident Selection specified 3 focus areas that may be predictive of resident performance: psychomotor abilities, cognitive skills, and the affective domain.⁵ Specifically, the affective domain includes the applicant’s personal and professional values, such as integrity, trustworthiness, responsibility, reliability, professionalism, and communication skills. While deemed an important predictor, the authors noted the lack of objective evaluative tools to assess the qualities of the affective domain.⁵

There is clearly a burgeoning interest in improving resident selection across all specialties,^6-8 with a more specialty-specific focus noted in the highly competitive fields.^3-5 A recent systematic review of medical selection across all specialties revealed “a picture of quantity over quality,” suggesting the existence of several studies on the subject that present only moderate quality data at best.⁸ The major criticism leading to the low quality of medical selection research has been the distinct lack of theory-driven studies that examine issues related to validity and the constructs being measured.⁸ Therefore, the goal of this systematic review is to evaluate which applicant characteristics available to an otolaryngology selection committee are associated with future performance in residency or practice.

Methods

This systematic review was conducted in the manner specified in the PRISMA guidelines (Preferred Reporting Items for Systematic Reviews and Meta-analyses).⁹ The PRISMA flowchart can be found in Figure 1 . This study was exempt from evaluation by the Institutional Review Board.

Figure 1.

PRISMA flow diagram describing article screening and inclusion.

Search Strategy

With the assistance of a medical librarian, a systematic search was performed with PubMed, Scopus, ERIC, Health Business, Psychology and Behavioral Sciences Collection, and SocINDEX up to December 15, 2015. A search syntax was designed with synonyms for any terms that may relate to otolaryngology selection and performance ( Figure 2 ).

Figure 2.

Search syntax.

Study Eligibility

Study selection was performed in accordance with the PRISMA protocol ( Figure 1 ). The initial literature search yielded 439 studies for screening, of which 95 were excluded as duplicates. The remaining 344 studies were screened by 2 independent reviewers (S.N.B. and A.M.L.) to determine if studies met the established inclusion criteria—specifically, each study had to include otolaryngology residency applicants, evaluate a measurable predictor of residency success and a measurable outcome to determine that success, and correlate a relationship between them. Seven full-text articles were eligible for evaluation. The references of these studies were analyzed to identify any additional articles that could be relevant for this review. Upon examination, 1 article was noted to be a review and therefore was excluded from qualitative analysis. In total, 6 studies underwent data abstraction, assessment, and analysis.

Data Abstraction and Quality Assessment

Information obtained from each article included title, year of publication, years that the study spanned, timing of assessment, research questions, study design, sample size, predictors, outcomes, statistical analysis, and results/findings. To assess for study bias, each study was further evaluated with the Quality in Prognosis Studies tool to assess risk of bias in studies of prognostic factors.¹⁰ This tool evaluates 6 areas to assess a prognostic study’s validity and bias: study participation, study attrition, prognostic factor measurement, outcome measurement, study confounding, and statistical analysis and reporting.

Results

The literature search identified only 6 articles that fulfilled all inclusion and exclusion criteria.^11-16 Table 1 presents a summary of the key characteristics of each study (for an unabridged version of Table 1 , see Appendix A, available in the online version of the article).

Table 1.

Key Characteristics from Included Studies.

Study: First Author, Years (n)	Predictors	Outcomes	Results
Calhoun,¹² 1986-1997 (175)	Medical school GPA Class rank percentile Academic “supervariable”^a AOA Research during medical school Surgery/medicine honors	CPR global, technical, + interpersonal skills (range, 3 = best to 18 = worst) COR program director’s opinion of resident as an alumna or colleague (range, 2 = best to 7 = worst) Overall ranking (excellent, average, or potential problem)	Students with more favorable CPR were more likely to have fulfilled the criteria for the academic supervariable Honors on core surgery rotation correlated with excellent performance in residency
Daly,¹³ 1983-1993 (36)	Awards (DR) Research (DR) In-service scores (DR) Interviewers (%) favoring selection (>50% vs <50%) USMLE Step 1 score (>570 vs <570) Exceptional trait in LOR AOA	Academic appointment between 1993-1997 Clinical faculty ranking (continuous rank + highest tertile)	Exceptional trait in LOR + AOA related to highest tertile of faculty ranking AOA related to current academic appointment USMLE Step 1 score related to year 2 in-service score Years 3 + 4 in-service scores correlated with faculty rank Interviewers (%) favoring selection + higher USMLE Step 1 score related to peer publications >1
Bent¹⁶ 2001-2008 (32)	NRMP rank number + group	Faculty + resident evaluations (means) Chief resident selection Graduating resident teaching award ABOto Board Pass	No significant correlation between NRMP + any outcome measure
Chole,¹¹ 2001-2010 (46)	USMLE Step 1 score AOA Honors (%) in clinical rotations LOR (3-point scale) Quality of medical school Residency interview (7-point scale)	Faculty rating	No correlation with USMLE Step 1 score, AOA, honors (%), LOR, clinical rotations, musical training Weak correlation with residency interview score + quality of medical school Moderate correlation with excellence in a team sport
Tang,¹⁴ 1990-2006 (62)	3 best + 3 worst soap carvings	Cognitive knowledge Visuospatial ability Manual dexterity Decision making/poise Top 50% vs bottom 50% of residents	No relationship between quality of soap carving + class ranking or resident qualities
Moore,¹⁵ 2007-2012 (24)	Composite microvascular score Attitudinal score	Faculty rating	Composite microvascular scores correlated with average faculty evaluation scores Attitudinal scores correlated with average faculty evaluation scores

Abbreviations: ABOto, American Board of Otolaryngology; AOA, Alpha Omega Alpha; COR, composite opinion rank; CPR, composite performance rank; DR, during residency; GPA, grade point average; LOR, letter of recommendation; NBME, National Board Medical Examination; NRMP, National Resident Matching Program; USMLE, United States Medical Licensing Examination.

“Supervariable”: GPA >3.4, honors in medicine and surgery, NBME I >650, class rank >85%, AOA.

All studies were retrospective cohort studies (level 4) with sample sizes ranging from 24 to 175 otolaryngology applicants. Studies ranged in length from 5 years up to 16 years and included assessments of success during residency and in clinical practice.

Standardized Testing

Three studies evaluated the utility of scores from the United States Medical Licensing Exam (USMLE) or National Board of Medical Examiners as predictors of success. Chole and Ogden¹¹ and Calhoun et al¹² found no relationship between USMLE or National Board of Medical Examiners scores and residency success, while Daly et al¹³ found a correlation between USMLE Step 1 score and both postgraduate year 2 in-service scores and having >1 peer-reviewed publication during residency. Similar to other studies, however, they did not find any correlation between USMLE score and faculty ranking of residents.

Alpha Omega Alpha Honor Medical Society

Alpha Omega Alpha (AOA) status was also evaluated as a potential predictor in 3 of the 6 studies. Again, 2 studies found no correlation.^11,12 In contrast, Daly et al¹³ found a significant relationship between AOA status and academic appointment postresidency. In addition, they found a significant relationship between AOA status and highest tertile of faculty rank. To obtain faculty rank, all residents included in the study (ie, residents in the program between 1983 and 1993) were ranked in 1997 by 7 faculty members on a scale of 1 (highest) to 40 (lowest) on their global performance as compared with all other participants. The highest tertile consisted of those residents who were ranked 1 to 13 by these faculty members.

Medical School Performance

Performance on medical school clinical rotations was examined in 2 studies. Chole and Ogden¹¹ did not identify any relationship between percentage of rotations in which honors was received and faculty rating of resident performance. In contrast, Calhoun et al¹² found that honors on a core surgery rotation was the only variable that correlated with excellent performance in residency. Interestingly, excellent performance in residency was determined by a questionnaire administered to program directors approximately 5 years after residents had completed their training. The top quartile of residents evaluated in the areas of global, technical, and interpersonal skills were considered to be “excellent” performers.

Letters of Recommendation

Daly et al¹³ found a significant relationship between an exceptional trait in a letter of recommendation and both academic appointment postresidency and highest tertile of faculty rank. Exceptional traits included clinical and academic skills, dedication, character, research contributions, and excellence in a nonacademic area. Chole and Ogden,¹¹ however, found no relationship between the strength of the letter of recommendation and residency success.

Interviews

Using a 7-point rating scale (1 = poor, 7 = outstanding) for applicant interviews, Chole and Ogden¹¹ evaluated the relationship between average interview score and residency performance. The authors noted only a weak correlation between interview score and faculty rank. In contrast, Daly et al¹³ did not identify any correlation in percentage of interviewers favoring selection of a candidate or not.

National Resident Matching Program Rank

Bent et al¹⁶ evaluated 8 consecutive residency classes with regard to National Resident Matching Program rank order. They failed to identify any correlation between rank number or group (quartile, thirds, or halves) and in-service scores, faculty evaluations, or coresident evaluations. Additionally, rank did not correlate with selection as chief resident or receipt of the resident teaching award.

Psychomotor Testing

Two studies evaluated psychomotor tasks as possible predictors of residency performance.^14,15 Tang et al¹⁴ found no correlation between soap carving and residency class ranking and no correlation with any other examined resident qualities—cognitive knowledge, visuospatial ability, manual dexterity, and decision making. Alternatively, Moore et al¹⁵ found that an applicant’s composite microvascular score and attitudinal scores during his or her interview correlated with average faculty evaluations and ratings.

Quality Assessment

Bias assessments were performed for the 6 studies with the Quality in Prognosis Studies tool to assess risk of bias across 6 domains.¹⁰ Table 2 shows the results for each domain and, subsequently, the overall risk of bias for the study. Of the 6 studies, 5 had an overall high risk of bias, with Bent et al¹⁶ being the only study with a moderate risk of bias.

Table 2.

Risk-of-Bias Assessment per Quality in Prognosis Studies Tool.

	Study		Measurement
First Author (Year)	Participation	Attrition	Prognostic Factor	Outcome	Confounding	Statistical Analysis and Reporting	Overall Bias
Calhoun¹² (1997)	Moderate	Moderate	Moderate	High	High	High	High
Daly¹³ (2006)	Low	Moderate	Moderate	High	High	High	High
Bent¹⁶ (2011)	Low	Low	High	Low	High	High	Moderate
Chole¹¹ (2012)	Low	Moderate	High	High	High	High	High
Tang¹⁴ (2014)	Low	Moderate	High	High	High	High	High
Moore¹⁵ (2014)	Low	Low	Moderate	High	High	High	High

Discussion

In 2014, Christophel and Levine noted “an excess of applicants is a great problem for the field of otolaryngology–head and neck surgery to have, but the electronic matching process has fostered a shotgun approach by medical students.”¹⁷ This commentary was in response to the overwhelming number of applications that individual residency programs were receiving. In light of this, programs have increasingly depended on various cutoffs and criteria (eg, USMLE scores, AOA status, publications) to help them identify truly exceptional candidates. With this practice in mind, we sought to systematically evaluate the applicant characteristics that have been identified as having a correlation with performance during otolaryngology residency.

Other highly competitive specialties, including ophthalmology, dermatology, and orthopedic surgery, have evaluated which applicant attributes are predictive of residency success in their respective fields.^3-5 A common theme among all was the importance of nonacademic attributes in predicting success, but the relative inability to assess this in a valid and reliable manner was also noted. Within this systematic review, specific focus was directed at the characteristics actually available to an otolaryngology selection committee and its correlation with future performance in residency or practice. The studies included in this review were able to provide relatively few criteria that correlated with residency success, with generally inconsistent results among studies. Furthermore, the majority of the studies suffered from a high risk of bias, thus limiting the generalizability and implications of their findings.

Lee et al³ identified the need for further research within the ophthalmology community to correlate variables available at the time of application with measures of resident performance. Harfmann and Zirwas⁴ suggested the utility of multi-institutional longitudinal studies of medical school applicants to dermatology and subsequent performance in residency to identify the most pertinent qualities predictive of success. In a systematic review of the entire medical literature, Patterson and colleagues highlighted the need for collaborative multi-institutional studies to collect and analyze high-quality data about the effectiveness and stakeholder acceptability of different selection methods when designing a selection system.⁸

Notably, there is a dearth of theory-driven studies focused on medical selection.⁸ Extensive research in the field of personnel psychology has been performed to evaluate methods for selecting individuals for employment.¹⁸ Cognitive ability, while considered the most important stand-alone measure, is best supplemented with other selection strategies to increase predictive success. For example, tests of integrity—which is a compound personality construct related to conscientiousness, agreeableness, and emotional stability—produce large gains in predicting job performance. A comprehensive review of personnel selection is beyond the scope of this systematic review; however, it is a valuable topic that we believe deserves further consideration as we seek to improve our specialty’s approach to residency selection. While the articles included in this systematic review had inconsistent results limiting utilization, they certainly provide historical insight into the predictors and criteria (eg, outcomes) previously deemed pertinent by otolaryngology–head and neck surgery program directors and faculty.^11-16

There are a few limitations to our review. First, despite a thorough literature search, very few studies (n = 6) actually evaluated predictors of otolaryngology residency success. Additionally, all of these studies were relatively small retrospective cohorts (level 4 evidence), with moderate to high levels of bias, limiting the conclusions that can be surmised from their findings. Finally, as with any systematic review, when summarizing the results of multiple studies, one runs the risk of oversimplifying some of the finer nuances and intricacies of the original research.

Conclusions

Over the past decade, there has been a heightened focus on improving the resident selection process across all specialties and particularly within those considered highly competitive.^3-8 Previous research, though, has generally lacked a theoretical background, thus predisposing this work to inconsistent results and high risk of bias, limiting meaningful application of the data. The studies within this review, however, provide historical insight into the predictors and criteria previously deemed pertinent by otolaryngology–head and neck surgery program directors and faculty.^11-16 Combining past research knowledge with consideration for the principles of personnel selection provides a foundation with which to engage in theory-driven longitudinal studies on otolaryngology resident selection moving forward.

Author Contributions

Sarah N. Bowe, substantial contribution to conception and design; acquisition, analysis, and interpretation of data; drafting and revising; final approval; accountability for all aspects of work; Adrienne M. Laury, substantial contribution to conception and design; acquisition, analysis, and interpretation of data; drafting and revising; final approval; accountability for all aspects of work; Stacey T. Gray, substantial contribution to design; interpretation of data; revising; final approval; accountability for all aspects of work.

Disclosures

Competing interests: None.

Sponsorships: None.

Funding source: None.

Footnotes

No sponsorships or competing interests have been disclosed for this article.

The views expressed herein are those of the authors and do not reflect the official policy or position of Brooke Army Medical Center, the US Army Medical Department, the US Army Office of the Surgeon General, the Department of the Army, the Department of Defense, or the US government.

Supplemental Material

Additional supporting information is available in the online version of the article.

References

Keck

Arnold

Willoughby

Calkins

. Efficacy of cognitive/noncognitive measures in predicting resident-physician performance. J Med Ed. 1979;54:759-765.

Schaverien

. Selection for surgical training: an evidence-based review. J Surg Ed. 2016;73:721-729.

Lee

Golnik

Oetting

. Re-engineering the resident applicant selection process in ophthalmology: a literature review and recommendations for improvement. Surv Ophthalmol. 2008;53:164-176.

Harfmann

Zirwas

. Can performance in medical school predict performance in residency? A compilation and review of correlative studies. J Am Acad Dermatol. 2011;65:1010-1022.

Egol

Collins

Zukerman

. Success in orthopedic training: resident selection and predictors of quality performance. J Am Acad Orthop Surg. 2011;19:72-78.

Hamdy

Prasad

Anderson

. BEME systematic review: predictive values of measurements obtained in medical schools and future performance in medical practice. Med Teach. 2006;28:103-116.

Kenny

McInnes

Singh

. Associations between residency selection strategies and doctor performance: a meta-analysis. Med Ed. 2013;47:790-800.

Patterson

Knight

Dowell

Nicholson

Cousans

Cleland

. How effective are selection methods in medical education? A systematic review. Med Ed. 2016;50:36-60.

Moher

Liberati

Tetzlaff

. Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. BMJ. 2009;339:b2535.

10.

Hayden

Côté

Bombardier

. Evaluation of the quality of prognosis studies in systematic reviews. Ann Intern Med. 2006;144:427-437.

11.

Chole

Ogden

. Predictors of future success in otolaryngology residency applicants. Arch Otolaryngol Head Neck Surg. 2012;138:707-712.

12.

Calhoun

Hokanson

Bailey

. Predictors of residency performance: a follow-up study. Otolaryngol Head Neck Surg. 1997;116:647-651.

13.

Daly

Levine

Adams

. Predictors for resident success in otolaryngology. J Am Coll Surg. 2006;202:649-654.

14.

Tang

Hilsinger

Jr Cruz

Schloegel

Byl

Jr Rasgon

. Manual dexterity aptitude testing: a soap carving study. JAMA Otolarynogl Head Neck Surg. 2014;140:243-249.

15.

Moore

Price

Van Abel

Carlson

. Still under the microscope: can a surgical aptitude test predict otolaryngology resident performance? Laryngoscope. 2015;125:E57-E61.

16.

Bent

Colley

Zahtz

. Otolaryngology resident selection: do rank lists matter? Otolaryngol Head Neck Surg. 2011;144:537-541.

17.

Christophel

Levine

. Too much of a good thing. JAMA Otolaryngol. 2014;140:292-292.

18.

Schmidt

Hunter

. The validity and utility of selection methods in personnel psychology: practical and theoretical implications of 85 years of research findings. Psychol Bull. 1998;124:262-274.

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.03 MB