Including the Assessment of Nontraditional Factors in Pharmacy School Admissions

Abstract

Spanish

French

OBJECTIVE:

To review the relevant literature regarding the predictive ability of cognitive measures (ie, Pharmacy College Admissions Test [PCAT] scores and prepharmacy grade point average [GPA]) on both academic and clinical performance and discuss the inclusion of nontraditional assessment during the admissions process.

DATA SOURCES:

Articles were identified through searches of International Pharmaceutical Abstracts (1970–April 2004), MEDLINE (1966–January 2004), and ERIC (1966–April 2004) using the key words admissions variables, predictors of success, Pharmacy College Admissions Test, grade point average, cognitive variables, and noncognitive variables.

STUDY SELECTION AND DATA EXTRACTION:

Cross-sectional, longitudinal, and retrospective studies, as well as reviews, regarding pharmacy school and other higher education admissions' variables were included in this review.

DATA SYNTHESIS:

Many pharmacy school admissions committees give a majority weight to the traditional cognitive markers of prepharmacy GPA and PCAT scores when selecting viable applicants. Doing so may be problematic due to the magnitude of the relationship between traditional cognitive predictors and pharmacy school performance.

CONCLUSIONS:

Based on this review, a cogent argument is advanced for the need to examine, in addition to PCAT scores and GPAs, such nontraditional factors as empathy, citizenship, and ethical behavior. This may increase the predictive ability of preadmission factors on pharmacy school didactic and clinical performance. Schools of pharmacy can do this through questionnaires and interviews designed to assess nontraditional variables.

Keywords

nontraditional variables Pharmacy College Admissions Test

As pharmacy schools in the US strive to graduate practitioners who embrace pharmaceutical care, it may be useful to examine the problems associated with the traditional cognitive predictors of success in pharmacy school in which pharmacy school admissions committees weigh most heavily. Specifically, while many schools of pharmacy heavily weigh these measures to the virtual exclusion of nontraditional characteristics such as citizenship, motivation, integrity, empathy, self-awareness, and leadership, traditional cognitive measures such as the Pharmacy College Admissions Test (PCAT) and prepharmacy grade point average (GPA) appear to explain very little of the variance associated with pharmacy school achievement.^1–6 As students move from the lecture hall to clerkships, it is probable that GPA and PCAT scores have even less predictive value on clinical decision-making.^7–12

A pertinent question that admissions committees must ask is: Beyond a minimum GPA and PCAT score, are there other variables that may predict pharmacy school achievement and clinical performance? Stated another way, an applicant must have a baseline intellect and aptitude to be able to complete a doctor of pharmacy program. Determining this baseline will require schools of pharmacy to conduct longitudinal research. For example, to ascertain minimum GPA and PCAT scores in which students are likely to succeed in the pharmacy curriculum, admissions committees could retrospectively look at the records of students who had great difficulty in the program (or dropped out) and attempt to attribute reasons why. For instance, if the research demonstrates that students who experienced difficulty had a prepharmacy GPA <3.0 (taking into account the quality of the student's undergraduate institution), then 3.0 may be a minimum GPA when considering applicants. After that, other factors may account for much of the variance associated with future success as a pharmacist. This paper presents a case for why nontraditional factors can add reliability and validity to the pharmacy school admissions protocol.

Data Sources

Articles were identified through searches of International Pharmaceutical Abstracts (1970–April 2004), MEDLINE (1966–January 2004), and ERIC (1966–April 2004). Search words used to identify articles included admissions variables, predictors of success, Pharmacy College Admissions Test, grade point average, cognitive variables, and noncognitive variables.

The types of pertinent studies identified and reviewed included cross-sectional, longitudinal, and retrospective investigations, as well as reviews, and pertained to pharmacy school and other higher education admissions variables.

Traditional Cognitive Predictors

In pharmacy, a multitude of studies have been conducted to assess predictors of success in pharmacy school. The most frequently examined cognitive variables appear to be applicants' PCAT scores and various GPA assessments (eg, math/science prepharmacy GPA, cumulative prepharmacy GPA). The majority of these investigations used traditional cognitive measures to predict first-year pharmacy school performance.

According to the American Association of Colleges of Pharmacy, approximately 60% of pharmacy schools use applicant PCAT scores in the admissions process.¹³ Several researchers have assessed the predictive ability of the PCAT.^1–5 Until 2004, the PCAT assessed performance in 5 areas: verbal, reading comprehension, quantitative ability, biology, and chemistry (changes incorporated in 2004 include a writing section).^1,5 Using data from the PCAT, Chisholm et al.¹ reported that the PCAT did not predict academic performance in the first year of pharmacy school. The authors found that the greatest predictors of first-year GPA included prepharmacy math/science GPA and possession of a 4-year undergraduate degree prior to entering pharmacy school.

Another study examined several predictors of pharmacy achievement and found that the best predictors for the first professional year were overall GPA, prepharmacy GPA, and PCAT scores.² The strongest predictors of success in practice-related courses and clerkships were PCAT and critical thinking scores (as measured by the California Critical Thinking Skills Test). Mathematics GPA, prepharmacy GPA, verbal PCAT scores, and composite PCAT scores have been shown to be significant predictors of pharmacy students' first-year GPA by others.³ Kidd and Latif⁴ demonstrated that PCAT, essay score, California Critical Thinking Dispositions Inventory and Skills Test were found to be significant predictors of final GPA of 3 graduating classes of pharmacy students. Another study sought to determine the value of the PCAT as a predictor of academic success (as measured by first-year GPA).⁵ The authors examined 3 classes of PharmD students and reported that both GPA and PCAT scores were significant predictors of first-year pharmacy GPA.

Problems Associated with Cognitive Predictors

PREDICTING ACADEMIC SUCCESS

As discussed here and elsewhere, studies that purport to link the traditional cognitive variables of PCAT and GPA scores to pharmacy school performance are problematic because of the magnitude of the relationship.^1–6 The vast majority of investigations report correlations in the range of 0.30–0.40 (ie, PCAT to first-year pharmacy school GPA). This means that PCAT scores explain about 9–16% of the variance associated with first-year pharmacy school performance. Even the investigation that accounted for the highest amount of variance in first-year GPA explained only 24% of the variance.⁵

There are several other problems associated with the PCAT. The test attempts to measure verbal, math, and science achievement. If one assumes that applicants taking the PCAT are generally the more capable of the college population, there is a tremendous restriction in the range of scores on the PCAT. In other words, there may be very little practical significance between percentile scores. In fact, this is often the case. The PCAT has scaled scores that represent equal units on a continuous scale.¹⁴ These scores range from 200 to 600 (mean 400; SD 25). Scaled scores are similar to Z scores in statistics in that they are standardized, account for slight variations in difficulty between different PCAT forms, and allow for facilitation of raw scores to a common scale, which can then be translated into percentile scores. As an example of range restriction, consider the following PCAT scaled score and percentile rank using 2003 norms: applicant A scores a scaled 390 on the composite PCAT, while applicant B scores a scaled 410.¹⁴ The difference in scores is about 5%; however, the percentile ranks are very different. Applicant A ranks at the 31st percentile and applicant B ranks at the 70th percentile. How many admissions committee members realize that a 5% difference in PCAT composite scores translates into an almost 40 percentile move? The reason for this is not because the PCAT is somehow not measuring achievement, but rather because applicants who take the PCAT are very similar in academic background and thus perform similarly on the test. In other words, applicants self-select, whereby the better students are much more likely to apply to pharmacy schools.

As predictors of success, GPAs are problematic for at least 2 reasons. First, as with the PCAT, applicants self-select in applying to pharmacy school. Therefore, many have a GPA >3.00 (on a 4.00 scale) and the range of variation is very limited. Perhaps equally troubling is the fact that grade inflation appears to be a documented problem at higher levels of education.¹⁵ Thus, as students receive higher and higher grades, range restriction becomes even more pronounced.

Predicting Clinical Decision-Making

Although pharmacy schools must admit students who can succeed academically, it can be argued that the ultimate goal is to graduate pharmacists who will embrace pharmaceutical care and optimize their clinical decision-making regarding their patients. Previous studies, both from the distant past and more recently, in the health professions have attempted to predict and explain clinician performance. These by and large have concluded that traditional cognitive measures, such as aptitude test and GPA scores, have been mediocre to poor predictors of clinical performance. For example, school grades and performance on aptitude tests have been used as proxies for the prediction of physicians' clinical performance skills.^16,17 Unfortunately, the results of these studies have offered little insight. A review of 27 studies performed between 1955 and 1972 found little relationship between school grades and subsequent performance.¹⁷

More recently, 2 investigations in nursing and physical therapy support the earlier studies discussed above.^18,19 Krichbaum et al.¹⁸ assessed, in part, the relationship between baccalaureate nursing students' scores on a clinical performance instrument, high school GPA, and college aptitude scores. Neither aptitude scores nor high school GPA accounted for a significant portion of the variance associated with clinical performance scores when analyzed by stepwise multiple regression. However, scores on moral reasoning were significant predictors of clinical performance. Another investigation compared a class of physical therapy students on traditional admissions variables.¹⁹ The results revealed that both cumulative and science GPA were not related to physical therapists' clinical performance (r = −0.037 and 0.054, respectively). Similar to the above nursing study, moral reasoning was shown to account for 19.4% of the variance associated with the physical therapists' clinical performance.

These studies clearly demonstrate that school grades and aptitude tests are, at best, mediocre predictors of healthcare professionals' performance. Is it possible to graduate healthcare professionals who have the academic ability to succeed in school and possess the additional abilities to excel at clinical decision-making? This may be possible. There is increasing evidence that noncognitive phenomena, such as interpersonal skills, social awareness, and moral reasoning, are better discriminators between good and bad clinical performance in medicine.^{10–12,14,15,18–22} However, because of the difficulty in quantitatively measuring these attributes, this line of inquiry has not been as rigorously pursued as cognitive parameters in examining determinants of clinical performance.

Sternberg²³ argued that “successful intelligence” is much more than cognitive intelligence: it includes both creative and practical intelligence. Individuals with high levels of creative intelligence use their cognitive intelligence by applying it in new and creative ways. For example, a pharmacy may be creative in procuring reimbursement for disease-state management services. Practical intelligence allows people to apply their cognitive and creative intelligence in ways that consumers find compelling and useful. The creative pharmacy that has established a disease-state management service must be able to articulate a vision for why consumers might find it useful to go to them rather than alternatives. Sternberg's research indicates that intelligence quotient explains from 4% to 25% of the variance associated with job success. Therefore, it is clear that other factors significantly contribute to job success.

Goleman's²⁴ research on emotional intelligence supports Sternberg's thesis that cognitive intelligence is a necessary but insufficient attribute for job success. Emotional intelligence comprises 5 components that, if mastered, would benefit all pharmacy students: self awareness, self-control, motivation, empathy, and communication skills. Self-awareness revolves around realizing one's strengths and weaknesses. It is important for pharmacists to be aware of their weaknesses so that they realize their limitations and work to overcome them.

Self-control assesses one's ability to control disruptive impulses and delay gratification for long-run success. Characteristics include trustworthiness, comfort with ambiguity, and openness to change. Motivation includes putting forth a high effort that goes beyond extrinsic motivators such as pay. This includes providing a high level of patient care. Empathy is important for pharmacists because it includes the ability to understand the emotional makeup of others and respond to those emotions in a skillful manner. Empathic responding to patient needs can make the difference between medication adherence and nonadherence. Communication skills include the ability to manage relationships and persuade others. Good communication skills are critical in both the pharmacist—patient and pharmacist—healthcare professional relationship.

Nontraditional Variables

As discussed above, a cogent argument can be made that cognitive variables do not explain the majority of variance associated with pharmacy school achievement (and perhaps less of that associated with clinical decision-making). The question that admissions committees must ask is, How can schools of pharmacy assess in an objective manner nontraditional variables in our applicants?

Different investigators have defined nontraditional variables in different ways. For example, Sackett et al.²⁵ defined them as extracurricular or nonacademic activities, while Willingham²⁶ defined them as motivational and personality variables. Sedlacek²⁷ defined them based on variables that appear to reflect Sternberg's creative and practical intelligence. He developed the Noncognitive Questionnaire (NCQ), which has been shown to be more predictive of success for both traditional and nontraditional students than standardized tests (Table 1).

Table 1.

Noncognitive Admission Variables²⁷

Positive self-concept or confidence

strong self-efficacy, determination, independence

Realistic self-appraisal

especially academic; high self-awareness regarding weaknesses and works hard to self-develop

Understands and deals with racism

realist based on personal experience of racism; motivated to improve existing system

Prefers long-range goals to short-term goals or immediate needs ability to self-regulate and delay gratification

Availability of strong support system to turn to during crises

Successful leadership experience

eg, church, sports, organizations

Demonstrated community service

Knowledge acquired in a field

Considerations for Pharmacy Admissions Committees

Assuming that pharmacy admissions committees wish to assess their applicants' nontraditional attributes, how should they proceed? Sedlacek's approach is rational, that schools should begin a research program using multiple methodologies to assess nontraditional variables as predictors of pharmacist success (ie, pharmacy school didactic achievement, clinical performance).^27–29 These methodologies can include both questionnaires and interviews.

QUESTIONNAIRES

Admissions committees at schools of pharmacy can use several nontraditional questionnaire-type assessments as criteria in admitting students. These may include the NCQ, moral reasoning, and emotional intelligence assessements. The NCQ has been shown to be reliable and valid in assessing 8 noncognitive variables (Table 1).²⁷ A version of the NCQ has been shown to increase the validity of accepting traditional and nontraditional students to health programs at one college.^28,29 When used in the admissions process, the NCQ has been shown to result in a greater number of acceptances of minority racial and ethnic groups.

Assessing moral reasoning has been shown to increase the reliability and validity of identifying students who may perform at a higher level of clinical performance.^{11,12,17–19,30} While it can be argued that evaluating moral reasoning is a nontraditional cognitive measure, it assesses how we organize social cooperation in society and may therefore help explain citizenship and caring behavior toward patients and other stakeholders.²⁸ Moral reasoning has been discussed in detail elsewhere.³¹

As stated by Goleman²⁴ and Sternberg,²³ beyond a certain level of cognitive intelligence, emotional intelligence appears to be a greater predictor of success than intelligence quotient. Several in the health professions have called for increased emphasis on emotional intelligence in curricula.^32–34 Although I could find no studies that included outcomes regarding assessing emotional intelligence in health professional schools, 2 schools have programs attempting to assess emotional intelligence in medical schools.^32,35 In one school, a reliable and valid emotional intelligence instrument, Emotional Quotient Inventory, is administered to students.³² The goal of this program is to identify early weaknesses in students' emotional intelligence and direct interventions toward overcoming these weaknesses. It is believed that, by doing so, the medical school will graduate more skilled and patient-oriented physicians.

INTERVIEWS

Interviews are an ideal way to assess the nontraditional characteristics of applicants to schools of pharmacy. Approximately 60% of pharmacy schools use the interview in the admissions process.¹³ However, many health professional schools could increase the reliability and validity of their interview process by increasing the structure of it.^6,7,36,37 A considerable amount of research demonstrates the value of the structured interview versus the unstructured interview.^38–41

Unstructured interviews do not have predetermined questions and often result in the interviewer asking different applicants significantly different questions.³⁹ In addition, unstructured interviews do not have standardized scoring rubrics with which to assess applicant responses to questions. According to Edwards et al.,⁴² unstructured interviews suffer from low reliability and validity because they introduce sources of bias. These biases include rater tendencies such as leniency, severity, order, halo effects, demographics, and stereotyping.

Structured interviews include predetermined items and scoring rubrics. According to research, structured interviews are more reliable, valid, and less likely to result in gender and/or racial bias.^38,39,42,43

How might schools of pharmacy begin or improve their interview process? My school of pharmacy has attempted to improve its interview process by increasing its structure. I presented a workshop to faculty members on the research regarding structured interviewing. There was general agreement that structured interviewing should be part of the admissions process. To enlist support for the process, we decided as a school what domains are important to assess in pharmacy applicants. After hours of brainstorming, 4 key domains were deemed to be critical to assess applicants. They included motivation, citizenship, integrity/moral reasoning/ethics, and communication skills. Based on these domains, over a 3-month period, the admissions committee (based on faculty input) developed approximately 6 items for each domain.

Since communication skills were thought to be a global assessment, this domain did not have specific items, but did have a global scoring rubric. Based on each domain, a scoring rubric was developed to score applicant responses. The scoring rubric is a Likert-type scale. Table 2 provides an example of the items for motivation, while Table 3 provides its corresponding scoring rubric. Interviewers are asked to cover at least 2 items from each domain and are encouraged to probe initial responses to obtain a greater sense of the applicants' motives and past behavior regarding the domains of interest. To improve inter-rater reliability, 2 individuals (1 faculty member, 1 current student) interview each applicant invited to campus for an interview. Neither the student nor the faculty interviewers have any information about the applicant prior to the interview. After the interview, each interviewer assesses the applicant independently based on the scoring rubric so as to reduce the likelihood of bias. Since this is a new process, no information can be provided at this time regarding its success in admitting students. However, the school believes the new strategy will improve the admissions process by accepting students who have both traditional and nontraditional attributes deemed to be important to success as a pharmacist.

Table 2.

Items and for Assessing Motivation Domain

Describe how you respond to a disappointing grade on an exam or an assignment.

Define what you consider a disappointing grade.

What is it about pharmacy that makes you want to pursue this as a career?

What other careers have you considered?

What have you done to prepare yourself?

Tell me about a time in your life that was very stressful. How did you deal with it?

Identify a setback or failure and describe the processes you went through to resolve it.

If you are not accepted to a pharmacy school this year, what will you do?

Students work on projects in groups of 5–7. Please describe how you would respond if one member of your group did not contribute to a group project. What would you do?

Table 3.

Scoring Rubric for Assessing Motivation Domain

Score	Behavioral Criteria
1	extremely poor
	answers to questions and demeanor during interview demonstrate little or no commitment to learning, willingness to work hard, love for learning, intellectual curiosity, and self-directedness
2	below average
	answers to questions and demeanor during interview demonstrate a below-average commitment to learning, willingness to work hard, love for learning, intellectual curiosity, and self-directedness
3	average
	answers to questions and demeanor during interview demonstrate an adequate commitment to learning, willingness to work hard, love for learning, intellectual curiosity, and self-directedness; the applicant may be strong on some motivation traits but weak on others
4	above average
	answers to questions and demeanor during interview demonstrate an above-average commitment to learning, willingness to work hard, love for learning, intellectual curiosity, and self-directedness
5	excellent
	answers to questions and demeanor during interview demonstrate a strong degree of commitment to learning, willingness to work hard, love for learning, intellectual curiosity, and self-directedness; applicant provides a compelling argument for why he/she wishes to attend pharmacy school

In addition, structured interviews may increase the likelihood of obtaining positive outcomes in lawsuits. For example, the University of Maryland Medical School uses the interview to assess applicants' noncognitive variables and has defended the validity of assessing nontraditional factors in a lawsuit questioning the fairness of the criterion.²⁸

Summary

While it is quite easy to state and present cogent arguments for improving the admissions process to admit students capable of performing well academically and who embrace patient-focused care, it is much more difficult to actually do it. The cost is heavy in faculty time, money, and other resources. However, schools of pharmacy have a societal obligation to graduate students who adhere to the tenets of professionalism and embrace pharmaceutical care. Based on the empirical research, there appears to be good and valid reasons for schools of pharmacy to proceed with assessing nontraditional characteristics in applicants. PCAT scores and GPAs should be considered along with nontraditional factors, such as motivation, citizenship, and moral reasoning, in the admissions process. Including nontraditional assessment using such methodologies as structured interviews and questionnaires may result in the likelihood of graduating patient-focused pharmacists.

References

Chisholm

Cobb

Kotzan

. Significant factors for predicting academic success of first-year pharmacy students. Am J Pharm Educ 1995;59: 364–70.

Allen

Bond

. Prepharmacy indicators of success in pharmacy school: Grade point averages, pharmacy college admission test, communication abilities, and critical thinking skills. Pharmacotherapy 2001;21: 842–9.

Hardigan

Lai

Arneson

Robeson

Significance of didactic merit, test scores, interviews and the admissions process: A case study. Am J Pharm Educ 2001;65: 40–3.

Kidd

Latif

. An evaluation of traditional and novel predictors of didactic and professional practice success of pharmacy students. Am J Pharm Educ 2003;67, Article 109.

Thomas

Draugalis

. Utility of the pharmacy college admissions test (PCAT): Implications for admissions committees. Am J Pharm Educ 2002;66: 47–51.

Latif

. Using the structured interview for a more reliable assessment of pharmacy student applicants. Am J Pharm Educ 2004;68, Article 21.

McGaghie

. Perspectives on medical school admission. Acad Med 1990;65: 136–9.

Reede

. Predictors of success in medicine. Clin Orthop Rel Res 1999;362: 72–7.

Greengross

What patients want from their doctors. In: Allen

Brown

Hughes

, eds. Choosing tomorrow's doctors. London: Policy Studies Institute, 1997:12–9.

10.

Sheehan

Husted

Candee

Cook

Bargen

Moral judgment as a predictor of clinical performance. Eval Health Prof 1980;8: 379–400.

11.

Latif

Berger

Harris

Barker

Felkey

Pearson

. The relationship between community pharmacists' moral reasoning and components of clinical performance. J Soc Adm Pharm 1998;5: 210–24.

12.

Baldwin

Jr Adamson

Self

Sheehan

Oppenberg

. Moral reasoning and malpractice: A pilot study of orthopedic surgeons. Am J Orth 1996;25: 481–4.

13.

Admission requirements-Auburn University Harrison School of Pharmacy. www.aacp.org/Docs/MainNavigation/InstitutionalData/6184_SchoolNarratives.pdf (accessed 2004 Jul 7).

14.

PCAT technical manual supplement. San Antonio, TX: PsychCorp. Harcourt Assessment. March 2004:3–5.

15.

Wilson

New research casts doubt on value of student evaluations of professors. Chron High Educ 1998;44:A12–4.

16.

Price

Taylor

Nelson

Lewis

Loughmiller

Mathiesen

Measurement and predictors of physician performance: two decades of intermittently sustained research. Salt Lake City, UT: Aaron Press, 1971.

17.

Wingard

Williamson

. Grades as predictors of physicians' career performance: An evaluative literature review. JAMA 1973;48: 311–22.

18.

Krichbaum

Rowan

Duckett

Ryden

Savik

The clinical evaluation tool: A measure of the quality of clinical performance nursing of baccalaureate students. J Nurs Educ 1994;33: 395–404.

19.

Sisola

. Principled moral reasoning as a predictor of clinical performance in physical therapy [dissertation]. Minneapolis: University of Minnesota, 1995.

20.

Rezler

. The assessment of attitudes. Pub Health Papers 1973;52: 70–8.

21.

Cook

Margolis

. Rating pediatric house officer performance. Pediatr Res 1974;8:472.

22.

Wagoner

Gray

. Report on a survey of program directors regarding selection factors in graduate medical education. J Med Educ 1979;54: 445–62.

23.

Sternberg

. The theory of successful intelligence. Rev Gen Psychol 1999;3: 292–316.

24.

Goleman

Working with emotional intelligence. New York: Bantom Books, 1998.

25.

Sackett

Schmidt

Ellignson

Kabin

. High-stakes testing in employment, credentialing and higher education: Prospects in a post-affirmative-action world. Am Psychol 2001;56: 302–18.

26.

Willingham

. Success in college: the role of personal qualities and academic ability. New York: College Entrance Examination Board, 1985, 5–12.

27.

Sedlacek

. Beyond the big test: noncognitive assessment in higher education. Philadelphia: John Wiley & Sons, 2004.

28.

Sedlacek

. Why we should use noncognitive variables with graduate students. Counseling Center Research Report 5–1. College Park, MD: University of Maryland, 2001.

29.

Noonan

Sedlacek

Suthakaran

Predicting the success of community college students using noncognitive variables. Counseling Center Research Report 5–0. College Park, MD: University of Maryland, 2001.

30.

Rest

Narvaez

Moral development in the professions: psychology and applied ethics. Hillsdale, NJ: Lawrence Erlbaum Associates, 1994:3–225.

31.

Latif

. An assessment of the ethical reasoning of United States pharmacy students: A national study. Am J Pharm Educ 2004;68, Article 30.

32.

Wagner

Jester

Moseley

. Use of the emotional quotient inventory in medical education. Acad Med 2001;76: 506–7.

33.

Carrothers

Gregory

Gallagher

. Measuring emotional intelligence of medical school applicants. Acad Med 2000;75: 456–63.

34.

Elam

. Use of “emotional intelligence” as one measure of medical school applicants' noncognitive characteristics. Acad Med 2000;75: 445–6.

35.

Bar-On

BarOn Emotional Quotient Inventory. In: Facilitator's resource manual. Toronto: Multi-Health Systems, 1998:2.

36.

Puryear

Lewis

. Description of the interview process in selecting students for admission to U.S. medical schools. J Med Educ 1981;56: 881–5.

37.

Patrick

Altmaier

Kuperman

Ugolini

A structured interview for medical school admission. Phase 1: Initial procedures and results. Acad Med 2001;76: 66–71.

38.

Mayfield

. The selection interview-a re-evaluation of published research. Person Psychol 1964;17: 239–60.

39.

Campion

Pursell

Brown

. Structured interviewing: Raising psychometric properties of the employment interview. Person Psychol 1988;41: 25–42.

40.

Hunter

. The validity and utility of alternative predictors of job performance. Psychol Bull 1984;96: 72–98.

41.

Reilly

Chao

. Validity and fairness of some alternative employee selection procedures. Person Psychol 1982;35: 1–62.

42.

Edwards

Johnson

Molidor

. The interview in the admission process. Acad Med 1990;65: 167–77.

43.

Huffcutt

Arthur

Hunter and Hunter (1984) revisited: Interview validity for entry-level jobs. J Appl Psychol 1994;79: 184–90.