Abstract
The recruitment and retention of students interested in STEM pursuits (science, technology, engineering, and mathematics) is a national priority obstructed by a lack of relevant measures. We developed a set of assessment devices applicable to program evaluation as well as the identification of STEM interested students and their self-efficacy levels. In this article, we report the psychometric properties of two interest and two self-efficacy measures of STEM-related activities and occupations. Exploratory factor analyses conducted on each measure with a randomly selected half of the sample (n = 213) resulted in one general STEM factor. Confirmatory factor analyses were conducted on shortened versions of the tests for the remaining half of the sample (n = 190). Goodness-of-fit indices all supported the one-factor model. Applications of the instruments to STEM assessment and education are discussed.
Science, technology, engineering, and mathematics (STEM) education and employment issues have received considerable attention in recent years. The United States has placed a high priority on increasing enrollment and retention in STEM studies (Committee on Science, Engineering, and Public Policy, 2007; CRS Report for Congress, 2008) to gain international competitive advantage. Many academically capable students do not pursue STEM majors (National Science Foundation, 2010a, 2010b). For example, approximately 17.5% of all college degrees conferred were in the STEM fields compared to 23% in the social sciences, 21% in business, and 20% in the arts and humanities (American Council on Education, 2007). STEM careers were described in previous decades as “nontraditional” for women and minorities, given that these individuals showed diminished pursuit of these occupations due to perceived barriers and lack of support (Betz & Hackett, 1996; Stout, Dasgupta, Hunsinger, & McManus, 2011; Walton & Cohen, 2007). This problem endures today and is especially tragic given our current national needs.
Research in career counseling and vocational psychology has examined the roles of both interest and self-efficacy in career decisions made by high school and college students (Lent, Sheu, Gloster, & Wilkins, 2010; Tracey, 2010; Tracey & Hopkins, 2001). Social cognitive career theory (Lent, Brown, & Hackett, 1994), derived from Bandura (1986), posits that interest and self-efficacy play separate roles in career choice and persistence (Armstrong & Vogel, 2009; Betz & Borgen, 2010; Betz, Harmon, & Borgen, 1996; Betz & Rottinghaus, 2006; Byars-Winston, Estrada, Howard, Davis, & Zalapa, 2010; Donnay & Borgen, 1999; Lent et al, 2010; Silvia, 2003; Tracey 1997, 2002, 2010; Tracey & Hopkins, 2001). Betz, Harmon, and Borgen (1996), for example, reported that interests and confidence assessments interpreted jointly are useful in working with clients making career decisions. Although early studies suggested that interest and self-efficacy were similar constructs (Donnay & Borgen, 1999; Tracey, 1997) yielding comparable findings (Betz et al., 1996), subsequent research suggested greater complexity. In 2002, Tracey found a reciprocal influence of interests leading to competence development and vice versa in elementary and middle school students. Also, Silvia (2003) noted that increasing self-efficacy did not lead to increases in interest for all areas of life, except for career decision making. Similar findings were reported by Byars-Winston, Estrada, Howard, Davis, and Zalapa (2010), Lent et al. (2008), and Lent, Sheu, Gloster, and Wilkins (2010).
The need to assess both interest and self-efficacy is now clearly established; both lead to career choice in college (Armstrong & Vogel, 2009; Lent et al., 2010; Tracey, 2010; Tracey & Hopkins, 2001). With respect to STEM fields in particular, Byars-Winston et al. (2010) found that high school students who had a high sense of self-efficacy in math and science expressed interests and aspirations to pursue a STEM degree in college. The stronger a student’s interest and self-efficacy beliefs are related to an occupational choice, the higher the certainty of that choice (Tracey, 2010) and the higher the persistence in that choice (Lent et al., 1994).
Unfortunately, there is a paucity of relevant measures. No tests specifically focused on STEM career interests exist. Several instruments related to STEM self-efficacy have been developed over the years (e.g., Dawes, Horan, & Hackett, 2000; Lent, Brown, & Larkin, 1986); but these have questionable validity support. They are often simply created from ad hoc lists of nontraditional jobs. There are also several STEM scales attached to full interest measures such as the Kuder Skills Assessment (Rottinghaus, 2009), but these are available only with the entire scale. There is a need for focused, brief, valid measures that can be used across contexts.
The STEM Career Interest Test (SCIT) and the STEM Career Self-Efficacy Test (SCS-ET) were the first pair of tests developed for this study. Their items consist of career activities rated on Likert-type scales reflecting interest and self-efficacy for engaging in each task. These tests were assembled by Horan (2010) using five subscales of the public-domain Basic Interest Markers (BIMs; Armstrong, Allison, & Rounds, 2008; Day & Rounds, 1997; Liao, Armstrong, & Rounds, 2008) that correspond to the STEM acronym, with “science” represented by separate physical and life scales. The BIMs scales were previously subjected to factor analysis and confirmatory factor analysis by Rounds and his associates where 31 scales emerged out of 343 total items. All scales had good psychometric properties; coefficient αs, for example, ranged from .85 to .95, and they have high correlations with the Basic Interest Scales from the Strong Interest Inventory (Liao et al., 2008). The full set of BIMs scales are an excellent vocational interest device, but it was not known how the five STEM subscales would perform in isolation and/or in conjunction with a STEM self-efficacy measure keyed to their specific career activities.
It is crucial that the STEM careers included in ad hoc measures of interests and self-efficacy accurately reflect the STEM field. Domain representation is essential but has not been established in these scales. Our first pair of measures derived from career activities identified by Day and Rounds (1997). We sought to develop an additional pair of measures that focus on interest and self-efficacy in a representative set of STEM occupational titles. These are the STEM Occupational Interest Test (SOIT) and the STEM Occupational Self-Efficacy Test (SOS-ET).
Likert assessment of interest and self-efficacy for specific occupational titles has proved durably reliable over the years, regardless of the job array selected (e.g., Dawes et al., 2000; Rotberg, Brown, & Ware, 1987); however, achieving consensus on what constitutes a STEM occupation is a formidable task. The 16 career clusters framework created by the U.S. Department of Education, Office of Vocational and Adult Education (OVAE), for example, is not consistent with the Department of Labor, and both would seem to be at odds with National Science Foundation priorities. A “faller” (i.e., lumberjack), for instance, is labeled as a science career by the Department of Labor, whereas a registered nurse, even with a bachelor’s degree, is excluded by both the Department of Labor and the 16 OVAE cluster systems. We wondered whether excluding health science and many technology careers from the STEM category could produce a host of psychometric difficulties given, for example, that graduate degrees in medicine and biology require interests and self-efficacy along the same academic path. So, we selected broadly from an enlarged domain of STEM occupations.
At the outset of our work, the Department of Labor identified 170 occupations as STEM when searched under the description “all STEM disciplines” but only 116 occupations when searched under “STEM career cluster.” Apart from the resulting confusion, many occupations listed (such as anthropologist, forester, or political scientist) do not fall under what the federal government and the National Science Foundation would view as STEM occupations. In selecting which STEM occupations to include in our interest and self-efficacy tests, we began by combining all occupations classified as STEM, health science, and technology in the 16-Federal Clusters and Department of Labor frameworks, and then excluded most careers that did not require a bachelor’s degree. Further winnowing involved eliminating obscure and redundant job titles and produced a working list of 39 occupations for consideration in our STEM Occupational Interest and Self-Efficacy Tests.
In sum, we developed two pairs of tests. The first addressed interest and self-efficacy for career activities. The second pair focused on interest and self-efficacy for occupational titles. The purpose of our study was to explore the psychometric properties of this assessment battery. Exploratory and confirmatory factor analyses were used to establish the factor structure and to shorten the current versions of the tests. Though a five-factor structure was suggested by Armstrong, Allison, and Rounds (2008) and Liao, Armstrong, and Rounds (2008), it was not clear if such would hold when using fewer items and/or occupational title data. We were also interested in exploring whether professions not typically on STEM lists but heavily loaded with preparatory science and/or mathematics courses would be differentiated from STEM fields listed by the Department of Labor and in the 16 OVAE cluster framework.
Method
Participants
Our sample consisted of 403 college students, 269 females (66.7%) and 134 males (33%), from a large Southwestern university. Their mean age was 23.7 (SD = 7.5), with a range of 18–58 years. Most were White (69%, n = 278); others self-identified as Latino-Hispanic American (9.4%, n = 38), African American (7.2%, n = 29), Asian American (5.5%, n = 22), Native American (2.7%, n = 11), other American (4%, n = 16), and foreign/international (2.2%, n = 9).
Measures
SCIT and SCS-ET
The first pair of tests that we evaluated assessed STEM career interest and self-efficacy each using 5-point Likert-type ratings of 55 items reflecting job activities in STEM fields. The items were taken from 5 of the 31 subscales in the BIMs (Liao et al., 2008), namely, engineering, information technology, mathematics, life science, and physical science, that correspond to the STEM acronym, with physical and life sciences represented as separate scales. Additional control items taken from the 12-item BIMs Social Science scale were interspersed with the STEM items for discriminative validity purposes. Likert anchors for the interest test were strongly dislike, dislike, neutral, like, strongly like. We used not confident, slightly confident, confident, very confident, (and) extremely confident for assessing self-efficacy. Essentially, the first pair of tests used the same career activities but differed on Likert rating—interest versus confidence.
SOIT and SOS-ET
The second pair of tests used occupational titles to assess STEM interest and self-efficacy. In selecting professions to include in our STEM occupational interest and self-efficacy tests, we began by combining all occupations classified as STEM, health science, and technology in the 16-Federal Clusters and the Department of Labor frameworks, and then excluded most occupations that did not require a bachelor’s degree as well as those that were obscure or redundant until the list of 39 occupations depicted in Table 1 was derived. Again, for discriminative validity purposes, additional social science occupational titles were interspersed when the test was administered. These were elementary school teacher, social and community service manager, school psychologist, marriage and family therapist, customer service representative, and sociologist.
STEM Occupations.
aIndicates majors or careers that have a high component of science courses as part of their curriculum but are not traditionally considered part STEM fields.
Participants were provided with the list of occupations, each followed by a brief Department of Labor description (e.g., astronomer: Observe, research, and interpret celestial and astronomical phenomena to increase basic knowledge and apply such information to practical problems). They were asked to rate on a Likert-type scale from 1 to 5, how interested they were and how confident they felt about performing the tasks associated with each occupation in their future career. The specific verbal anchors used in the first pair of tests were used in this pair as well.
Procedures
Participants were recruited from an undergraduate career development course and various undergraduate technology classes from a large Southwestern university. They received extra credit in their classes for completion of the test battery. All assessment devices as well as the informed consent document and demographic questionnaire were presented electronically via Surveymonkey (http://www.surveymonkey.com). In addition to the control non-STEM items, all four tests contained a validity check item repeated twice at different points within each test.
Of the 451 participants who agreed to participate, 412 finished all components of the assessment battery (91.4%). The 451 represented approximately 65% of those who were enrolled in the classes solicited.
No clear pattern appeared for the missing data. Of the 39 incomplete protocols, 16 participants answered “I agree” to participate and did not answer anything else; 15 agreed to participate and completed the demographic sheet but failed to fill out any of the assessments; and 8 filled out some but not all of the measures. Following Rubin (1976) and Sterner (2011), the missing data would appear to fall in the missing at random category since participants failed to provide large portions of information and there was no identifiable pattern. Given the large data set and the low percentage of incomplete protocols, we used listwise deletion as the remediation method.
Response agreement to the additional validity check procedure (repetition of 1 item in each test) was assessed by noting the number inconsistent responses made by each participant. Nine participants made more than one inconsistent response and were excluded from the data set. Our final analyses were conducted on the 403 participants who provided complete and consistent responses.
Results
The sample was randomly split in half. The first half (n = 213) was used to establish the factor structure of the tests and to obtain the highest loading items to create shorter versions. The second half (n = 190) was used to confirm factor loadings and for validation purposes.
Exploratory Factor Analysis (EFA)
EFAs using principal axis factoring were individually performed on the STEM career activity and self-efficacy tests to determine their factor structure. The number of factors retained was determined using the scree tests and interpretability. Eigenvalues and variances for the first six factors of the SCIT were as follows: 21.54 (32.15%), 7.70 (11.49%), 6.50 (9.70%), 4.44 (6.63%), 1.80 (2.69%), and 1.54 (2.30%). These six factors account for 64.96% of the total variance. Eigenvalues and variances for the first six factors of the SCS-ET were as follows: 25.55 (38.14%), 8.60 (12.83%), 5.08 (7.58%), 4.48 (6.69%), 2.43 (3.63%), and 1.25 (1.87%). These six factors account for 70.74% of the total variance.
The first factor to emerge in the unrotated solution of the SCIT was a strong general STEM factor comprised of items across all STEM categories: physical science, life science, technology, engineering, and mathematics. There were 48 items that loaded into this factor. The second factor that emerged was social science; it included all the social science items that were added to the test for discriminative validity purposes. The last factor was rather small; it consisted of 7 items—3 life science and 4 technology that double loaded into the first and third factor. Other extracted factors did not follow clear patterns and were judged as not interpretable. Essentially, then a strong STEM factor was found and it was clearly differentiated from social science items.
Analyses of the SCS-ET produced similar results. A strong STEM factor composed of 54 items across all five STEM categories emerged first and was clearly differentiated from a second factor containing all the social science items. One item loaded into a factor by itself; the other factors extracted again did not follow any pattern and were thus considered uninterpretable.
Eight high loading items representing all STEM areas were chosen to construct short forms for each test. Table 2 depicts the factor loadings for each item.
Factor Loadings on Short Versions of STEM Career Interest and Self-Efficacy Tests.
Note. Extraction Method: Principal Axis Factoring; SCIT = STEM Career Interest Test; SCS-ET = STEM Career Self-Efficacy Test.
As with the STEM career interest and self-efficacy data, EFAs using principal axis factoring were performed on the STEM occupational interest and self-efficacy tests to determine their factor structure. The number of factors retained was determined using the scree tests and interpretability. Eigenvalues and variances for the first six factors of the occupational interest test were as follows: 16.79 (37.3%), 6.25 (13.89%), 3.47 (7.71%), 1.58 (3.51%), 1.45 (3.21), and 1.17 (2.59%). Eigenvalues and variances for the first six factors of the occupational self-efficacy test were as follows: 20.89 (46.42%), 5.56 (12.35%), 3.36 (7.47%),
The factor structure of the STEM occupational interest and self-efficacy tests were similar to each other and to the STEM career interest and self-efficacy tests described previously. A strong STEM factor comprised of items across all STEM areas was first to emerge on the occupational interest test; 35 of the 45 items loaded into this general factor. The second factor contained occupations not on traditional STEM lists that we added because of their high math and science educational requirements along with four of the six social science occupations that were included for discriminative validity purposes. A third factor contained the remaining two social science items.
Likewise, analysis of the occupational self-efficacy data revealed a strong STEM factor comprised of 36 items. A second factor contained all of the social science items plus three of the four occupation items that we added because of their high math and science educational prerequisites. So, both EFAs yield a strong STEM factor differentiated from both social science occupations and those having just STEM-related educational requirements.
Because of their single-factor structure, we shortened each test by retaining 6 items in each with the highest factor loadings while controlling for correlations with other items. Table 3 depicts the items and their factor loadings.
Factor Loadings on Short Versions of STEM Occupational Interest and Self-Efficacy Tests.
Note. Extraction Method: Principal Axis Factoring; SOIT = STEM Occupational Interest Test; SOS-ET = STEM Occupational Self-Efficacy Test.
Confirmatory Factor Analyses
Confirmatory factor analyses with Mplus were conducted on the shortened scales for the second half of the original data to test the one-factor model. For the SCIT, the χ2 test of fit was found to be significant, χ2(df = 7, n = 190) = 26.88, p < .04. The comparative fit index (CFI) yielded a value of .99, above the .95 cutoff suggested by Hu and Bentler (1998), and the root mean square error of approximation (RMSEA) was .06 with a 90% confidence band of [0.00, .13]. An RMSEA score between .05 and .08 is considered indicative of reasonable fit (Browne & Cudeck, 1993; MacCallum, Browne, & Sugawara, 1996). The standardized mean square residual (SMSR) was .03, also within the standards for good fit (Kline, 2005; Quintana & Maxwell, 1999). These three indices indicate a good fit of the one-factor model with the career interest data.
The respective values for the other three assessment devices evaluated in this study were as follows: For the SCS-ET, the chi-square test of fit was found to be significant, χ2(df = 7, n = 190) = 34.04, p < .02, CFI = .98, RMSEA = .07, and SMSR = .03; for the SOIT, the χ2 test of fit was found to be nonsignificant, χ2(df = 7, n = 190) = 13.45, p < .06, CFI = .99, RMSEA = .07, and SMSR = .03; and for the SOS-ET, the χ2 test of fit was found to be nonsignificant, χ2(df = 6, n = 190) = 9.768, p < .13, CFI = .99, RMSEA = .06, and SMSR = .02. All these indices indicate a good fit for the one-factor model across all four tests.
Finally, correlation coefficients were computed between the longer original and short versions of the scales. As shown in Table 4, significant positive relationships emerged between all the scales. The test items emerging from all of these analyses are depicted in the Appendix.
Summary of Correlational Analysis.
Note. n = 190; α = Cronbach’s alpha; M = mean; SD = standard deviation; SCIT = Stem Career Interest Test; SCS-ET = STEM Career Self-Efficacy Test; SOIT = STEM Occupational Interest Test; SOS-ET = STEM Occupational Self-Efficacy Test.
**p < .01.
Discussion
Our EFA results suggest that STEM assessment is not composed of separate factors represented by the STEM acronym or by the BIMs model when the 5 scales are separately analyzed. Rather, a unique STEM factor emerged on each of our tests that accounted for almost all of the STEM-related items regardless of which STEM area they belonged to. Our CFA results likewise suggest that a one-factor general STEM model best captures interest. Similar results emerged for self-efficacy. That is, people who show interest or confidence in STEM fields do so consistently across science, technology, engineering, and mathematics rather than predominantly in one area over another. This is supported by the high covariation between the measures regardless of the length of the measure or if it was interest or self-efficacy being assessed. People who showed high levels of interest in STEM activities or occupations also showed high levels of self-efficacy in those activities or occupations.
We began with an expanded list of STEM occupations, some not classified as such in the Department of Labor and 16-Federal Cluster frameworks, but included by us because of their STEM academic prerequisites. These were dietician/nutritionist, pharmacist, registered nurse, physician/dentist, and medical technologist. In both EFA analyses, registered nurse and dietician/nutritionist loaded with social science rather than with STEM occupations. Physician/dentist loaded with social science in the analysis of interest data, but with STEM occupations in the self-efficacy analysis. Pharmacist and medical technologist consistently loaded in the STEM factor in both analyses. This suggests inadequacies in offerings by the Department of Labor and 16 OVAE cluster frameworks. Some occupations requiring STEM academic prerequisites should probably be included in the STEM occupational category (e.g., pharmacist and medical technologist); others perhaps not (e.g., registered nurse, dietician/nutritionist). Still others (e.g., physician/dentist) seem to straddle the boundary. We can speculate about the possible interacting roles of educational level and patient contact here, but for the moment we can only conclude that our suspicions were partially confirmed. The STEM category may be too restrictive in excluding some Health Science and Technology occupations.
The implications of our results for independent variable (e.g., curriculum) construction seem clear. Had separate factors emerged representing each letter in the STEM acronym, it would be easy to experimentally evaluate outreach efforts by, say, a college of engineering in motivating its particular brand of STEM student. Tests representing the science, technology, and mathematics fields could serve as construct validity controls (McNamara & Horan, 1986; Horan, 1995) against direct effects emerging on the E scales. Given that STEM is a single factor in interest and self-efficacy assessment, we might instead hypothesize across the board improvement for general or specific STEM interventions.
Although this study reports promising reliability and validity information, there are some limitations that we will address in our future work on these scales. For example, the appropriate use of all narrow focus interest scales is tempered by the confounding bias of the general factor. Tracey (2012) demonstrated that all interest scales have content-specific information (here STEM) and general responding content. It is impossible to pull these apart in narrow scales, thus their relationships with other criteria are not predictable. A simple means of correcting for this bias is to include items that are opposite from the narrow band. In the case of STEM scales, the inclusion of 5–10 social items would provide an assessment of the general factor, which could then be used to correct the STEM scores obtained. These corrected STEM scores would then be specific to STEM content and thus yield more accurate validity information.
The extent to which these findings generalize across gender and ethnicity is limited by the lack of variability in our sample. Given the nationwide focus on STEM access and retention for everyone, it is essential to examine the structural invariance of these scales across gender and ethnicity. Moreover, STEM career choice is a developmental task often placed on adolescents because this is a key time in their interest development. It is thus important to examine the validity of these instruments for different age groups, especially younger ones. Finally, more work is needed to examine the concurrent validity of these scales with other criteria.
The results of this study provide support for two pairs of STEM interest and self-efficacy tests that can be used in a variety of contexts. All of the tests are brief and easily deployed, for example, in the identification of STEM students and in the evaluation of focused interventions to boost interest and self-efficacy for STEM studies and careers.
Footnotes
Appendix: Items by Scale
Authors’ Note
Any opinions, findings, and conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of the National Science Foundation.
Declaration of Conflicting Interests
The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: The development of these tests was supported in part by the National Science Foundation (Grant No. 0631754). Additional support was obtained by a grant from the National Career Development Association.