Abstract
Academic journals are the primary mode of communication among researchers, and they play a central role in the creation, diffusion, and use of knowledge. This article updates previous attempts to identify a core set of journals that most education scholars would acknowledge as consequential sources. On the basis of nominations from a panel of experts, 11 primary journals were identified; 3 of these—American Educational Research Journal, Educational Researcher, and Review of Educational Research—were nominated by at least one third of the respondents. The impact of these journals is assessed using a number of alternative metrics. In addition, differences in impact on policy and practice versus scholarship are considered.
Because education is informed by and cuts across a number of disciplines, it constitutes a sort of metadiscipline (Bates, 1999; see also Paisley, 1984), informed, for example, by social sciences such as anthropology, sociology, economics, and psychology. The consequent intellectual complexity is also magnified by the variety of subareas within education (e.g., policy, administration, teacher education, higher education). Education researchers are, as Feuer, Towne, and Shavelson (2002) observed, an eclectic mix whose level of “diversity has made the development of common ground difficult” (p. 7). Yet it is important that education researchers develop common ground, both for disciplinary cohesion and in order to present helpful findings to the practitioners and policy makers they seek to help.
Journals are where scholars influence each other’s thinking through the exchange of ideas and findings. Because education is inherently interdisciplinary, its many journals vary widely in what they present. Yet to the extent that there is disciplinary cohesion, it would be reasonable to see it manifest in a core set of journals, which provide education researchers the opportunity to read, think about, and respond to the same material. Past efforts to examine the prestige and citation patterns of education journals (i.e., Budd, 1990; Luce & Johnson, 1978; Smart & Elton, 1981) have revealed little consensus about which journals are critical resources. But the topic remains important and so warrants a more current reexamination.
We were interested in what a core set of education journals might be, assuming that these journals provide the point at which a common ground within the discipline can occur. But because we also assumed that those core journals would influence not only scholars but practitioners and policy makers as well, we also address in this essay some of the conceptual and measurement issues related to journal impact and the associated concept of journal quality. Finally, we report a study that identified the journals that education scholars perceive as having a high impact on either scholarship or policy and practice. We also attempt to validate impact by using and comparing several citation-based indexes and expert judgment.
Broad Endorsement and Impact as Essential Criteria for Identifying Core Journals
It is intuitive that a core journal is one that would have broad endorsement by those in its particular discipline. But once this criterion is met, it is reasonable to assume a second criterion as well: demonstrated impact. Impact, though, is a multifaceted concept that is difficult to capture with any single measure (see, e.g., Truex, Takeda, & Cuellar, 2009). Impact overlaps with the less explicit concepts of journal prestige, reputation, and quality. But despite the overlap among these several constructs, our specific concern was with impact on the thinking of other scholars and on that of policy makers and practitioners. Scholars, and the institutions in which they work, increasingly assess this in terms of one or more citation indexes, particularly when impact on other scholars is the focus.
The use of citation data to assess the impact of scholars dates only from 1964, when Garfield established the Science Citation Index (Garfield, 1964), not quite a decade after having proposed the use of citation data (Garfield, 1955). The most widely used citation measure of journals is the Web of Science (WOS) journal impact factor (JIF), which is the average number of citations obtained by all articles that a given journal has published during the previous 2 years (Garfield, 1999). For example, an impact factor of 2.0 in 2008 means that the articles published in that journal in 2006 and 2007 were cited an average of two times. Despite the widespread adoption of JIF scores as a journal metric, there are important caveats to their use (see, e.g., Dong, Loh, & Mondry, 2001). For instance, Amin and Mabe (2003) noted,
In general, fundamental and pure subject areas have higher average impact factors than specialized or applied ones . . . [often so much so] that the top journal in one field may have an impact factor lower than the bottom journal in another area. (p. 348)
Within the social sciences journals, those in education rank approximately in the lower one third with regard to JIF scores. For example, 2009 WOS data show that the mean journal impact score for education and education research journals was 0.70, whereas JIF scores for other representative social sciences include: psychiatry, 1.77; clinical psychology, 1.50; law, 0.91; and sociology, 0.75.
Often, in the case of promotion and tenure decisions, external experts in the candidates’ fields are asked to make judgments not only of their scholarship, but also of the journals in which they published. These expert judgments should reasonably concur with impact rankings obtained through the use of citation data. But mixed findings have been obtained in several studies in which the two methods (i.e., expert judgments and impact ratings) have been compared. Meho and Sonnenwald (2000) found that the two methods were highly correlated, but Maier (2006) found no significant positive relationship between them. Oppenheim (1997) found that ratings of faculty by experts were highly related to their citation data when the faculty members being evaluated were ranked high or low. However, there was only a small relationship between these measures for those in the middle of the distribution. Thus it is clear that the relationships among the several means of rating impact are yet to be determined, particularly in education journals.
One possible, although largely untested, alternative to the JIF score as a measure of journal impact is the h-index, which also relies on citations. In the short time since Hirsch (2005) proposed the h-index to characterize the impact of scholars, it has rapidly become the key index used in citation sources (e.g., WOS). Simple citation counts can provide a misleading picture, as they can be affected by one or two high-impact works in a portfolio of otherwise uncited work. But the h-index provides a measure of sustained impact and is easily calculated: When an author’s articles range from the one with the most citations to the one with the lowest, the h-index is the last rank at which the number of citations meets or exceeds the ranking (e.g., a person with an h-index of 15 would have at least 15 articles with at least 15 citations each).
Braun, Glänzel, and Schubert (2006) and Vanclay (2007) argued that the h-index should work well with journals. Vanclay, for example, noted that the h-index “is an integer, and so it avoids the false impression of precision conveyed by the three decimal points in the ISI impact factor and is much easier to verify than most alternatives” (p. 1547). Harzing and van der Wal (2008) apparently were the first to examine this empirically. They found that the h-index scores of economics and business journals correlated highly (.72) with the JIF, providing tentative validation of the h-index. Although the strength of this association might be expected given the two measures’ common use of citation data, it should be noted that the authors used Google Scholar (GS) citation data for the h-index and obtained the JIF scores from WOS, a fact that suggests the robustness of the relationship.
Harzing and van der Wal (2008) used GS data but likely would have obtained somewhat different results were they to have used WOS data. The WOS imposes clear criteria with respect to what it includes but at the potential cost of overlooking important journals. For example, Educational Researcher (ER), one of the flagship journals of the American Educational Research Association (AERA), is not indexed by the WOS. Further, GS has the advantage of being available to anyone with an Internet connection, and citation indexes can be easily obtained from it using free software such as Publish or Perish (Harzing, 2009). But the fact that GS captures so-called gray literature (e.g., unpublished papers and technical reports) means that the GS citation score for any given publication often will exceed the corresponding WOS score. To some extent, the differences between GS and WOS approximate the distinctions Bollen, Rodriguez, and de Sompel (2006) made between measures of popularity (GS) and prestige (WOS).
Manuscript acceptance rates of journals are one additional measure of quality and often are reported in promotion and tenure dossiers. The received wisdom is that the lower the acceptance rate, the better the journal. However, as Zuckerman and Merton (1971) demonstrated, acceptance rates vary widely across disciplines. Although journals in the social sciences and humanities have low acceptance rates, those in the physical sciences accept a much higher proportion of submissions. Thus it is difficult to assert confidently that acceptance rates correspond to journal quality. At least one study from the field of management (Coe & Weinstock, 1984) showed that those rates correlated poorly with journal rankings. These relationships so far have remained unexamined with respect to education journals.
Each of the means of assessing impact discussed previously is grounded in the assumption that researchers want primarily to influence the work of other scholars. Therefore, as Goldhaber and Brewer (2008) observed, “The university system . . . provides no guarantee that the research will be useful or relevant for policymakers or practitioners” (pp. 208–209). But it is important that research in an applied field like education also has the potential to affect policy and practice. It would be useful to know, then, which journals education researchers believe will have the greatest influence on policy makers and practitioners.
Consequently, we sought to determine which journals education researchers would nominate as being high impact, the measurable impacts of those journals, and the extent to which such measurable impacts relate to one another. We provide here a brief summary of the methods we used and what we found; a more detailed version of our research protocol can be obtained upon request to the first author.
Evidence Regarding Core Journals and Their Impacts
The primary data for our analyses come from a two-stage Web-based survey of education faculty in U.S. research universities. Participants represented the range of specialties ranked by the US News and World Report and in schools highly ranked in one or more of those specialties. In the first stage, we asked two samples of scholars to nominate the highest impact journals in education. Participants in one sample (n = 134) were asked to make their nominations while considering each journal’s impact on scholarship. Faculty members in the other sample were asked to consider each journal’s impact on policy and practice (n = 117). The first stage helped create a corpus of journals that were nominated as representing high-impact journals. Once this list was created, a second survey asked education faculty in both samples to rate the degree of impact of the most frequently nominated journals; 303 responded.
The data also included JIF scores, obtained from the online version of the WOS. We obtained two h-index scores, the first using WOS data and the second using GS data. We utilized the online version of the WOS to obtain the first h-index. We used the Publish or Perish software (http://www.harzing.com/resources.htm#/pop.htm) to obtain h-index measures from GS.
Time is a significant factor in assessing the h-index because those journals with a longer history will have higher scores. Therefore, to standardize analyses we considered only the previous 20 years (i.e., 1989–2008) in deriving these two h-index scores. Finally, the online version of Cabell’s Directory of Publishing Opportunities (www.cabells.com) was used to obtain most of the acceptance rate data. Cabell reports that those data were updated during 2008. Most acceptance rate data were presented as a range (e.g., 6%–10% or 21%–30%), in which case the midpoint of that range was the score used in the analyses.
Impact Analysis Findings
We developed a list of 126 journals; the final list includes all journals that at least one respondent had nominated. Of the total list, 40 journals appeared in both the scholarship and the policy–practice panels, 40 appeared only in the policy–practice panel, and 46 appeared only in the scholarship panel. To identify core journals from this larger list, we employed the rule that at least 10% of the experts from either panel (policy–practice or scholarship) must have nominated the particular journal. Eleven journals met this very liberal criterion; of these, 1, the American Educational Research Journal (AERJ), was nominated by a strong majority of the experts. A second journal, ER, was nominated by nearly one half of the expert panels.
We examined several quality indicators to determine whether the 11 core journals were different from the other 115 in the total list. We began with the JIF (because ER is not indexed by the WOS, it had neither an impact score nor a WOS-derived h-index.) We also examined whether the 11 top-rated journals could be differentiated from one another with respect to their types of perceived impact both on policy and practice and on scholarship. The first indicator was journal nominations, where we found that 8 of the 11 met the 10% threshold of nominations by both panels of nominators. Finally, we examined the relationships between and among the several measures of impact and quality, using data for all 126 nominated journals; in this regard, the two measures of h-index and the impact factor all were highly correlated. For a summary of these findings, please refer to Tables 1 and 2.
Education Journals Nominated by at Least 10% in Either Impact on Policy and Practice or Impact on Scholarship
Note. WOS = Web of Science; GS = Google Scholar.
Correlation Table for Measures of Journal Impact: 126 Journals That Received Any Nomination for Either Type of Impact (Scholarship vs. Policy and Practice)
Note. GS = Google Scholar; WOS = Web of Science; N concerns the number for which these particular types of data were available.
Lessons From the Analyses
The most striking finding from our analyses is the lack of any clear consensus about core education journals. As previously indicated, a scant 11 journals met our very liberal inclusion criteria. Only two journals were nominated by (approximately) half or more of the respondents—AERJ and ER. The third highest nominated journal, Review of Educational Research (RER), was listed by only one third of the respondents. These findings echo those of a study that Luce and Johnson (1978) conducted approximately 30 years earlier. Also similar to the findings therein, the nominated journals in our study were primarily those related to one’s area of specialization. Finally, we note that the impact factors of these top journals are low relative to those of other fields (even other social science fields). It seems likely that this is related to the knowledge on which education journals focus. Specifically, as Larabee (2003) observed, “If we think of knowledge as ranging from hard to soft and from pure to applied, educational knowledge is both very soft and very applied” (p. 14).
Our findings also raise the important question of the extent to which education scholars share a common core of knowledge and attitudes. The lack of consensus on core journals, coupled with the generally low impact scores of our final list, suggests that in considering the balance of diversity versus unity, education research appears to be skewed in favor of diversity (if not a persistent sense of what could be characterized as disarray). Unsurprisingly, and perhaps as a result of the immense problems and the endemic complexities of those problems, education as an academic field and profession has come to comprise disciplinary specialties that are now only loosely connected (Ranis & Walters, 2004). As we noted in the introduction to this article, the diversity of discursive perspectives in education might be considered a strength of the field. However, some patterns of unity, as established through consensus regarding a set of core ideas or hypotheses, are also imperative for an academic discipline to disseminate meaningful knowledge. Because education is an applied discipline, that knowledge also should be useful to educational practitioners and policy makers.
On a promising note for unity, the 11 core journals in our list did meet our expected criterion of having relatively greater impact than the other education journals. In fact, compared to the other 115 journals, the core journals demonstrated higher impact on all the measures we considered. Notably, AERA, the one professional association common to many education researchers, publishes 4 of the 11 journals that had a higher impact as indicated in our study: professional association journals tend to be those of higher quality and visibility (Bensman, 1996).
Of the four AERA journals, RER demonstrates the highest impact scores, which is characteristic of journals that publish review articles (Bourke & Butler, 1996; Moed, Van Leeuwen, & Reedijk, 1997). Unfortunately, because WOS does not index ER, impact data for it were not available from that source. Interestingly, ER was the second most frequently nominated journal and had the second highest h-index scores from GS data. These indicators underscore the journal’s importance in the education field. Because the WOS continues to be the industry standard, its omission of ER is significant and seems to require rectifying.
Data on manuscript acceptance rates yield two conclusions: The first confirmed that the 11 core journals had an average lower acceptance rate than that of the rest of the group. The second was that, overall, acceptance rates correlate poorly with impact measures (see Table 2). This finding was consistent with the earlier findings of Coe and Weinstock (1984) and calls into question the prevalent use of acceptance rate as a proxy for journal quality, particularly by promotion and tenure committees.
We also attempted to discern if education scholars could differentiate between journals with a high impact on scholarship versus those with a high impact on policy and practice. In fact, they rated the impact on scholarship higher than on policy and practice for 9 of the 11 core journals. These included even journals nominated as having a particular impact primarily on policy and practice. A reasonable interpretation is that the education researchers who served as our raters perceived that the ideas presented in most of these education journals have a greater effect on scholarship than they do on policy or practice.
Our panel of education scholars reported only two journals from this core set as having greater impact on policy and practice than on scholarship. These journals were Educational Leadership and Phi Delta Kappan. This finding was consistent with that of Zirkel (2007), who found that school superintendents and professors consistently nominated these journals as the best in educational leadership. We should note as well that our study relied on the perceptions of education scholars and that if we had sought the perceptions of practitioners and policy makers, the results might well have been different. For example, Zirkel found that school superintendents rated School Administrator and the American School Board Journal highly, but education professors did not.
Tracey, Claiborn, Goodyear, Lichtenberg, and Wampold (2008) found that, for individual scholars, the h-index correlated highly with other indicators of productivity. We found the same with respect to journals. In fact, the correlation we obtained between the GS h-index measure and the JIF score (.58) was generally similar (.72) with what Harzing and van der Wal (2008) obtained with business journals. These data do suggest that those seeking information about journal impact might consider adopting the GS-based h-index alone or in combination with other impact data. The GS measure correlated well with the other impact measures and has two particular advantages. First, it is available at no cost and therefore is available to virtually anyone with a computer and an Internet connection. Second, the GS measure provides impact data for those journals not indexed by the WOS, including, for example, ER.
Our findings converge with other recent nominations of top journals (Zirkel, 2007). Five of the 11 journals on our list (AERJ, Educational Leadership, Educational Evaluation and Policy Analysis, ER, and Phi Delta Kappan) also appear in the Zirkel study. The differences between our studies likely are accounted for by Zirkel’s more specific target (i.e., educational leadership journals).
Most readers will be familiar with Gertrude Stein’s now famous observation about the city of Oakland, California, that “there is no there there” (http://en.wikipedia.org/wiki/Gertrude_Stein#Quotations). Given the pronounced intellectual silos that exist in U.S. university schools and colleges of education, as we pointed out previously, we began our study with some concern that Stein’s observation might extend to the discipline of education itself and what appears to be an inherent lack of unity. Our analyses would seem to affirm that there is a core, a sense of there, with regard to education as a field of study but that not everyone in the discipline values or perhaps is even aware of it. The education scholars in our survey nominated a large list of journals but with clear convergence around only a few, only one of which was endorsed by more than one half of both nominating panels. It is reasonable to conclude, therefore, that education scholars are not looking to a common literature. And because a discipline’s literature affects how its members think about issues, it therefore becomes unclear what it means to teach education students to “think like” (Shulman, 2005, p. 52) an educator.
This inherent diversity within the field might also be a primary factor in explaining the relatively low journal impact scores for education journals in general. We believe that this and the continued attempts at separating large segments of the education research enterprise is a cause for concern. Solutions, of course, are far less obvious than the problem identification we have offered and are greatly and inherently complicated by the nature of academe, research funding, and ingrained intellectual traditions that many are resistant to changing. We hope that our preliminary empirical analyses provide some basis for a discussion of possible ways to increase the impact of education research on both scholarship and policy and practice.
Footnotes
Acknowledgements
We would like to express our appreciation to June Ahn for his help with an earlier draft of this article.