Abstract
Journal quality metrics (also referred to as bibliometrics), such as impact factors, are increasingly being used as a measure of researchers’ and educators’ success and prestige. Occupational therapists who submit articles to peer-reviewed journals may face a professional and research dilemma: Do they submit their articles to journals that largely have a professional audience and potentially do not have an impact factor, or do they opt not to publish their research material in occupational therapy–oriented journals? Occupational therapy authors can consider other journal quality metric alternatives, in addition to the impact factor option, including the Eigenfactor Score, Article Influence Score, h-index, SCImago Journal Rank (SJR), Source Normalised Impact per Paper (SNIP), and discipline-specific generated journal quality measures. These other journal quality metrics can be important reference points for occupational therapists who publish and may encourage authors to publish in journals relevant to the discipline. This process, in turn, will build the occupational therapy body of knowledge as well as provide an essential, growing reference source for evidence-based practice.
Many occupational therapy educators, researchers, university faculty members, and practitioners who publish articles in peer-reviewed journals face a professional, scholastic, research, funding, and career progression dilemma (Corr et al., 2005; Crepeau, Thibodaux, & Parham, 1999; Green, Davis, & Corcoran, 2008; Vassantachart & Rice, 1997; Walker, 2005; White 2000). Prospective occupational therapy journal article authors have, in many cases, one of two choices: (1) They can choose to publish their articles in peer-reviewed journals that largely have a professional audience and likely do not have an impact factor (IF), or (2) they can instead choose to submit their articles to peer-reviewed journals outside the occupational therapy field that have an established IF.
As a result, there is a potential loss to the occupational therapy body of knowledge, because relevant research findings may not be readily accessible to occupational therapy students and practitioners. The purpose of this article is to present a brief overview of occupational therapy journal publications, describe the features of journal IFs, outline how IFs can be manipulated, and discuss alternative journal quality metrics (also referred to as bibliometrics) that occupational therapists can use.
What Is the Scope of Occupational Therapy Journals?
Several occupational therapy–specific journals are published. A list of English-language occupational therapy journals that include peer review as part of the article selection process is included in Figure 1. Some journals, such as the American Journal of Occupational Therapy (AJOT) or the British Journal of Occupational Therapy (BJOT), are published by professional associations, whereas others, such as Physical and Occupational Therapy in Pediatrics (POTP) or Occupational Therapy International (OTI), are published by private companies. Some journals, such as the Australian Occupational Therapy Journal (AOTJ) or the Canadian Journal of Occupational Therapy (CJOT), aim at a general audience; others have a specialty audience. Examples of occupational therapy specialty journals include Physical and Occupational Therapy in Geriatrics; Occupational Therapy in Mental Health (OTMH); and the Journal of Occupational Therapy, Schools, and Early Intervention.
English-Language Occupational Therapy Journals
The publishing practices and characteristics of occupational therapy journals have been profiled in previous studies (Brown, 1997; Brown, Rodger, & Brown, 2005; Johnson & Leising, 1986; Madill, Brintnell, & Stewin, 1989; Ottenbacher & Short, 1982; Reed, 1988). One study completed by Rodger, McKenna, and Brown (2007) surveyed the authors of articles published in occupational therapy journals to determine the authors’ perceptions of indicators of journal quality and authors' ratings of 19 occupational therapy journals. Authors of articles published in occupational therapy peer-reviewed journals between 2003 and 2005 were surveyed. Questionnaires were sent to 554 authors, and 184 (33%) responded. The top five quality indicators rated as very important were (1) reputation or prestige of the journal, (2) availability, (3) rigor and quality of the journal article review process, (4) timeliness of review and publication, and (5) impact on policy and practice. The six occupational therapy journals rated the highest by respondents across most quality indicators were AJOT, AOTJ, BJOT, CJOT, Occupational Therapy Journal of Research (OTJR), and Scandinavian Journal of Occupational Therapy (SJOT).
Journal Citation Reports and Impact Factors
Journal quality metrics have been around since the 1960s (Garfield, 1999, 2006; Opthof, 1997). The primary journal quality metrics used by universities, research review agencies, and funding bodies are those published by the Thomson Scientific ISI Web of Knowledge (a private, for-profit company) and reported in Journal Citation Reports (JCR), which is available only to paying subscribers (Brumback, 2008). The Science Citation Index/Expanded (SCI⁄E) contains data from >7,300 journals in science and technology, and the Social Science Citation Index (SSCI) contains data from >2,200 journals in the social sciences, which can collectively be searched through the Thomson Reuters ISI Web of Knowledge database. Journals must apply to get listed in the JCR database, and many that apply for inclusion on the JCR database are not accepted (Adams, 2001). For example, >400 peer-reviewed nursing journals are published internationally, but only 46 are included in the JCR database, and most of those are published in the United States (Johnstone, 2007; Neuberger & Counsell 2002; Urquhart, 2006).
One key journal metric used is the IF, a measure of the frequency with which the average article in a journal has been cited in a particular year. It is calculated by dividing the number of citations in the current year to articles published in the previous 2 yr by the total number of articles published in the previous 2 yr (Garfield, 1999). IFs are increasingly being used in academic, research, and grant funding arenas as a quality indicator of individual staff members and as an eligibility criteria for promotion, pay increases, grand funding access eligibility, and access to resources within higher education contexts. JCR metrics have an inherent bias in favor of American, English-language, basic science, general-subject journals (Adams, 2001; Brumback, 2008; Ioannidis, 2006; Kurmis, 2003). Journals that are not American, publish articles that are not in the English language, and focus on specialized fields of knowledge generally have lower IFs (Lewison, 2002).
JCR metrics can be manipulated and artificially inflated (Falagas & Alexiou, 2008; Neuberger & Counsell, 2002). For example, some authors self-cite, some journals encourage the submission of review articles (which are more apt to be subsequently cited), some journals include conference abstracts and letters to the editor as part of the publications cited, and some journal editors encourage prospective authors to cite references from the journals to which they are submitting articles (Opthof, 1997; Rey-Rocha, Martin-Sempere, Martinez-Frias, & Lopez-Vera, 2001; Seglen, 1997). Indeed, it is not clear whether the number of citations an article receives measures its actual quality or simply reflects the volume of publications in that particular field of research and whether an actual difference exists between them (Saha, Saint, & Christakis, 2003). Therefore, relying on JCR IFs as the primary source of perceived quality and academic–research esteem of researchers’, clinicians’, and educators’ impact, achievement, and distinction is neither sound nor reliable (Kurmis, 2003; Postma, 2007; Walter, Bloch, Hunt, & Fisher, 2003; White, 2000).
Only five occupational therapy journals are listed on the Thomson Scientific ISI Web of Knowledge JCR database: AJOT, OTJR, AOTJ, SJOT, and the Hong Kong Journal of Occupational Therapy (HKJOT; Brown, 2010; Holguin, 2009). The AOTJ and HKJOT were not accepted for inclusion in the ISI database until 2008 and the SJOT was accepted in 2009. The 2009 IFs for four of the five journals were as follows: AJOT, 1.419; OTJR, 0.358; HKJOT, 0.462, and AOTJ, 0.438. The 2009 SJR journal rankings for AJOT, OTJR, HKJOT, and AOTJ were 17⁄55, 50⁄55, 31⁄36 and 30⁄36, respectively.
Other high-quality, peer-reviewed occupational therapy journals such as BJOT, CJOT, OTI, Occupational Therapy in Health Care (OTHC), Irish Journal of Occupational Therapy, POTP, and OTMH are not listed in the JCR database; hence, they do not have an IF calculated for them. This situation creates a dilemma for occupational therapy researchers, clinicians, university faculty staff, and educators who write for publication: Do they publish for a professional audience in occupational therapy journals, or do they play the journal quality metric game by publishing only in journals with high IFs that are listed in the JCR database?
Other Journal Quality Metrics
Occupational therapy authors could consider using other journal quality metrics instead of focusing only on JCR IFs. One is the Eigenfactor Score (EF), a measure of the overall value provided by all the articles published in a given journal in a year. The Eigenfactor algorithm is calculated using Thomson ISI data in a model in which readers follow chains of citations as they move from journal to journal. It ignores self-citations. The EF ranks journals in a manner similar to that used by Google for ranking the importance of Web sites in a search. Eigenfactors are calculated by eigenfactor.org, where they can be freely viewed (see www.eigenfactor.org).
A related measure is the Article Influence Score (AI), which is a measure of a journal’s prestige based on per-article citations. AI is calculated by dividing the EF by the percentage of all articles recorded in the JCR and, therefore, is conceptually similar to the IF. EF scores and AI scores rank journals with algorithms using the structure of the entire citation network (adjusting for citation differences across disciplines) to evaluate the importance of each journal. The EF is not corrected by article count and so is a measure of the influence of a particular journal; bigger and more highly cited journals will tend to be ranked highly. The EF score for AJOT is 0.00137499 and the AI score is 0.204081. No other EF or AI scores for other occupational therapy journals could be located on the www.eigenfactor.org site.
Another journal metric is the h index, which attempts to measure both the scientific productivity and the apparent scientific impact of a scientist or researcher (Hirsch, 2005). The h index is based on the set of a researcher’s most cited articles and the number of citations that he or she has received in the publications of others (Baldock, Ma, & Orton, 2009). It can also be applied to the productivity and impact of a group of researchers, such as a department, faculty, university, or country.
The h index was suggested by J. E. Hirsch, a physicist at the University of California, San Diego, as a tool for determining theoretical physicists’ relative quality and is sometimes called the Hirsch index or Hirsch number (Hirsch, 2005). Put simply, a researcher with an index of h has published h articles, each of which has been cited by others at least h times. Thus, the h index reflects both the number of publications and the number of citations per publication. The h index is designed to improve on simpler measures, such as the total number of citations or publications, and works properly only when comparing scientists working in the same field. The main drawbacks of the h index are that it can only increase, never decrease, and that older articles do better than younger, more recently published articles (Sidiropoulos, Katsaros, & Manolopoulos, 2007).
Another alternative is the SCImago Journal Rank (SJR), which is a portal that includes quality indicators of journals contained in the Scopus database from 1996 (SCImago, 2007). It is located at www.scimagojr.com/index.php. SCImago takes the mathematical approach behind the PageRank algorithm (central to Google’s success as an Internet search engine) and has adapted it to journal metrics to calculate the SJR. The PageRank approach weights citations from journals according to how highly cited the journal itself is using an iterative approach (Falagas, Kournaos, Arencibia-Jorge, & Karageorgopoulos, 2008). The SJR is similar to the EF except that it is based on the larger Scopus dataset from Elsevier. The SJR expresses the average number of weighted citations received in the selected year by the documents published in the selected journal in the previous 3 yr (e.g., weighted citations received in Yr X to documents published in the journal in Yr X – 1, Yr X – 2, and Yr X – 3; SCImago, 2007).
Whereas IFs are derived from citations in a single year to articles from the preceding 2 yr, the SJR calculation compares citations made in a 3-yr period to articles published in an earlier, but overlapping, 3-yr period. This calculation makes the SJR a more stable indicator of trends than IFs, which often fluctuate substantially from year to year (Siebelt et al., 2010). Two primary advantages of the SJR over the Thomson Scientific ISI Web of Knowledge JCR are that the SCImago portal is freely available online, whereas JCR is only accessible to paying subscribers. In addition, the SCImago database covers a much wider breadth of scientific, technical, medical, nursing, and allied health journals (16,500) than the 9,500 journals covered by JCR because SCImago makes use of data supplied by Scopus (Falagas et al., 2008).
A more recent type of journal metric, the Source Normalized Impact per Paper (SNIP), was developed by H. Moed at the Centre for Technology and Technology Studies, University of Leiden, the Netherlands. The SNIP is defined as the ratio of the journal’s citation count per paper and the citation potential in its specific subject field (e.g., occupational therapy, biology, anatomy, medicine) and was designed to measure the contextual citation impact by weighting citations based on the total number of citations in a subject field (Moed, 2009). A journal’s subject field is defined as the set of papers citing that journal. Therefore, the impact of a single citation is given higher value in subject areas in which citations are less likely, and vice versa—a marked limitation of the IF. It further develops Garfield’s (2006) notions of a field’s citation potential, defined as the average length of reference lists in a field (such as occupational therapy), and determines the probability of being cited and the need in fair performance assessments to correct for differences between subject fields (e.g., medicine vs. nursing).
The features of the SNIP include (1) measuring contextual citation impact by “normalizing” citation values; (2) taking a field of research’s citation frequency into account; (3) considering immediacy by taking into account how quickly an article is likely to have an impact in a given field; (4) taking into account how well a field is covered by the underlying database; and (5) countering any potential for editorial manipulation (Leydesdorff & Opthof, 2010). The SNIP is therefore another journal metric option to consider; it can be accessed at www.journalindicators.com/SearchJournal.aspx.
The SJR, h index, and SNIP for 12 occupational therapy journals listed in Scopus are reported in Table 1. POTP, AJOT, and SJOT have the highest SJR, whereas OTMH, OTHC, and the HKJOT have the lowest. The occupational therapy journals with the three highest h-index scores are AJOT, BJOT, and CJOT. The journals with the highest SNIP values are AJOT, CJOT, and AOTJ, whereas the three journals with the lowest SNIP ratings are POTG, HKJOT, OTMH. Therefore, the SJR, its related h index, and the SNIP can be used as an alternative to the JCR IF as a quantitative measure of a journal’s citation impact and perceived quality.
SCImago Journal Rank (SJR), h Index, and Source Normalized Impact per Paper (SNIP) for Occupational Therapy Journals Listed on Scopus (2009 Data)
Discipline-Specific Journal Quality Metrics
Another journal quality metric alternative is to generate discipline-specific quality measures. In the occupational therapy field, Rodger, McKenna, and Brown (2007) completed a study of the quality and impact of occupational therapy journals from authors’ perspectives. Authors of articles in published peer-reviewed occupational therapy journals between 2003 and 2005 were surveyed to determine their perceptions of journal quality indicators and ratings of 19 occupational therapy journals on these indicators (Rodger et al., 2007). The results indicated that the top five quality indicators rated as very important were (1) reputation and prestige of the journal, (2) availability, (3) rigor and quality of the article review process, (4) timeliness of review and publication, and (5) impact on policy and practice. Six journals were rated highly by respondents across most quality indicators: AJOT, AOTJ, BJOT, CJOT, OTJR, and SJOT.
Quality indicator ratings for the peer-reviewed occupational therapy journals (profession-specific global journal quality rating) and profession-specific rankings (profession-specific journal ranking) were reported in the article by Rodger et al. (2007). The profession-specific global journal quality ratings out of 10 were as follows: AJOT, 7.4; AOTJ, 6.9; BJOT, 7; CJOT, 7.2; OTJR, 7; and SJOT, 7 (Rodger et al., 2007). The profession-specific journal rankings out of 13 were as follows (lowest number indicates the highest ranking): AJOT, 1; AOTJ, 4; BJOT, 3; CJOT, 2; OTJR, 3; and SJOT, 3 (Rodger et al., 2007). These ratings can be used as discipline-specific generated journal perceived esteem quality indicators.
Finally, the World Federation of Occupational Therapists (WFOT) could generate a list of peer-reviewed occupational therapy journals and then complete a peer ranking of them in terms of perceived quality, usefulness, prestige, and contribution to the occupational therapy body of knowledge. This procedure could be completed by a panel of nominated international occupational therapy experts. Other disciplines have generated similar esteem and perceived quality factor metrics for journals published in their professions or fields of study. Again, these metric data could be used when applying for promotions, completing grant applications, and determining where they might submit future articles.
Conclusion
Journal quality metrics are here to stay. However, several other viable options should be considered in addition to those published by Thomson Scientific ISI Web of Knowledge. The JCR and IFs have their place, but they should be taken in the context that they can be manipulated and are not the complete panacea for determining journal quality and impact. Another useful option is the SCImago Journal Rank because it is open access, includes a larger number of journals, and uses the Google PageRank algorithm to calculate its journal quality metrics. Other viable options to consider are the EF, the AIS, the h index, and the SNIP. In addition, discipline-specific metrics, such as those reported by Rodger et al. (2007), can be generated.
I recommend that the WFOT take the lead and establish a working group on this issue and generate a list of quality journals relevant to the field. It should also establish a discipline-specific esteem factor estimate and journal quality rating for recognized, peer-reviewed occupational therapy journals. Such a rating will be an invaluable reference point for occupational therapists who publish and, I hope, will encourage authors to publish in journals that are relevant to the discipline. This process, in turn, will build the occupational therapy empirical body of knowledge as well as provide an essential growing reference for evidence-based practice.