Abstract
Background:
Citation rate and journal impact factor have traditionally been used to assess research impact; however, these may fail to represent impact beyond the sphere of academics. Given that social media is now used to disseminate research, alternative web-based metrics (altmetrics) were recently developed to better understand research impact on social media. However, the relationship between altmetrics and traditional bibliometrics in orthopaedic literature is poorly understood.
Purpose:
To (1) assess the extent that altmetrics correlate with traditional bibliometrics and (2) identify publication characteristics that predict greater altmetrics scores.
Study Design:
Cross-sectional study.
Methods:
Articles published in The American Journal of Sports Medicine (AJSM), The Journal of Bone and Joint Surgery, Clinical Orthopaedics and Related Research, Acta Orthopaedica, and Knee Surgery, Sports Traumatology, Arthroscopy between January 2016 and December 2016 were analyzed. Among the extracted publication characteristics were journal, number of authors, geographic region of origin, highest degree of first author, study subject and design, sample size, conflicts of interest, and level of evidence; number of references, institutions, citations, tweets, Facebook mentions, and news mentions; and Altmetric Attention Score (AAS). Multivariate regressions were used to determine (1) publication characteristics predictive of AAS and social media attention (mentions on Twitter, Facebook, and the news) and (2) the relationship between AAS and citation rate.
Results:
A total of 496 published articles were included, with a mean AAS of 8.6 (SD, 31.7; range, 0-501) and a mean citation rate of 15.0 (SD, 16.1; range, 0-178). Articles in AJSM (β = 19.9; P < .001), publications from North America (β = 8.5; P = .033), and studies concerning measure validation/reliability (β = 25.5; P = .004) were independently associated with higher AAS. Greater AAS score significantly predicted a greater citation rate (β = 0.16; P < .0001). The citation rate was an independent predictor of greater social media attention on Twitter, Facebook, and the news (odds ratio range, 1.02-1.03; P < .05 all).
Conclusion:
AAS had a significant positive association with citation rates of articles in 5 high-impact orthopaedic journals. Articles in AJSM, studies concerning measure validation and reliability, and publications from North America were positively associated with greater AAS. A greater number of citations was consistently associated with publication attention received on social media platforms.
Citation rate and journal impact factor reliably indicate literature quality and have traditionally been the standardized metrics for quantifying academic impact.6,7,17,18 However, citation rate and impact factor metrics have several drawbacks. Journal impact factor obscures differences in individual article citation rates, is highly variable and dependent on research discipline, and fails to represent article impact beyond the sphere of academics.25,28 Recently, alternative web-based metrics, named altmetrics, have been created owing to the emergence and popularity of social media. 15 Altmetric the company tracks and quantifies the dissemination of research on various social media platforms, such as Twitter, Facebook, and online news stories, to generate an aggregate score called the Altmetric Attention Score (AAS).2,3,28 Various potential benefits of using altmetrics as opposed to traditional bibliometrics have been proposed: altmetrics allow for the assessment of publication impact beyond academia and the evaluation of different types of less commonly cited scholarly content, such as data sets and software. In addition, altmetrics provide real-time feedback on impact on the order of days to weeks, as opposed to several years, after publication. 6 Given the increasing role of social media in disseminating research, it is essential that the use of web-based metrics, such as the AAS, and their association with traditional bibliometrics be understood.
Previous studies have investigated AAS trends in the fields of neurology, 15 neurosurgery,14,28 neuroimaging, 13 oncology, 11 anaesthesia, 9 and emergency medicine. 5 Correlation statistics between the AAS and citation counts have ranged from moderately high16,21 to fairly weak,8,12 whereas the AAS and journal impact factor have been found to be weakly associated.9,15,21 Xia et al 29 analyzed the correlation between tweets and citations of articles in Nature across differing publication years, disciplines, and Twitter user types and reported a discrepancy between the most cited articles and most tweeted articles, suggesting that altmetrics and citation-based metrics may be assessing different types of impact. However, research containing important social and cultural implications may be otherwise neglected by traditional citation-based and impact factor metrics, 22 and the significance of the AAS may vary by field. In other words, because social media is a growing and inherent part of society and research and because the citation rate (a traditional metric) was not designed to reflect the ability to access and popularize research through multiple public media (Twitter, Facebook, online news, etc), it is important to better understand how research is being accessed and appraised through such media.
Despite emerging research in other fields of medicine and science, the role of the AAS, factors that influence the AAS, and the relationship of the AAS to traditional bibliometrics in the field of orthopaedic surgery remain poorly understood. The purpose of the current study was to (1) assess the extent to which the AAS correlates with traditional bibliometrics and (2) identify characteristics of published articles that predict greater AAS in the orthopaedic surgery literature. We hypothesized that the AAS would be significantly associated with the citation rate of an article and that several publication characteristics would positively or negatively influence the AAS.
Methods
Study Eligibility and Selection
The current study was exempt from institutional review board approval. Electronic versions of articles published in The American Journal of Sports Medicine (AJSM), The Journal of Bone and Joint Surgery (JBJS), Clinical Orthopaedics and Related Research (CORR), Acta Orthopaedica, and Knee Surgery, Sports Traumatology, Arthroscopy (KSSTA) were obtained between January 2016 and December 2016. Specifically, 4 issues from the beginning of each quarter were chosen to obtain a representative sample of all articles published throughout the entire 2016 calendar year and to allow for a large sample size relative to other studies in medicine that have studied almetrics.5,11,13-15,21,28 This year and these 5 journals were chosen on the basis of (1) having the highest journal impact factors in the field of orthopaedic surgery 17 in 2016 and (2) having reported the AASs for their published articles. The year 2016 was chosen to allow sufficient time for articles to be disseminated among the research community and to accrue citation rates (the total number of citations that each article accrued since its publication) and AAS. Citation rate and AAS for each article were collected simultaneously in July 2019. Previous research has demonstrated that altmetrics are particularly responsive to the online attention surrounding more recent publications, 14 whereas other studies have suggested that a citation window of 3 to 5 years is most appropriate for analysis of citation rate. 1
Within the sampled studies, the following published article categories were applied for inclusion criteria: basic science/computer models, prospective randomized controlled trial, prospective cohort, prospective case series, retrospective cohort/case-control, retrospective case series, case study/case report, cross-sectional, systematic review/meta-analysis, concept review, and technical notes. Published articles classified as expert opinions, letters to the editor, technique tips, and responses were excluded from analysis.
Data Collection
Collected variables for all included published articles were determined a priori: journal, issue, article title, date of publication, highest degree of first author, number of authors, number of academic institutions, geographic region of origin of the publication, disclosure of any conflict of interest (presence or absence of general self-reported conflict of interest), subject of study, study design, sample size, number of referenced studies, and number of citations. Subjects of study included the following: biomechanics/basic science, technology/technical descriptions, imaging, knee, shoulder, trauma/fracture, elbow, hip, biopsychosocial, socioeconomic/costs, nonoperative treatment, measure validation/reliability, foot and ankle, pediatric/adolescent, epidemiology, and other study subjects that did not correspond to the aforementioned categories. All included variables were based on previous studies that investigated factors associated with the citation rate in the orthopaedic surgery literature. 18 Level of evidence was assessed according to guidelines proposed by Sackett et al. 24
Determination of the AAS
The AAS is an automatic algorithm that was created by Altmetric, and it is calculated through weighted scores of social media attention that a given published article receives. 2 Altmetric began tracking online attention regarding published research in 2011 and provides analysis of activity on various web-based sources, including Twitter, Facebook, policy documents, news stories, blogs, Mendeley, Reddit, Wikipedia, Google+, and others. 3 Additional variables collected for each published article are as follows: AAS, number of tweets, number of Facebook mentions, and number of news stories mentioning the published article. Mentions were procured from the AAS. All queried articles reported altmetrics information through each journal’s website, and all journals used the same Altmetric score.
Statistical Analysis
Statistical analysis was performed with Stata statistical software (v 16.0; StataCorp). Publication characteristics and social media–related information were summarized with descriptive statistics. Continuous variables were quantified with means and standard deviations, while binary variables were quantified as relative frequency with percentages. Normality was determined with the Shapiro-Wilk test. Multivariate linear and logistic regressions were performed to determine (1) publication characteristics predictive of AAS and social media attention (mentions on Twitter, Facebook, and the news) and (2) the relationship between AAS and citation rate. Secondarily, we sought to evaluate the influence of publication characteristics on the likelihood of being mentioned on Twitter, Facebook, and the news, as well as the relationship between the AAS and citation rate. Statistical significance was defined as P < .05.
Results
Article Characteristics
A total of 496 journal articles published in 2016 across the 5 journals were included in the final analysis after 32 were excluded from AJSM, 28 from JBJS, 92 from CORR, 14 from Acta Orthopaedica, and 10 from KSSTA. Publication characteristics are quantified in Table 1. Of these articles, 120 (24.2%) were published in AJSM, 41 (8.3%) in JBJS, 80 (16.1%) in CORR, 66 (13.3%) in Acta Orthopaedica, and 189 (38.1%) in KSSTA. The mean AAS among all articles was 8.6 ± 31.7 (range, 0-501), and the mean citation rate was 15.0 ± 16.1 (range, 0-178).
Publication Characteristics of Included Journals a
Values are presented as n (%) or mean (95% CI) [range]. AJSM, The American Journal of Sports Medicine; COI, conflict of interest; CORR, Clinical Orthopaedics and Related Research; JBJS, The Journal of Bone and Joint Surgery; KSSTA, Knee Surgery, Sports Traumatology, Arthroscopy; NA, not applicable; RCT, randomized controlled trial.
Other author degrees include BS, MS, PhD, MD PhD, MD MBA, MD MS, MD MPH, and PT.
Level of evidence determined from criteria established by Sackett et al. 24
AAS Analysis: Influence of Journal Selection
Analysis of variance testing indicated a significant between-group effect in terms of the relationship between AAS and journal (P < .0001), as well as citation rate and journal (P < .0001), with articles in AJSM having the highest AAS and citation rate among all journals on average (Table 2).
Mean Altmetrics Scores and Citation Rates Among Included Journals a
Values are presented as mean ± SD. AAS, Altmetric Attention Score; AJSM, The American Journal of Sports Medicine; CORR Clinical Orthopaedics and Related Research; JBJS, The Journal of Bone and Joint Surgery; KSSTA, Knee Surgery, Sports Traumatology, Arthroscopy.
Linear and logistic regression models were subsequently constructed to determine the predictive value of journal selection alone on AAS. Linear regression analysis indicated that publication in AJSM was the strongest predictor of greater AAS (β = 19.9; 95% CI, 14.4-25.4; P < .001), whereas publication in CORR (β = −12.5; 95% CI, −21.3 to −3.8; P < .005), Acta Orthopaedica (β = −17.9; 95% CI, −27.2 to −8.7; P < .0001), and KSSTA (β = −18.0; 95% CI, −25.1 to −10.9; P < .0001) was a significant predictor of lower AAS. Publication in JBJS was not significantly associated with the AAS (P = .90).
This analysis was also conducted to determine whether the journal selection was a significant predictor of achieving greater than the mean AAS (8.6), with publication in AJSM set as the reference variable given that it had the greatest mean AAS at baseline. Publication in JBJS (P = .087) and CORR (P = .056) was not significantly predictive of achieving greater than the mean AAS. However, publications in Acta Orthopaedica (odds ratio [OR], 0.08; 95% CI, 0.024-0.28; P < .0001) and KSSTA (OR, 0.13; 95% CI, 0.07-0.27; P < .0001) had significantly lower odds of achieving greater than the mean AAS.
AAS Analysis: Influence of Publication Characteristics
Multivariate linear and logistic regression models incorporating all collected publication characteristics were constructed to determine which modifiable characteristics were independently associated with greater AAS in addition to achieving greater than the mean AAS. The multivariate linear regression model indicated that publications from North America (β = 8.5; 95% CI, 0.7-15.7; P = .033) and studies concerning measure validation and reliability (β = 25.5; 95% CI, 8.1-42.8; P = .004) were independently associated with achieving a greater AAS. The multivariate logistic regression model indicated that only publications from North America (OR, 2.7; 95% CI, 1.4-5.2; P = .003) conferred a higher likelihood of achieving greater than the mean AAS.
Predictors of Social Media Attention
A series of multivariate logistic regression models were constructed to determine independent predictors of the likelihood of a publication’s being mentioned on Twitter, Facebook, and online news articles. A greater number of citations (OR, 1.03; 95% CI, 1.01-1.07; P = .004) was independently associated with a higher likelihood of a publication being mentioned on Twitter. Number of citations (OR, 1.02; 95% CI, 1.0-1.04; P = .045), publications concerning trauma or fracture management (OR, 9.1; 95% CI, 1.1-54.3; P = .040), and cross-sectional studies (OR, 10.9; 95% CI, 1.9-63.7; P = .008) were independently associated with a higher likelihood of a publication’s being mentioned on Facebook. A greater number of citations (OR, 1.02; 95% CI, 1.0-1.05; P = .038), publications from North America (OR, 4.4; 95% CI, 1.09-18.1; P = .038), and a greater number of authors (OR, 1.68; 95% CI, 1.4-1.9; P = .026) were independently associated with a higher likelihood of a publication being mentioned in the news.
Relationship Between Citation Rate and AAS
Bivariate correlation analysis indicated that the AAS had a weak positive correlation with the citation rate of an article (r = 0.32; P < .0001). A subsequent linear regression analysis (Figure 1) evaluating the influence of the AAS on citation rate alone indicated that the AAS was positively associated and weakly predictive of the citation rate (β = 0.16; 95% CI, 0.12-0.21; P < .0001).
Linear regression model for Altmetric Attention Score and citation rate.
Discussion
The main finding of the current study was that the AAS had a positive weak association with the citation rate of a publication. Other principal findings of the current study were as follows: (1) The mean AAS among the 5 journals was 8.6 (range, 0-501). (2) The journal was significantly associated with the AAS, with publications in AJSM conferring the greatest positive influence on achieving a greater AAS. (3) The AAS had independent positive associations with studies concerning measure validation and reliability, as well as with publications from North America. (4) A variety of publication characteristics were independent predictors of an increased likelihood of a publication’s being mentioned on social media platforms, with a greater number of citations having the most consistent association.
The AAS of a publication demonstrated a significant positive correlation with the citation rate of an article and was found to be a predictor of a greater number of citations. It should be noted that the beta coefficient for this finding was relatively low at 0.16, and this may not be clinically significant. However, this is in accordance with a recent systematic review of medical research that also found a significant positive correlation between traditional citation rate and altmetrics, 19 as well as other areas of medicine that have identified this relationship.5,23 This finding suggests that the AAS represents a quantifiable assessment of a publication’s impact and ability to attract a larger readership within its niche scientific community. It is possible that promotion through multiple social media platforms expands the number of readers who are ultimately reached. Altmetrics have been framed as a complement to the conventional metrics to assess immediate impact given how quickly research can be shared online as compared with traditional metrics (eg, citation rate), which require subsequent publications. 4 A recent study concerning pharmacoepidemiologic research concluded that altmetrics confer a more comprehensive evaluation of research impact, as they are able to incorporate the activity and engagement of scholarly and nonacademic communities alike. 10 Future research will need to focus on better delineating the impact of altmetrics in the age of growing social media, as it appears that altmetrics may have a large role in disseminating publications regarding the orthopaedic surgery literature.
Although one hypothesis for the relationship between AAS and the citation rate is generating a larger and more diverse readership, it is worth noting an alternative hypothesis that may also explain this finding. Another hypothesis is that the efforts made by publishers and editorial boards of journals generate social media traffic. Indeed, in recent years, many articles have gained 1 additional citation solely by an editorial board member performing a commentary on each published article in that particular volume. 14 According to a recent review article, 27 the number of physicians utilizing social media for professional purposes has increased to 90%, and these online profiles represent important avenues for sharing and discussing research. Furthermore, publishers and journals are now posting videos, blogs, webcasts, and podcasts to further publicize their own research. A recent study found that journals with social media accounts in the field of neurosurgery possessed significantly higher Altmetric scores for their articles as compared with journals without a social media presence. 28 Such efforts likely directly influence the AAS through “social media buzz” by the same readers and researchers ultimately involved in generating citations. As the AAS continues to be used to evaluate the social media attention that an article receives, future research should seek to determine how much of this contribution is made by the authors of the article, journals, and editorial personnel responsible for publishing the article. Furthermore, effort should be made to determine and compare the range and degree of efforts among these journals.
Across the 5 journals and 496 articles included, the mean AAS was low at 8.6. However, other areas of medicine have reported lower AASs, including neurosurgery (4.7), 29 radiology (3.3), 23 and emergency medicine (1.9). 5 In the current study, AAS was significantly associated with the journal, with articles published in AJSM obtaining the highest AAS. As the internet continues to evolve as a growing source for health care information—as evidenced by recent statistics estimating that 84% of adults use the internet on a regular basis and 60% of those users report having accessed health-related information online in the previous month 20 —the use of the internet to share research findings is becoming an important aspect of reaching a journal’s readership.
In the current study, the AAS was influenced by numerous journal characteristics that are modifiable and may allow for authors to influence the extent to which their articles are shared among popular online media. Specifically, studies concerning measure validation and reliability (β = 25.5; P = .004) as well as those published in North America (β = 8.5; 95% CI, 0.7-15.7; P = .033) were independently associated with achieving a greater AAS. The finding that publications from North America are significantly associated with the AAS is not surprising, given that 3 included journals had the majority of their publications from North America and previous studies have demonstrated that subject of study and publications from North America are associated with higher citation rates 18 and AAS. 9 Considering that the current study identified a weak positive association between AAS and citation rate, this is a plausible and logical finding. The association among citation rate, AAS, and publications from North America is likely a reflection of high rates of publication and institutional preference for research performed in the United States. 21 Authors of future studies may attempt to maximize the impact of their publications through emphasizing these characteristics such that their work has more exposure on social media as measured by altmetrics scoring.
Specifically, a number of popular social media platforms, such as Facebook, Twitter, and online news articles, are incorporated into calculating the AAS and are accessed by members of the orthopaedic community, as well as the general population. The current study identified independent predictors of maximizing the likelihood of a publication’s being mentioned on these platforms: (1) a greater number of citations was the most consistent factor in that it was independently associated with a higher likelihood of being mentioned on Twitter, Facebook, and the news; (2) publications concerning trauma or fracture management and cross-sectional studies were independently associated with a higher likelihood of a publication’s being mentioned on Facebook; and (3) publications from North America and a greater number of authors were independently associated with a higher likelihood of a publication’s being mentioned in the news. Notably, the OR for the predictive performance of the citation rate on social media attention ranged from 1.02 to 1.03, which may not be clinically meaningful. However, this is in accordance with other medical literature that has identified associations between social media exposure on Facebook and Twitter and the number of citations that a publication receives. 26 This relationship may be the manifestation of publications that ultimately have a higher impact gaining more social media exposure.
There are limitations of the current study that should be mentioned in the context of the interpretation of the findings. The current study examined only a small subset of the orthopaedic surgery literature, as a much larger number of journals exist. However, we sought to examine the orthopaedic literature of the highest impact that also used altmetrics, and a number of journals were excluded, as the use of altmetrics scoring or the methods of their calculation of an aggregate altmetrics score were not transparent. Future studies should examine the use of altmetrics as more journals trend toward using this method of publication evaluation. Another limitation is the unequal number of publications procured for analysis from each journal; however, this is a limitation inherent to each journal, for which we could not control in the analysis. Finally, the current analysis examined only a subset of publication characteristics, although these specific characteristics were used in previous literature and deemed feasible for appropriate analytic use and, therefore, were included in this study.
Conclusion
The AAS had a significant positive association with the citation rates of articles in 5 high-impact orthopaedic journals. Publications in AJSM, studies concerning measure validation and reliability, and publications from North America were positively associated with greater AAS. A greater number of citations was consistently associated with publication attention received on social media platforms.
Footnotes
Submitted August 19, 2019; accepted December 4, 2019.
One or more of the authors has declared the following potential conflict of interest or source of funding: B.U.N. has received educational support from Smith & Nephew and hospitality payments from Stryker, Wright Medical, and Zimmer Biomet. J.C. is an unpaid consultant for Arthrex Inc, Smith & Nephew, and CONMED Linvatec and has received educational support from Arthrex Inc and Smith & Nephew. S.J.N. receives publishing royalties and financial or material support from Springer and Ossur; is a paid consultant for Ossur and Stryker; receives research support from Allosource, Arthrex Inc, Athletico, DJ Orthopaedics, Linvatec, Miomed, Smith & Nephew, and Stryker; and has received educational support from Elite Orthopedics. AOSSM checks author disclosures against the Open Payments Database (OPD). AOSSM has not conducted an independent investigation on the OPD and disclaims any liability or responsibility relating thereto.