Abstract
Background
Traditional measures of evaluating scholarly output do not capture the impact social media can provide in disseminating and promoting research. We sought to better understand the level of online attention that high-quality hand research received.
Methods
Scientific manuscripts published from 2017 in Journal of Hand Surgery (American Volume) (JHS-A), Journal of Hand Surgery (European Volume) (JHS-E), and HAND were recorded. Manuscript characteristics were determined, including the number of citations. Altmetric Attention Score (AAS), a measure of a manuscript’s online attention and impact, was determined, as well as Twitter mentions, Facebook mentions, and news outlet mentions. Spearman rho (ρ) correlation coefficient was used to evaluate the relationship between the number of citations and AAS. Multivariable linear regression analysis was performed to determine which manuscript factors were associated with AAS.
Results
A total of 323 manuscripts were identified. There was a weak positive correlation between the number of citations and AAS; however, this relationship did not exist for each individual journal. Publication in HAND and JHS-E were associated with lower average manuscript AAS when using JHS-A as the reference group. Two additional factors were also associated with increased manuscript AAS: (1) being a clinical study focused on a specific upper extremity anatomical location; and (2) increasing number of institutions on a study.
Conclusions
Publication in HAND and JHS-E were associated with lower manuscript AAS when using JHS-A as the reference group, suggesting that HAND and JHS-E have room for improvement in using social media to share their high-quality hand surgery scientific articles.
Introduction
Historically, the success of basic science and clinical hand surgery researchers has been determined by the quantity of peer-reviewed publications in journals indexed in PubMed. 1 Current incentives—for medical students, residents, and faculty—are aligned to promote publication quantity over quality; however, there is a growing call in orthopedics to focus on quality. 2 From a journal’s standpoint, impact factor (IF)—a measure of the frequency with which articles published in a given journal are cited in the literature—has long been considered the “gold standard” used to evaluate journal quality. 3 While these are reliable and relatively easy to determine quality metrics, such measures may be missing other factors that should be considered when evaluating research and scientific journal quality. These traditional metrics are typically only important within academia and take a notable amount of time (eg, years) to appreciate. 4 In addition, the influence and discussion of research on and by colleagues, as well as its “reach” to a wider, general audience, is not considered with metrics like IF.
With the growing popularity of social media, both generally and among orthopedic surgeons,5,6 there is increasing interest in its role in disseminating scholarly work and creating “buzz” around scientific publications. In 2011, this idea led to the founding of Altmetric, an independent, commercial technology company under the parent company Digital Science (London, United Kingdom) with a mission “to track and analyze the online activity around scholarly research outputs.” 7 One way in which the goal to track and analyze scientific work online materializes is via the Altmetric Attention Score (AAS), which is calculated based on the level of propagation of research on social media platforms and news outlets. While the AAS has not yet been validated, increasing interest in AAS, including its association with classic bibliometric measures, has led to research across a number of medical fields, including neurology, 8 radiology,9,10 emergency medicine, 11 urology, 12 pediatric surgery, 13 and orthopedic surgery, 14 among others. The findings in previous studies show weak to moderately high correlations between citations and AAS, 14 with 1 study identifying a weak correlation between AAS and IF. 13 However, to date, we are unaware of such research within the hand literature.
In addition to understanding factors associated with conventional research quality metrics such as IF, there is growing value and interest in understanding factors associated with alternative quality measures, such as AAS, as well as AAS’ relationship to traditional bibliometrics. Furthermore, many high-quality, peer-reviewed hand surgery publications have sought to grow their presence online, including social media. However, this focus is quite new. Thus, in this preliminary study, we asked the following study questions: (1) What is the average AAS, presence of Twitter mentions (ie, tweets), presence of Facebook mentions, and presence of news outlet mentions for original scientific manuscripts published in Journal of Hand Surgery (American Volume) (JHS-A), Journal of Hand Surgery (European Volume) (JHS-E), and HAND? (2) What is the correlation between AAS and the number of PubMed/National Center for Biotechnology Information (NCBI) citations? and (3) What manuscript characteristics are associated with AAS scores?
Materials and Methods
This study used publicly available data; thus, institutional review board approval was not required.
Original scientific manuscripts, including systematic reviews and meta-analyses, published from January to December 2017 in the JHS-A, JHS-E, and HAND were identified from official journal websites.15-17 The decision to include only these 3 journals was made before data collection. We sought to focus our work on the most commonly read hand surgery journals that were also the official scientific publications of major United States and European hand societies. For example, JHS-A is the official journal of the American Society for Surgery of the Hand, JHS-E is the official journal of the Federation of European Societies for Surgery of the Hand and British Society for Surgery of the Hand, and HAND is the official journal of the American Association for Hand Surgery (and official English language journal of the Asociacion Argentina de Cirugia de la Mano, Brazilian Society of Hand Surgery, and the Romanian Society for Surgery of the Hand).
To ensure consistency across journals, publication in 2017 was considered if the scientific article was published in the official journal issue during that calendar year. Thus, manuscripts that were posted on each journal’s official website or PubMed in 2017 (eg, “Epub Ahead of Print”) but not in an official journal issue were excluded. In addition, all nonoriginal scientific articles (eg, editorials, commentaries, general/concept review articles) were excluded. The year 2017 was chosen to allow for at least 2 full years for all included articles to accrue citations and be shared online via different platforms.
Once all appropriate manuscripts were identified, the following article characteristics, which were determined a priori, were collected directly as stated, whenever possible, from each manuscript: article title; journal (JHS-A, JHS-E, or HAND); issue number; month of issue publication; number of authors; number of academic institutions involved in each study; geographic region (North America or other, including Europe, Asia, Africa, South America, and Oceania); presence or absence of a self-reported conflict of interest; subject of study (biomechanics/basic science; technology/technical descriptions; imaging; general upper extremity trauma/fracture; general upper extremity; hand/wrist; forearm; elbow; shoulder; biopsychosocial; socioeconomic/cost analyses; measure validation/reliability; pediatric/adolescent; epidemiologic; other); manuscript-reported study design (basic science/computer model; prospective randomized controlled trial; prospective cohort; prospective case series; retrospective cohort/case-control; retrospective case series; cross-sectional; Delphi study/expert consensus; technical note; systematic review/meta-analysis); level of evidence; sample size; and number of references. Of note, HAND does not publish level of evidence; thus, it was recorded as “Other/Not Reported.”
In addition to article characteristics, the following data points were collected from April 26, 2020, to May 2, 2020: number of citations, number of tweets, number of Facebook mentions, number of news outlet mentions, and AAS. All data collected in this time period reflected the total social media mentions and AAS accrued since the publication date of each individual article.
The AAS is calculated using an automated algorithm, and it is a “weighted count of all of the attention a research output has received.” 18 For example, a mention of a research output in a news report is weighted much more than a single tweet. 19 This is because the assumption made as part of the AAS automated algorithm is that inclusion in an average news report is much more likely to bring attention to the research output than an average tweet. 19 Furthermore, the AAS is always a whole number, so rounding does occur. Importantly, the AAS is a measure calculated for each manuscript not for a journal overall. Thus, the reported AAS in this manuscript for each journal is the average score per article published. To determine the AAS for each manuscript, the free Altmetric “Bookmarklet for Researchers” was used. 20 To use this tool, each article page was visited and the Altmetric bookmarklet was clicked to automatically return article-level metrics. The numeric AAS was recorded. If Altmetric returned the calculation of a manuscript’s AAS as “?” with the message, “Altmetric hasn’t picked up any sharing activity around this article yet,” then a score of zero was recorded.
Number of tweets, number of Facebook mentions, and number of news outlet mentions were determined from the AAS. In lieu of using the reported AAS number of citations, the number of citations from the PubMed/NCBI was used. This was to capture the citation rate most commonly considered by basic science and clinical researchers.
Statistical Analysis
Descriptive statistics for all articles, as well as for each journal’s manuscripts, were calculated and reported. Continuous variables were reported as means, standard deviations (SD), and ranges, while categorical variables were reported as counts and percentages. Spearman rho (ρ) correlation coefficient was used to evaluate the correlation of AAS and PubMed/NCBI citations. A multivariable linear regression analysis was conducted to determine which manuscript factors are associated with AAS. For the multivariable linear regression analysis, the level of evidence was excluded because of the inconsistency with which data were available. In addition, subject of study was split into 3 categories: biomechanics/basic sciences; clinical study by anatomical location (hand/wrist; forearm; elbow; shoulder); and other. Significance was set a priori at P < .05.
Results
A total of 323 manuscripts met inclusion criteria, with 151 (47%), 94 (29%), and 78 (24%) manuscripts published in JHS-A, JHS-E, and HAND, respectively (Table 1). Across all journals, the AAS was 3.6 (SD: 5.7; range: 0-59) and a majority of manuscripts had Twitter mentions (n = 210 [65%]) (Table 1). Less than half of all manuscripts had a Facebook mention (n = 144 [45%]) and only 6 manuscripts (1.9%) had a news outlet mention (Table 1). The average number of PubMed/NCBI citations across all included manuscripts was 7.2 (SD: 7.0; range: 0-53) (Table 1). The average AAS for manuscripts published in JHS-A, JHS-E, and HAND was 5.0 (SD: 6.0; range: 0-59), 3.1 (SD: 4.5; range: 0-23), and 1.4 (SD: 5.7; range: 0-48), respectively (Table 1).
Comparison of Manuscript Characteristics by Journal (n [%] or mean [SD] [Range]).
Note. RCT = randomized controlled trial; AAS = Altmetric Attention Score; NCBI = National Center for Biotechnology Information; COI = conflict of interest.
When evaluating the correlation between AAS and the number of manuscript citations across all hand journals, there was a small positive correlation appreciated (ρ = 0.21; P < .05). However, when analyzed individually, JHS-A (ρ = 0.15; P = .06), JHS-E (ρ = 0.07; P = .48), and HAND (ρ = 0.19; P = .10) demonstrated no significant correlation between AAS and the number of manuscript citations (Table 2).
Correlation of Altmetric Attention Score and Number of Manuscript Citations.
In multivariable linear regression analysis, scholarly work published in HAND and JHS-E was associated with lower AAS (HAND: β: −4.3 [95% confidence interval {CI}, −5.9 to −2.7], P < .05; JHS-E: β: −2.2 [95% CI, −4.0 to −0.4], P < .05) when using JHS-A as the reference group (Table 3). In contrast, clinical study by anatomical location (β: 1.9 [95% CI, 0.04 to 3.8]; P < .05) and the number of institutions (β: 0.8 [95% CI, 0.2 to 1.3]; P < .05) were associated with higher AAS (Table 3).
Manuscript Factors Associated or Not Associated With Altmetric Attention Score.
Note. CI = confidence interval; JHS-A = Journal of Hand Surgery (American Volume); JHS-E = Journal of Hand Surgery (European Volume); COI = conflict of interest.
Discussion
Typically, academic success for basic science and clinical hand surgery researchers has been measured by the number of peer-reviewed publications, with manuscripts that are more often cited and/or in high IF journals providing additional possible benefit. In addition, journal editors are often concerned about their publication’s IF as a measure of quality. While such metrics are important considerations, the growth of social media has created a new space where scholarly work can be disseminated and has a positive impact. This has led many high-quality, peer-reviewed hand surgery publications to seek out avenues to grow their presence online, including social media. One way to measure the attention research gains across the Internet, including social media, in real time is with the AAS. In this preliminary study of original scientific articles published in 3 major hand journals over the 2017 calendar year, we determined the average manuscript AAS (overall and by journal); demonstrated that citation rate and AAS were positively, weakly correlated for manuscripts published across hand surgery journals but not for any specific journal; and established that being published in HAND and JHS-E was associated with a lower AAS, whereas being a clinical study focused on a specific upper extremity anatomical location and increasing the number of institutions involved in a given study were associated with an increased AAS. Importantly, through targeted efforts to expand their social media presence, which appears to be currently underway, it is certainly possible for publication in HAND and JHS-E to become associated with higher AAS. However, this study provides a starting point from which to assess future success or failure of initiatives in this area.
To get a full picture of the impact of scholarly work, it may be beneficial to consider AAS in conjunction with traditional metrics, such as citations and IF. Thus, it is of value to better understand the relationship between the 2. In this study, we found that AAS and the number of citations demonstrated a small correlation for manuscripts published in hand surgery journals overall, but not for any specific journal. Indeed, no such correlations existed for JHS-A, JHS-E, or HAND. Similar to our findings, an analysis of publications in 5 orthopedic journals (3 general orthopedic surgery journals and 2 sports medicine orthopedic journals) demonstrated a weak positive correlation between AAS and the number of citations. 14 Indeed, no correlation or a weak to moderate positive correlation between AAS and the number of citations has been appreciated in published literature on the topic across different medical specialties.21-23 The no correlation to weak positive correlation results suggest that AAS and the number of citations may be capturing different measures of scholarly work impact. Interestingly, this also seems to be the case when comparing AAS with IF. Indeed, 1 study identified only a weak correlation between Altmetric scores and IF, 13 further confirming that the 2 measures evaluate different areas of impact. Therefore, considering both AAS and traditional scientific metrics such as the citation rate and IF when evaluating the true overall reach and influence of publications may be most appropriate.
In total, we found 2 factors associated with manuscripts having a lower AAS: (1) publication in HAND with JHS-A as the reference journal; and (2) publication in JHS-E with JHS-A as the reference journal. We also found 2 factors associated with manuscripts having a higher AAS: (1) clinical studies by anatomical location; and (2) number of institutions involved in producing a given study/manuscript. We hypothesize that the finding that publication in HAND or JHS-E is associated with manuscripts having a lower AAS may be secondary to a growing and active online presence of JHS-A and its editorial team. For example, JHS-A is active on Twitter. Indeed, not only does JHS-A promote their recent publications and encourage discussion and sharing of manuscripts—as the other journals do as well—but they also tweet eye-catching visual abstracts. Furthermore, the JHS-A Editor-In-Chief publishes a monthly video reviewing the most recent journal issue and hosts podcasts on “Editor’s Choice” manuscript, as well as fosters scientific debate and discussion via a Twitter journal club using the hashtag #JHSJC. 15 However, the use of visual abstracts and other social media endeavors, such as Twitter journal club discussions, does not appear to be used by JHS-E or HAND; this could explain differences found between JHS-A, JHS-E, and HAND. However, it is crucial to remember that these findings are quite preliminary, as AAS is a novel measure of online attention and social media engagement of scientific work, and hand surgery journals are still determining how best to optimize AAS for manuscripts published in their journals. Thus, future work is certainly warranted in this area, but this study helps set an important baseline from which to move forward—both for researchers and journal (eg, HAND or JHS-E) editorial teams. In addition, we feel the other 2 factors (clinical studies by anatomical location and number of institutions involved in producing a given study/manuscript) with high AAS intuitively make sense. Clinically active hand surgeons are more likely interested in clinical studies that may directly affect their daily practice; thus, they may be more willing to share such articles online. In addition, we believe the general population is more likely to share nonbasic science studies that are potentially more easily understood. Finally, as authors from more institutions become involved in a given project that leads to publication, there is more opportunity for different institutions to help share the findings online.
This study has limitations that readers should consider when evaluating our findings. First, while we evaluated original scientific articles in 3 well-regarded hand surgery journals, our findings may not reflect all the hand surgery literature. Nonetheless, we believe our work provides important insight on the larger reach and impact of manuscripts from 3 hand surgery journals that many researchers seek to publish in. Second, not all social media platforms that play a role in disseminating research within the field of hand surgery are incorporated into the AAS. For example, Instagram is another commonly used platform that was not included in the calculation of AAS. However, AAS incorporates many of the key platforms, including Facebook and Twitter; therefore, we believe that our findings still provide valuable insight into the role of social media in hand surgery knowledge dissemination. Third, for consistency reasons, we only used the available information provided by each journal. However, this led to incomplete data for certain characteristics, such as the level of evidence. We believe the impact of level of evidence and sample size on understanding AAS is likely quite limited though. Fourth, AAS, number of Twitter mentions, number of Facebook mentions, number of news outlet mentions, and number of citations were all determined within the timeframe of a single week. Because such numbers can change over time, our findings may not remain consistent. However, we believe drastic changes in AAS or number of citations are unlikely long term. Furthermore, we were unable to assess the social media impact for a standard amount of time following publication due to the limitations of the Altmetric “Bookmarklet for Researchers,” which may have introduced time-dependent bias. However, by evaluating manuscripts from 2017, we believe we accomplished 2 key goals: (1) ensured adequate time for citations to accrue; and (2) provided enough time for manuscripts to be shared online via different platforms. We wanted to have a minimum of 24 months following publication for all analyses. In addition, because the publication dates and number of issues are not the same across all 3 journals, we chose to include all articles published during the entire 2017 calendar year to remove potential bias. Thus, while there may be slight variations in our findings over time or in different years, we believe the core takeaway messages will likely remain constant. Fifth, we recognize that not all articles are cataloged in the PubMed database and may therefore not contribute to the number of citations counted for the articles included in this study. Of note, international articles may only be found in Scopus or Web of Science. However, by using PubMed, a widely used, free search engine with millions of citations from the medical literature, we believe we captured most of the relevant citations. In addition, while it is important to acknowledge that authors may often cite their own work, we believe that the effects of this on the overall correlation between citation count and AAS would be limited and that the key takeaways of this preliminary study would remain the same. There may be other factors not part of a manuscript or journal itself that could affect AAS. Because AAS does not distinguish between tweets made by the authors themselves and those made by readers, the level of social media use of a hand surgeon and his or her coauthors, as well as their institutions’ investment in promoting research through social media, could have an impact on AAS. However, we believe that our key findings would still remain consistent, though future study is needed. In addition to further exploring the correlations between AAS and traditional metrics, it may be of particular interest to explore the relationship between AAS and scientific quality of research. Future studies may also seek to explore AAS trends over a longer period of data collection. Finally, the promotion of an article through social media does not ensure it is a good article or any better than those that have not been promoted through these means, nor does it reflect the quality of the journal. However, it does serve as another method to promote an article among peers, patients, and others, as well as the journal in which the manuscript is published.
High-quality, methodologically sound research is critical to moving the field of hand surgery forward. Traditional metrics to evaluate the quality of manuscripts and journals, including the number of citations and IF, are important. However, with the growth of social media and other online platforms, these classic scholarly measures may not capture a manuscript’s full influence or breadth of dissemination. This is especially true as the number of physicians and patients active online continues to increase. 6 Thus, consideration of both traditional metrics and more novel measures, such as AAS, may be of value. We believe our preliminary findings in this study suggest that a focused effort by hand journals and their editorial teams to promote high-quality and innovative scientific work over social media, while also encouraging active dialogue and discussion, can not only boost AAS but also move the field of hand surgery forward.
Footnotes
Author Contributions
D.N.B. and W.C.H. researched the literature and conceived the study. D.N.B., M.A.R., and W.C.H. were involved in protocol development and study plan. D.N.B. and M.A.R. were involved in data extraction, cleaning, and analysis. D.N.B. and M.A.R. were involved in writing the first draft of the article. All authors reviewed and edited the article and approved the final version of the article.
Ethical Approval
This study used public information only. Therefore, ethical committee approval was not required. This study has been performed in accordance with the ethical standards in the 1964 Declaration of Helsinki. This study has been carried out in accordance with relevant regulations of the US Health Insurance Portability and Accountability Act.
Statement of Human and Animal Rights
All procedures followed were in accordance with the ethical standards of the responsible committee on human experimentation (institutional and national) and with the Helsinki Declaration of 1975, as revised in 2008 (5).
Statement of Informed Consent
This study used public information only. There was no evaluation of patient data in this study; thus, informed consent was not needed.
Statement of the Location Where the Work was Performed
The work was performed at the Department of Orthopaedics & Physical Performance, University of Rochester Medical Center, Rochester, New York.
Declaration of Conflicting Interests
The author(s) declared the following potential conflicts of interest with respect to the research, authorship, and/or publication of this article: D.N.B. reports grants from the American Orthopaedic Foot & Ankle Society and personal fees from the Institute for Strategy And Competitiveness at Harvard Business School and Horizon Therapeutics, outside the submitted work. The other authors declare they have no conflict of interest to disclose.
Funding
The author(s) received no financial support for the research, authorship, and/or publication of this article.
