Abstract
There is only one thing in the world that is worse than being talked about, and that is not being talked about.
Have you ever read an article in an orthopaedic journal and wondered how much attention it is getting in the medical community or wider society? When SAGE Publishing converted The American Journal of Sports Medicine (AJSM) and its affiliated publications, Sports Health: A Multidisciplinary Approach and The Orthopaedic Journal of Sports Medicine, to the Atypon electronic publishing platform at the close of 2016, a new world of article-specific metrics opened for our readers. For the first time, they could access a wealth of continuously updated information about the scientific and societal impact of each individual article. Some of these metrics are conventional: the number of times that an article has been downloaded and the full list of subsequent citations in Web of Science or Crossref. Others are more unconventional, alternative metrics of the article’s impact, such as mentions in Twitter or online blogs.
The specific set of alternative metrics featured in our journals is the one produced by the company simply known as Altmetric. Founded in 2011, Altmetric is not the only purveyor of alternative journal metrics, but it is one of the most widely used. The full range of individual article metrics can be viewed by clicking on the Article Metrics link in the Article Menu that appears on the left-hand side of the HTML version of every AJSM paper. Once there, the reader will find a continuously updated total of views and downloads, hot links to all citing articles, and the link to all Altmetric elements.
The most distinctive feature of the Altmetric link is its colorful “donut,” a swirling ring of bright hues surrounding a large numeral. Each color of the ring represents a different type of venue where the article has received a mention: conventional news outlets, online blogs, policy documents, YouTube videos, patents, Wikipedia, F1000, Reddit, Twitter, Facebook, and Mendeley. To the right of this donut sits a color key listing every venue type in which the article has appeared along with the running total of mentions in each. Every element of this key is a hot link that will take the reader to the original sources. Clicking on the Twitter link, for example, connects the reader to all the relevant tweets. From there, a click on the Summary tab brings up a world map displaying the geographic origin of the tweets as well as a demographic breakdown of the tweeters into categories including health practitioners; scientists; science communicators such as journalists, bloggers, and editors; and members of the public.
The numeral in the center of every donut is the article’s Altmetric Attention Score (AAS), the weighted total of all these mentions. Not all venues are treated equally: common weightings include 8 for a conventional news outlet, 5 for a blog, 3 for a Wikipedia article, 1 for a tweet, 0.85 for a retweet, and 0.25 for a Facebook post. Within these categories, the weightings are further tiered: a mention in a national news outlet counts more than one in a “niche publication,” and a tweet from the article’s publisher counts less than one from an unconnected member of the public. Saves by users of the Mendeley reference manager, although tracked, do not influence the AAS at all, although the geographic distribution, professional status, and discipline of the Mendeley readers are displayed. 7
Because many of these alternative measures of impact register quickly after the date of publication, many researchers have wondered whether they can predict the likelihood that a study will be cited in the future. In this issue of AJSM, Kunze and colleagues 9 explore possible relationships between alternative and traditional metrics of orthopaedic sports medicine research in response to the following question: “What Is the Predictive Ability and Academic Impact of the Altmetrics Score and Social Media Attention?” The authors selected 5 orthopaedic journals, including AJSM, that are highly ranked by impact factor and utilize the Altmetric platform. They focused on the 2016 volumes of these journals, so that a reasonable amount of time had passed for the articles to accumulate citations, and analyzed 496 papers selected from the 4 quarters of that year. A smorgasbord of variables was collected on each paper: journal, issue, article title, date of publication, highest degree of first author, number of authors, number of academic institutions, geographic region of origin of the publication, disclosure of any conflict of interest, subject of study, study design, sample size, number of referenced studies, cumulative number of citations, and AAS as of July 2019.
The mean AAS for all 496 articles was 8.6, but it ranged from 0 to 501. Similarly, the number of citations per article varied widely from 0 to 178, with a mean of 15. Among the factors studied, publication in AJSM, authors based in North America, and studies concerning measure validation or reliability were associated with a higher AAS. The average AJSM article had an AAS of 19.9 and attracted 23.2 citations. This latter figure is the mean number of total citations per article between the date of publication in 2016 and July 2019, a different metric from the impact factor, which is the mean number of citations per article in a restricted index of journals in 1 of the first 2 years following publication. Over all 5 journals, the AAS had a significant positive association with the number of citations, but the relationship was weak. When individual components of the AAS were examined, the number of citations was independently associated with the number of Twitter mentions, Facebook posts, and news articles. These relationships were also weak, with odds ratios ranging between 1.02 and 1.03.
These results are in alignment with similar studies in other medical specialties. Most, but not all, 8 of these studies have shown positive but weak correlations between an article’s AAS and subsequent citations.1,3,5,10,11,13,16 In 2017, Patthi et al 10 published a systematic review of medical research reported from 2010 through November 2016 that examined the possibility of an association between a variety of alternative metrics and traditional citation counts. After searching 5 major databases, they identified 7 pertinent studies. Finding that the correlation between alternative metrics and citations was positive but “feeble,” they concluded that these 2 types of impact measurements are “not entirely different from each other” but also that alternative metrics can be a “complement to the conventional citations.”
It is probably inevitable that readers and researchers would focus on a paper’s AAS as a shorthand summary of the article’s societal impact. This behavior is similar to the attention given to the 2-year impact factor in Journal Citation Reports, which is usually quoted in exclusion of the 5 other metrics reported for each indexed journal. Yet, the AAS is a composite score that reflects the attention paid to an article in a number of venues, each of which may have different contributing factors and correlations. For example, several studies have found that citations correlate specifically with the number of tweets that an article receives.3,15,16 In a 2016 bibliographic analysis of the prominent journal Nature based on Twitter and Facebook altmetrics data, Xia and colleagues 16 reported that correlations between tweet volume and citations varied among scientific disciplines; it was stronger in the biological sciences and earth and environmental sciences while actually negative in the chemical science category.
Altmetric does not factor the number of Mendeley users into the AAS because the company cannot display the accounts themselves. Nevertheless, this may be an important figure. It makes sense that researchers who bookmark an article in their Mendeley accounts might eventually cite the paper when they are writing a relevant publication. In fact, several studies have reported positive correlations between Mendeley saves and subsequent citations.2,4,5,13,15
While alternative metrics may bear some relationship to traditional ones, and may even influence them, they are, after all, alternative metrics that reflect different types of impact in a universe of stakeholders beyond medical professionals. This universe can be very large: a review in the Journal of Critical Care of the social media impact of sepsis studies uncovered tweets from 143 countries. 8 In the words of a group of pharmacoepidemiological researchers, “Despite the limited theoretical underpinnings, possible commercial influence, potential for gaming and manipulation, and numerous data quality-related issues, altmetrics are promising as a supplement to more traditional citation-based metrics because they can ingest and process a larger set of data points related to the flow and reach of scholarly communication from an expanded pool of stakeholders.”6(p93)
Alternative metrics tell us a lot about the impact of a scientific study, but they still may not tell the entire story. In 2018, a group of Spanish authors from the fields of journalism, psychology, and sociology pointed out the distinction between dissemination of research on social media and actual social impact. 12 In other words, does a study simply generate a large number of tweets and retweets, or does it also lead to legislation or changes in treatment protocols? Examining 10 research projects in different fields of knowledge, they found that only about 0.43% of tweets or Facebook posts provided linkages with information about actual social impact, with a range of 0% to 4.98% per project.
It seems likely that the interaction between alternative metrics and the medical literature will continue to evolve. The AAS itself has changed since 2011, with several sources of attention that once contributed to the overall total having been removed, usually owing to lack of full transparency. Specific social media venues can go in and out of fashion; in a few years’ time, a different option may replace Twitter or Facebook in popularity. The communication habits of sports medicine stakeholders are in a state of evolution as well. As more digital natives enter the professional ranks, and older digital immigrants become more media savvy, use of social media should continue to expand.
For the first time this year, Brett Owens and his colleagues on the AJSM Electronic Media Editorial Board have created a podcast (http://sageorthopaedics.sage-publications.libsynpro.com/ajsm-special-edition-podcast-top-10-altmetric-articles-of-2019) highlighting the 10 articles with the highest AAS that appeared in our journal in 2019. This is an eclectic group of studies, ranging from epidemiology and injury prevention to running biomechanics to systematic treatment reviews. Some of these articles probably have special appeal to the general public, while others would seem primarily to interest medical professionals. The highest AAS for 2019 belongs to a randomized trial of platelet-rich plasma for patellar tendinopathy. 14
The AAS is a neutral indicator of the volume of conversation about an article in visible public spaces. It indicates how much a study is being “talked about” but not what is being said. Critical comments count the same as supportive ones. Next time you’re reading an AJSM paper online and you see a colorful donut next to the title, click on the separate links to read what is being said and who is saying it. If you prefer to read the print journal, consider checking the online edition to sample the buzz on articles that pique your interest. You may even be tempted to join in the conversation.