Measuring scientific prestige of papers with time-aware mutual reinforcement ranking model

Abstract

Quantitative methods for determining the quality of scientific publications evolved gradually from popularity methods to prestige methods. However, existing methods have some drawbacks, such as inability to account for important factors and mutual reinforcement between different entities, and limitation of using novel information techniques like artificial intelligence (AI) methods. This study proposes an intelligent time-aware mutual reinforcement ranking (TAMRR) model that accounts for mutual reinforcement, and temporal factors, such as the time of citation, to measure the prestige of scientific papers. The method also considers the distribution of the co-authors’ contributions, which indicates the credit allocation of citations. Moreover, mutual reinforcement which indicates interactive impact between different entities by means of the extension of an AI algorithm, i.e., Hyperlink-Induced Topics Search (HITS) algorithm, is adopted to further explore the interactions of papers, journals and authors. Another AI algorithm, i.e., PageRank, is also enhanced to measure the prestige of papers, journals, and authors in citation networks, which are then used as the inputs to the modified HITS. Experiments on temporal factors and heterogeneous networks reveal that these factors are likely to be informative in prestige measurements. Analysis of correlations suggests that our proposed intelligent ranking method is reasonable. This study offers an intelligent method for researchers, authors, and entrepreneurs to quantify the importance of scientific papers and the conclusions are likely to be of importance for researchers in both the academic and enterprise domains.

Keywords

Scientific prestige of papers artificial intelligence citation networks time-aware PageRank HITS mutual reinforcement

1 Introduction

With the continuing explosion of information and development of artificial intelligence (AI) techniques, intelligent methods for efficient information retrieval are becoming increasingly important. Zhang et al. argued that acquiring accurate, timely, rapid, and comprehensive information from the large-scale and fast-growing literature is difficult but urgent in the big data era [21]. Journals, papers, and authors are three basic categories in bibliometric research, and prestige-measuring methods adopted in respective entities can be versatile to some extent. Accordingly, we cannot deny the mutual interaction between journals, papers, and authors in prestige-measuring methods. Consequently, in the present study we mainly focus on using AI approaches to paper ranking, which would account for different factors that may affect the objects’ prestige. Similar methods can be used to identify the prestige of other entities, such as journals and authors.

Although, some altmetric analysis in bibliometric is arisen these days [37], measuring the prestige and popularity of scientific publications based on citation network is still a mainstream. Two classes of such methods exist. One is the popularity method which is based on the number of citations and its extensions such as the h-index [19], the h² index [26], the g-index [22], the c-index [23]; this approach is still used by many researchers. In such circumstances, every citation can be considered to be a vote, and the more citations a paper receives, the more popular it is [33]. Another class is recognized as prestige method using a variety of algorithms to rank journals, papers, and authors, based on the measurement of prestige. An AI algorithm, PageRank [36] which was initially developed for internet search engine, have been widely used for ranking web pages, literature, journals, authors, and research trends. Another similar algorithm, Hyperlink-Induced Topics Search (HITS) [20], has also been used to measure scientific prestige. However, traditional methods based on homogeneous networks cannot account for relations between different scientific entities; to this end, methods that assume heterogeneous networks have been proposed recently, such as the P-Rank [14], and the Mutual-Rank [39]. By exploring the aforementioned two patterns, it is known that measuring the scientific prestige is such a gradually evolving process that there is no gold standard to eliminate all the problems [25].

It is also found that traditional PageRank assigns equivalent importance to every node, leading to a noticeable bias. Citations of an identical entity undoubtedly have complex temporal distribution, so time factor must be taken into consideration to reduce the bias [7, 9]. In addition, if a paper is cited by prestigious papers, it tends to become prestigious in practice. Similarly, if a paper is published by a high-prestige author in a prestigious venue, the paper is more likely to be recognized by the readers. Moreover, heterogeneous networks can affect the prestige of different entities and call for consideration of the interactions between them [10, 39].

In this study, we integrate the journals, papers, and authors to establish heterogeneous networks that are referred to journal-paper networks, author-paper networks, and journal-author networks, and the time-aware factor that accounts for the timeinterval between first cited and published time of the paper is further explored and used to distribute the prestige across different cited papers. Citation number of journals is used as a weight for prestige measurements. The distribution of the co-authors’ contributions are modeled as weights on the edges of the author citation network, reflecting the fact that an author could gain prestige according to their contribution. In addition, HITS is extended to account for the mutual reinforcement, which indicates interrelationship between different entities, after the integration with the outputs of various weighted PageRank algorithms. Some principal accomplishments are as follows. (1) The “time-aware” modifications for papers based on the homogeneous network, takes into account the time factors of cited time, allowing to further identify whether this factor affects the prestige ranking of papers. Citation number of journals is considered to measure their prestige indicating that each edge between cited and citing journals is weighted by the number of citations. Different contributions of co-authors are used to distribute the citation credit to authors in the author citation network. Various weighted PageRank algorithms are applied to measure the prestige of papers, journals, and authors in the homogeneous network. (2) The results of the weighted PageRank algorithm, obtained in (1), are used as the inputs to the mutual reinforcement iteration process, i.e., the modified HITS algorithm, until the search converges. In this way, papers can be ranked by taking into consideration both inter-network and intra-network effects. (3) Compared with the existing paper ranking methods, such as Citation Counts, standard PageRank, HITS, P-rank [14], and MutualRank [39], the feasibility and the efficiency of the proposed time-aware mutual reinforcement ranking (TAMRR) model are evaluated. In addition, analysis of correlations is performed to further validate our proposed model.

The remainder of this paper is organized as follows. Section 2 provides an overview of related work, where a variety of approaches to paper-ranking are reviewed in both the homogeneous network and heterogeneous network. In Section 3, our proposed model is described in detail, and its integration of two AI algorithms, i.e., weighted PageRank algorithm and modified HITS algorithm are elaborated. Section 4 describes data collection and experimental processes, while the experimental results are evaluated and discussed in Section 5. Conclusions and future research plans are listed in Section 6.

2 Related work

The ever-increasing amount of information generated by the modern society motivates to develop intelligent methods for more efficient extraction of relevant information. Different scientific disciplines are generating vast amounts of information. Thus, some AI techniques are needed for gauging the importance of a certain publication in a certain academic field.

2.1 Intelligent prestige methods in homogeneous network

Various methods have been proposed for determining the prestige of publications, not only considering the number of citations but also the prestige structure of citations. After Brin & Page [19] who firstly proposed the PageRank algorithm to rank the prestige of web pages, which has since been used for literature analysis by Chen et al. [31] and Ma et al. [29], many variations have been proposed to modify or extend the standard PageRank algorithm to address more complicated scenarios. Su et al. [6] proposed the PrestigeRank algorithm by taking into consideration the missing data; Yan [12] proposed a topic-based PageRank approach to identify the prestige of papers, journals, and authors at the topic level.

With the further exploration in prestige methods, some considerable time-aware factors have been increasingly recognized owing to the observation that recently published papers may receive somewhat less citations. Walker et al. [19] proposed the CiteRank algorithm, which is similar to the PageRank algorithm, by distributing random surfers exponentially according to age, to reduce the time bias toward recently published papers. Singh et al. [4] also proposed a similar modified version of the time-aware PageRank algorithm.

All these methods recognized that publications are independent from other entities. However, heterogeneous factors have been increasingly noticed by researchers.

2.2 Intelligent prestige methods in heterogeneous network

As is commonly recognized, papers are not isolated from journals and authors. The concept of mutual reinforcement has been proposed and used in extensive fields, such as authority ranking [42], academic performance prediction and library book recommendation [8], sustainability and innovation in development [35], indicating interactive impact of several interrelated entities. Some researchers have attempted to explore mutual reinforcement of papers, journals, and authors in bibliometrics. Yu et al. [34] noticed that citations may really make a difference for the paper’s prestige, and proposed the Timed PageRank algorithm, and they further investigated the prestige of papers published in different venues by different authors. FutureRank [18] used the authorship network and the publication time of an article to predict future citations, with the premise that the higher the prestige of an actor is, the more citations it will receive in the future. Yan & Ding [13] proposed weighted PageRank, which takes into account the citing journals and citations time interval. This method assumes that if a publication is published in a higher-prestige venue, this publication will be more prestigious. Qiao et al. [17] took into account the prestige of journals and the correlation between two papers to measure the prestige of publications owing to the fact that correlated actors interact more compared with interdependent ones. All these researches suggest that the prestige of authors and the publication venue significantly affects the paper’s prestige. To further explain this interaction between papers, journals, and authors, an AI algorithm, HITS, was adopted to identify the mutual reinforcement process, in which the authority value and hub value are computed to measure the quality of a target web page.

Zhou et al. [11] established a co-ranking framework in a heterogeneous network of authors and documents. Yan et al. [14] proposed a new informetric indicator, P-Rank, for measuring prestige in heterogeneous scholarly networks with different weights assigned to the citing papers, citing journals, and citing authors. Jiang et al. [39] proposed a new graph-based ranking algorithm, MutualRank, to mutually reinforce the interaction between papers, researchers, and venues, to achieve more synthetic, accurate, and less biased ranking results. Yu et al. [10] presented a multiple-link mutually reinforced journal-ranking (MLMRJR) method, which not only considered prestige on the intra-network level, but also the mutual reinforcement on the inter-network level, based on the PageRank and HITS algorithms. Ayala-Gómez et al. [15] also used a novel rank model based on knowledge graph to recommend citations for scientific publications. In this sense, publications, journals, and author are coined and reinforce the prestige of each other.

Although our proposed TAMRR model also used the rank model to measure the value of a publication, we have combined two AI algorithms, i.e., modified PageRank and extended HITS, to deal with the recommendation of publications in a more intelligent way. In addition, the proposed TAMRR model differs from the current prestige measurement. We consider the citation time interval to differentiate the weight according to the cited paper. Further, the contributions distribution of co-authors of a given paper is considered in this study, indicating that an author inherits prestige from citing authors according to their contributions. TAMRR model differentiates the edge weights of authors based on the distribution of their contributions. More details are provided in whatfollows.

3 Method

In this study, we apply AI and computer science methods to measure the prestige of scientific publication. The method accounts for the time factors of papers by assigning a time-aware weight to every cited node rather than assuming them to be equal; the journals’ edges are weighted by their citation counts; and the contribution distribution of authors in a paper is further considered while measuring the authors’ prestige. We assume that the following factors may affect the corresponding entity in some way.

In a paper citation network, if a paper is cited by any other papers immediately, it means that the paper has immediately attracted the attention of the readership and thus is more likely to be prestigious. Nevertheless, there is a specific phenomenon called the “sleeping beauty” [2, 3], in which a published paper is cited relatively infrequently until a “prince” awakes the “beauty”, following which the number of citations increases in explosive manner [1]. Here, for the sake of simplicity, the “sleeping beauty” phenomenon is not taken into account owing to its scarce manifestation.

The journal citation network differs from the paper citation network in which a journal can cite the same journal more than once, and the more times a paper is cited, the more prestige will be inherited.

In the author citation network, a paper may have more than one author and the distribution of credit across authors should be in line with their contributions to the paper; similar to the journal citation network, the citation count of authors is further considered.

In addition, to address the disadvantages of the homogeneous network paradigm, two AI algorithms, weighted PageRank algorithm and the modified HITS algorithm, were combined to mutually reinforce these processes inheriting the prestige from journals and authors. The basic principles in this paper are as follows. First, if a paper is cited by a more prestigious one, then this paper tends to be more prestigious, and so it is in the other two citation networks. That is, if cited by a higher-prestige journal, this journal will be facilitated with higher prestige; if cited by a more prestigious author in a certain field, the cited author is considered to be more prestigious compared with what is cited by less prestigious ones. In addition, since journals, papers, and authors are considered to be interdependent entities, the prestige of any one out of the three will affect the other two. In this study, we focus on paper ranking for representation; thus, we deem that if a paper is published in a more prestigious venue and written by authors with higher prestige, it tends to be more prestigious, similar to the assumption for authors and journals.

To further express these ideas, the TAMRR model is constructed and explained in the following sub-sections. We define the time-aware factors and contributions, and explain the approach by considering the intra-network and inter-network.

3.1 Intra-network

To intuitively express the relationships, we built three types of networks for different entities: 1) the paper citation network, 2) the journal citation network, and 3) the author citation network.

3.1.1 Paper citation network

Let PCN = (P, E^P) be a directed, paper citation network, and P ={ P₁, P₂,. . . , P_{N
_P} } represents a set of papers, E^P ⊆ P × P be a set of edges between papers and their citations, the exact expression of E^P is as follows:

$E (P_{r}, P_{s}) = {\begin{matrix} 1 & if paper P_{r} cites P_{s} \\ 0 & otherwise \end{matrix},$ (1) where P_r, P_s ∈ P, and P_r cites P_s. Every cited node weight can be defined as the time from publication to the first citation. The weights can be given by the following formula: $W_{P_{s}} = e^{α (t_{P_{sc}} - t_{P_{sp}})},$ (2) which indicates that if a paper is cited by others immediately after it is published, the paper is more likely to be prestigious. W_{P
_s} represents the weight of P_s. The parameter α captures the relationship between the time interval and the number of citations; this is discussed in detail in Section 4. In Eq. (2), t_{P
_sc} denotes the time at which P_s was first cited by others. The time t_{P
_sp} is the published time of P_s. Thus, for a given paper, its prestige is computed recursively, taking into account its corresponding node-weight, as follows [36]: ${PR}_{P_{s}} = \frac{1 - d}{N_{P}} + d \cdot \sum_{P_{r}, P_{s} \in P} \frac{W_{P_{s}} \cdot {PR}_{P_{r}}}{N_{P_{r}}},$ (3) where PR_{P
_s} and PR_{P
_r} denote the PageRank values of P_s and P_r, respectively, N_P is the total number of papers in our dataset, N_{P
_r} is the number of out-links of P_r, that is, the total number of references of P_r in our dataset, and d is the damping factor which was set to 0.85 in [36] and to 0.5 in [9]. As of now, there is still no consensus regarding the value of the damping factor in citation networks [7].

3.1.2 Journal citation network

Let JCN = (J, E^J, T^J) be a directed, edge-weighted journal citation network, and J ={ J₁, J₂,. . . , J_{N
_J} } represents a set of journals, E^J ⊆ J × J be a set of edges between journals and their citations, which can be expressed as follows:

$E (J_{u}, J_{v}) = {\begin{matrix} 1 & if journal J_{u} cites J_{v} \\ 0 & otherwise \end{matrix},$ (4) where J_u, J_v ∈ J and J_u cites J_v. Let T^J be an N_J × N_J matrix of citation count weight of journals. $T (J_{u}, J_{v}) = {\begin{matrix} n_{J_{uv}} if journal J_{u} cites J_{v} n_{J_{uv}} times \\ 0 otherwise \end{matrix} .$ (5)

Since a journal can be cited by the same journal more than once, the number of citations can be considered as the weight of an edge. Thus, edge-weighted PageRank for journal ranking can be written as follows [10]: ${PR}_{J_{v}} = \frac{1 - d}{N_{J}} + d \cdot \sum_{J_{u}, J_{v} \in J} \frac{n_{J_{uv}} \cdot {PR}_{J_{u}}}{N_{J_{u}}} .$ (6) In the above equation, PR_{J
_v} and PR_{J
_u} denote the PageRank values of J_v and J_u, N_J is the total number of journals in our dataset, N_{J
_u} indicates the number of out-links of J_u which represents the total number of citations of J_u, d is the damping factor which is similar to that for the paper citation network, and n_{J
_uv} represents the count weight of the edge E_uv, E_uv ∈ E^J, indicating the number of times J_v was cited by J_u.

3.1.3 Author citation network

Let ACN = (A, E^A, T^A) be a directed, edge-weighted network of author citations, let A ={ A₁, A₂,. . . , A_{N
_A} } be a set of authors, E^A ⊆ A × A be a set of edges between authors and their citations, which can be formulated as follows:

$E (A_{o}, A_{q}) = {\begin{matrix} 1 & if author A_{o} cites A_{q} \\ 0 & otherwise \end{matrix},$ (7) where A_o, A_q ∈ A and A_o cites A_q. Let T^A be an N_A × N_A matrix of contribution distribution weight of authors which can be noted as follows:

$\begin{matrix} T (A_{o}, A_{q}) = \\ {\begin{matrix} W_{A_{oq}} & if author A_{o} distributes W_{A_{oq}} credit to A_{q} \\ 0 & otherwise \end{matrix}, \end{matrix}$ (8) where W_{A
_oq} is the credit distribution from A_o to A_q. What is specific for the author citation network is that a paper may have several authors, with different authors playing different roles on the same paper, owing to their distinct contributions. Clearly, the first author contributes more to the paper compared with other co-authors, and the contributions of authors decrease with their ranks. For simplicity, we do not consider here scenarios with corresponding authors and with alphabetically listed authors. Overall, for a certain paper, the contribution distribution of the authors’ credits takes the following form [40]:

$d_{P_{j}} (i, k) = k^{- \frac{1}{l}} i^{- (1 - \frac{1}{l})},$ (9) where k represents the total number of co-authors of paper P_j, i is the ith co-author of P_j, i ∈ k, l is a constant and its value determines the specific distribution of the authors’ credits, the exact value of l will be discussed in Section 4. To sum up to 1 for the distribution, we normalize the weights as follows: $D_{P_{j}} (i, k) = \frac{d_{P_{j}} (i, k)}{\sum_{i = 1}^{k} d_{P_{j}} (i, k)} .$ (10)

For a given author, the contribution distribution weight is given as: $W_{A_{oq}} = \sum_{P_{j} \in P} D_{P_{j}} (i_{A_{q}}, k),$ (11) where D_{P
_j} (i_{A
_q}, k) is the distribution of credits of A_q for paper P_j, and i_{A
_q} means that A_q is the ith author of paper P_j. Thus, the weighted PageRank algorithm can be applied to obtain the following: ${PR}_{A_{q}} = \frac{1 - d}{N_{A}} + d \cdot \sum_{A_{o}, A_{q} \in A} \frac{W_{A_{oq}} \cdot {PR}_{A_{o}}}{N_{A_{o}}},$ (12) where PR_{A
_q} and PR_{A
_o} are the weighted PageRank values of A_q and A_o, N_A is the total number of authors in our used dataset, and N_{A
_o} is the number of out-links of A_o. W_{A
_oq} represents the weight between A_o and A_q concerning the distribution of contributions. The parameter d is the damping factor in the PageRank algorithm.

In conclusion, different weighted PageRank algorithms can be used for measuring the respective prestige of papers, journals, and authors in the homogenous network. The application of the modified HITS algorithm, which is aimed at inheriting the impact of three entities in the heterogeneous network, will be described in what follows.

3.2 Inter-network

Besides the aforementioned three directed intra-networks, three undirected networks were constructed to express their interrelationship, that is, the paper-journal network (PJ or JP), the paper-author network (PA or AP), and the author-journal network (AJ or JA). These three networks can be coupled to their corresponding intra-networks.

Let the paper-journal network (PJ or JP) incorporate the paper citation network and the journal citation network, let PJN^PJ = PJN^JP = (P ∪ J, E^PJ) be an undirected, unweighted paper-journal graph, let P ∪ J be a series of vertices and E^PJ ⊆ P × J denote the edges between papers and journals. For example, there are two edges (P_i, J_k) and (J_k, P_i) between them, (P_i, J_k) , (J_k, P_i) ∈ E^JP. Then, the adjacency matrix PJ (P_i, J_k) or JP (J_k, P_i) can be expressed as:

$\begin{matrix} PJ (P_{i}, J_{k}) = JP (J_{k}, P_{i}) \\ = {\begin{matrix} 1 & if paper P_{i} is published in journal J_{k} \\ 0 & otherwise \end{matrix} . \end{matrix}$ (13) Let the paper-author network (PA or AP) couple the paper citation network and the author citation network, and let PAN^PA = PAN^AP = (P ∪ A, E^PA) be an undirected, unweighted paper-author graph. Let P ∪ A be a set of vertices and E^PA ⊆ P × A be a set of edges between papers and their corresponding authors, where the two edges (P_i, A_j) , (A_j, P_i) ∈ E^PA connect paper P_i and author A_j. Then, the adjacency matrix PA (P_i, A_j) or AP (A_j, P_i) can be expressed as:

$\begin{matrix} PA (P_{i}, A_{j}) = AP (A_{j}, P_{i}) \\ = {\begin{matrix} 1 & if paper P_{i} is written by \\ author A_{j} \\ 0 & otherwise \end{matrix} . \end{matrix}$ (14)

Finally, let the journal-author network (AJ or JA) integrate the journal citation network and author citation network, JAN^JA = JAN^AJ = (J ∪ A, E^JA), let J ∪ A be a set of vertices and E^JA ⊆ J × A represent a series of edges connecting the journals and authors. For example, the two edges (J_k, A_j) , (A_j, J_k) ∈ E^JA connect A_j and J_k. Then, the adjacency matrix JA (J_k, A_j) or AJ (A_j, J_k) can be expressed as:

$\begin{matrix} JA (J_{k}, A_{j}) = AJ (A_{j}, J_{k}) \\ = {\begin{matrix} n_{jk} & if author A_{j} published n_{jk} \\ papers in J_{k} \\ 0 & otherwise \end{matrix} . \end{matrix}$ (15)

3.3 Time-aware mutual reinforcement ranking model

As we argued above, the prestige of a paper is determined not only by the paper citation network, but also by the other two inter-networks: the paper-journal network and the paper-author network. The same holds for journals and authors. Therefore, mutual reinforcement is a key point in measuring scientific prestige which combines the interactive impact of papers, journals, and authors. According to our aforementioned works, we have constructed three adjacency matrices P, J, and A from the intra-network indicating the citation relations of papers, journals and authors, P is an N_P × N_P matrices, J is an N_J × N_J matrices, A is an N_A × N_A matrices and six adjacency matrices PJ, JP, PA, AP, AJ, and JA from the inter-network. $\bar{P}$ , $\bar{J}$ , $\bar{A}$ , $\bar{PJ}$ , $\bar{JP}$ , $\bar{PA}$ , $\bar{AP}$ , $\bar{AJ}$ , and $\bar{JA}$ are their corresponding transitionmatrices.

Our proposed model is based on the model of [41]. There are three significant measurements in our proposed model: 1) PR_P, the prestige of papers; 2) PR_J, the prestige of journals; and 3) PR_A, the prestige of authors. Taking the initial vectors PR_P, PR_J, and PR_A as input, these vectors are constantly and iteratively updated until convergence, as follows:

For each PR_P at iteration t, denoted as ${PR}_{P}^{(t)}$ , the values of PR_P, PR_J, PR_A and can be reinforced at iteration t + 1. $δ {PR}_{P}^{(t)}$ represents the distribution for reinforcement of ${PR}_{P}^{(t + 1)}$ , which indicates the value of PR_P at iteration t + 1, and the remaining $(1 - δ) {PR}_{P}^{(t)}$ is divided into two parts: $ϑ (1 - δ) {PR}_{P}^{(t)}$ to reinforce ${PR}_{J}^{(t + 1)}$ which indicates the value of PR_J at iteration t + 1, and the alternative part $(1 - ϑ) (1 - δ) {PR}_{P}^{(t)}$ to reinforce the value of PR_A at iteration t + 1, denoted as ${PR}_{A}^{(t + 1)}$ .

Similarly, each PR_J at iterationt, denoted as ${PR}_{J}^{(t)}$ , reinforces the value of PR_J at iteration t + 1 and the values of PR_P and PR_A at iteration t + 1. A portion $δ {PR}_{J}^{(t)}$ is used to reinforce ${PR}_{J}^{(t + 1)}$ , and the remaining part $(1 - δ) {PR}_{J}^{(t)}$ is divided into two parts: $ϑ (1 - δ) {PR}_{J}^{(t)}$ is used to reinforce ${PR}_{A}^{(t + 1)}$ and another portion $(1 - ϑ) (1 - δ) {PR}_{J}^{(t)}$ reinforces ${PR}_{P}^{(t + 1)}$ .

The value PR_A at iteration t is distributed among three parts in the same way and reinforces the values of PR_P, PR_J, and PR_A at iteration t + 1, respectively. For the first part, $δ {PR}_{A}^{(t)}$ is used for reinforcing the value of PR_A at iteration t + 1, denoted as ${PR}_{A}^{(t + 1)}$ . The second part $ϑ (1 - δ) {PR}_{A}^{(t)}$ is used to reinforce ${PR}_{P}^{(t + 1)}$ , and the remaining third part $(1 - ϑ) (1 - δ) {PR}_{A}^{(t)}$ is used to reinforce the value of ${PR}_{J}^{(t + 1)}$ .

In summary, both δ and ϑ are parameters representing a portion. The parameter δ indicates the probability of random walkers to jump at the intra-network and the parameter ϑ indicates the probability of walks between inter-networks. Then, the TAMRR model is formulated as follows [10, 39]:

$\begin{matrix} {PR}_{P}^{(t + 1)} = δ {\bar{P}}^{T} {PR}_{P}^{(t)} & + & (1 - ϑ) (1 - δ) {\bar{JP}}^{T} {PR}_{J}^{(t)} \\ + & ϑ (1 - δ) {\bar{AP}}^{T} {PR}_{A}^{(t)}, \end{matrix}$ (16)

$\begin{matrix} {PR}_{J}^{(t + 1)} & = ϑ (1 - δ) {\bar{PJ}}^{T} {PR}_{P}^{(t)} + δ {\bar{J}}^{T} {PR}_{J}^{(t)} \\ + (1 - ϑ) (1 - δ) {\bar{AJ}}^{T} {PR}_{A}^{(t)}, \end{matrix}$ (17)

$\begin{matrix} {PR}_{A}^{(t + 1)} & = (1 - ϑ) (1 - δ) {\bar{PA}}^{T} {PR}_{P}^{(t)} \\ + ϑ (1 - δ) {\bar{JA}}^{T} {PR}_{J}^{(t)} + δ {\bar{A}}^{T} {PR}_{A}^{(t)} . \end{matrix}$ (18) Eqs. (16), (17), and (18) can be organized as follows: ${PR}^{(t + 1)} {= MRP}^{(t)} .$ (19) Where $PR = {[{PR}_{P}^{T}, {PR}_{J}^{T}, {PR}_{A}^{T}]}^{T}$ , $M = [\begin{matrix} δ {\bar{P}}^{T} & (1 - ϑ) (1 - δ) {\bar{JP}}^{T} & ϑ (1 - δ) {\bar{AP}}^{T} \\ ϑ (1 - δ) {\bar{PJ}}^{T} & δ {\bar{J}}^{T} & (1 - ϑ) (1 - δ) {\bar{AJ}}^{T} \\ (1 - ϑ) (1 - δ) {\bar{PA}}^{T} & ϑ (1 - δ) {\bar{JA}}^{T} & δ {\bar{A}}^{T} \end{matrix}] .$ (20) The iterative process updates the vector $PR = {[{PR}_{P}^{T}, {PR}_{J}^{T}, {PR}_{A}^{T}]}^{T}$ until convergence, that is,iterations end when ∥PR ∥ ⩽ ɛ [27], and ɛ is set to 0.0001 in this study. Mutual reinforcement in a heterogeneous network is shown in Fig. 1, indicating that entity j cites entity i.

Fig. 1

Mutual reinforcement in a heterogeneous network.

4 Data selection and setup of parameters

To validate the proposed model, practical experiments should be performed on with real data, through which it can be verified that the model is not only theoretically feasible, but also practical.

4.1 Data used in this study

For the aforementioned purpose, we extracted publications from the Web of Science database, which has been used widely as a source of data for studies of this type. We extracted and analyzed publication data related to the fields of performance management and performance measurement. Owing to the fact that many studies have been performed in the performance management domain, an efficient method for identifying which publications are prestigious seems to be essential. Unlike other researchers who extracted papers from predefined sets of journals, we decided here to use some keywords to constrain the publication data in our field of interest, similar methods are widely used in bibliometric [16, 38]. Performance management seems to be a multidisciplinary field, with wide outreach to public sectors during the past several years [5]. Bititci et al. [38] used “performance measurement”, “performance management”, “performance indicators”, “management control” and “strategic control” as search strings to identify relevant papers, but the obtained dataset contained publications from a broad range of fields, owing to the multidisciplinary nature of the topic of interest. In the recent work by Cuccurullo et al. [5], the evolution of performance management was tracked based on the keywords “Performance management” and “Performance evaluations” in “Social Science Citation Index, SSCI”, yielding only 1534 publications. In this study, to obtain a more correlative and efficient database, we used “Performance management”, “Performance measurement”, and “Management control” as all subject terms because of the explorative nature of our study, and combined with no time limitation to further explore the comparative prestige of publications and how different factors and different networks affect the prestige of different actors. Moreover, performance management mainly belongs to the humanities and social sciences, so “Social Science Citation Index, SSCI” under “More settings” was checked. The final dataset included 3275 papers published in 658 journals, after canceling all of the isolated nodes by Sci2, with a total of 6330 authors. All of the citation network and bipartite network are extracted by Sci2 with its default set. Some erroneous entries, such as entries corresponding to papers for which the time of the first citation was before the time of publication were dropped while cleaning the data.

4.2 Values of parameters

The parameters in this study are α, δ, ϑ, l and d. The parameter α reflects the time factor that is determined by the relationship between the time interval and their citation numbers. According to Fig. 2, α was set to –0.489, based on [9], because the relationship between citation number of papers and their time intervals between published and first cited fits the trend y = e^-0.498x. Specifically, for the time interval of 1, i.e., when a paper was cited immediately after it was published for one year, the citation number of this paper tends to be enormous; but for the time interval of 0, the citation number is far less than one year. The potential reason is that most of the papers need time to be cited by others whether it is prestigious or not.

Fig. 2

Relationship between time interval and citation number.

In addition, d is the damping factor in PageRank and is usually set to 0.85. Many researchers discussed how different values of d affect the results of PageRank [24, 31, 41], so we will not discuss this issue in detail and will set d to 0.85 in what follows. The parameter δ indicates the probability of random walkers to jump from the intra-network to the inter-network, and the performances of different values of δ are listed in Table 1. Different values of δ may affect the final results of our proposed model but affect slightly. As can be seen, the performance of our proposed model is optimal when δ is set to 0.4, for most evaluation metrics. Citation Count is the baseline that we use when evaluating the effects of parameters, and this metric has been extensively used for ranking papers and some scholars have used it as a baseline metric, such as Nykl et al. [28] Although it seems that 0.4 is the optimal value for our model, the results vary for a range of different rankings. The parameter ϑ represents the probability of random walkers to jump between different inter-networks and it is set to 0.5, which means that their probability stepped in by random walkers is the same.

Table 1

Performance metrics according to ROC with baseline of Citation Count

	AUC	Specificity	Sensitivity	Gmean
0.2TAMRR	0.9560	0.5101	0.9766	0.7058
0.3TAMRR	0.9604	0.5503	0.9786	0.7339
0.4TAMRR	0.9597	0.5772	0.9798	0.7520
0.5TAMRR	0.9572	0.5772	0.9798	0.7520
0.6TAMRR	0.9567	0.5705	0.9795	0.7475
0.7TAMRR	0.9566	0.5570	0.9789	0.7384
0.8TAMRR	0.9554	0.5436	0.9782	0.7292

For parameter l, it defines different distributions of credits among co-authors. As shown in Fig. 3, the contribution of each co-author is evenly distributed while l = 1, $d (i, k) = \frac{1}{k}$ ; if l =∞, $d (i, k) = \frac{1}{i}$ , which can be expressed as harmonic counting [30] after normalization. For l = 2 or 3, we obtain different distributions of co-authors. The larger the value of l, the steadier the distribution declines; and if the value of l continuously increases, it tends to diverge. In our proposed model, l was set to 3 to distribute the credits of co-authors according to their contributions, as was done by [39].

Fig. 3

Distribution of contributions of co-authors.

5 Experimental results and evaluation

The performed experiments suggest that the method allows to intuitively explore ranking of large publication volumes, and it also allows to analyze the contribution of different factors to prestige. In this section, a detailed analysis from various perspectives is presented, which help to explain and support the proposed model.

5.1 Popularity and prestige

It is widely acknowledged that popularity is different from prestige; a popular paper is not necessarily prestigious, and vice versa. Citation count-based indices are commonly used for measuring popularity. Novel methods for prestige measurement are being constantly proposed and evaluated, with state-of-the-art ranking methods becoming increasingly reliable.

Our proposed model considers both the quantity and quality of citations, and adopts the heterogeneous network paradigm in which prestige is deduced from different networks. Publication’s popularity is usually assessed in terms of its number of citations, while our proposed model aims to assess prestige. Compared with purely citation count-based ranking, there are four classes of entities in our approach, which include: 1) high prestige and high popularity entities, 2) high prestige and low popularity entities, 3) low prestige and high popularity entities, and 4) low prestige and low popularity entities. We assume that top 20 papers returned by the Citation Count method are the most popular, while top 20 papers returned by the TAMRR method are highly prestigious. High popularity and high prestige are two relative concepts which vary across different situations. In Table 2, the grey-shaded papers have high prestige and low popularity, which corresponds to being in the top 20 in the set returned by the TAMRR method but being out of the top 20 according to the Citation Count method. These papers are cited by prestigious papers that are written by prestigious authors or published in prestigious journals, so they are more prestigious owing to their citations. Similarly, the grey-shaded papers in Table 3 have low prestige and high popularity, being in the top 20 according to the Citation Count method but out of the top 20 according to our proposed model, because their citations tend to be relatively less prestigious, or because these papers are written by less prestigious authors and published in less prestigious venues. The remaining papers in both two tables have high prestige and high popularity, being in the top 20 of papers returned by both two methods; these papers are characterized by high quality and high number of citations. For these scientific publications, the two methods seem to yield similar results. Low prestige and low popularity papers are not presented in these tables. However, in practice the boundaries between the different categories are not likely to be so clear as to enable comparative evaluation of prestige and popularity according to the Citation Count and TAMRR methods. Prestigemethods are continually raising more concern compared to traditional popularity method currently.

Table 2
Top 20 papers in TAMRR model and their corresponding ranks in Citation Count method

Title TAMRR Citation

Score Rank Count Rank

Lambert, 2001, J ACCOUNT ECON, V32, P3 0.001969695 1 39 45

Banker, 2000, ACCOUNT REV, V75, P65 0.00188619 2 43 40

Indjejikian, 1999, J ACCOUNT ECON, V27, P177 0.001662799 3 8 297

Ferson, 2002, J FINANCE ECON, V65, P249 0.001505293 4 16 156

Chenhall, 2003, ACCOUNT ORGAN SOC, V28, P127 0.001461785 5 228 1

Melkers, 1998, PUBLIC ADM REV, V58, P66 0.001412786 6 22 103

Ittner, 2003, ACCOUNT REV, V78, P725 0.001257553 7 77 14

Ferson, 1999, J FINANCE, V54, P1325 0.001206245 8 4 591

Ittner, 2001, J ACCOUNT ECON, V32, P349 0.001194905 9 67 22

Poister, 1999, PUBLIC ADM REV, V59, P325 0.001140942 10 70 19

Scott, 1999, ACCOUNT ORGAN SOC, V24, P263 0.001050527 11 31 62

Meer-Kooistra, 2000, ACCOUNT ORGAN SOC, V25, P51 0.000955801 12 50 33

Ittner, 2003, ACCOUNT ORGAN SOC, V28, P715 0.000928469 13 105 3

Neely, 1999, INT J OPER PROD MANAGE, V19, P205 0.000851006 14 94 8

Abernethy, 1999, ACCOUNT ORGAN SOC, V24, P189 0.000791608 15 69 20

Cohen, 2005, J FINANCE, V60, P1057 0.000706813 16 6 373

Behn, 2003, PUBLIC ADM REV, V63, P586 0.000697421 17 98 5

Bourne, 2000, INT J OPER PROD MANAGE, V20, P754 0.000678503 18 129 2

Hartmann, 2000, ACCOUNT ORGAN SOC, V25, P451 0.000670541 19 43 39

Dekker, 2004, ACCOUNT ORGAN SOC, V29,P27 0.000669616 20 73 16

Title	TAMRR	Citation
Lambert, 2001, J ACCOUNT ECON, V32, P3	0.001969695	1	39	45
Banker, 2000, ACCOUNT REV, V75, P65	0.00188619	2	43	40
Indjejikian, 1999, J ACCOUNT ECON, V27, P177	0.001662799	3	8	297
Ferson, 2002, J FINANCE ECON, V65, P249	0.001505293	4	16	156
Chenhall, 2003, ACCOUNT ORGAN SOC, V28, P127	0.001461785	5	228	1
Melkers, 1998, PUBLIC ADM REV, V58, P66	0.001412786	6	22	103
Ittner, 2003, ACCOUNT REV, V78, P725	0.001257553	7	77	14
Ferson, 1999, J FINANCE, V54, P1325	0.001206245	8	4	591
Ittner, 2001, J ACCOUNT ECON, V32, P349	0.001194905	9	67	22
Poister, 1999, PUBLIC ADM REV, V59, P325	0.001140942	10	70	19
Scott, 1999, ACCOUNT ORGAN SOC, V24, P263	0.001050527	11	31	62
Meer-Kooistra, 2000, ACCOUNT ORGAN SOC, V25, P51	0.000955801	12	50	33
Ittner, 2003, ACCOUNT ORGAN SOC, V28, P715	0.000928469	13	105	3
Neely, 1999, INT J OPER PROD MANAGE, V19, P205	0.000851006	14	94	8
Abernethy, 1999, ACCOUNT ORGAN SOC, V24, P189	0.000791608	15	69	20
Cohen, 2005, J FINANCE, V60, P1057	0.000706813	16	6	373
Behn, 2003, PUBLIC ADM REV, V63, P586	0.000697421	17	98	5
Bourne, 2000, INT J OPER PROD MANAGE, V20, P754	0.000678503	18	129	2
Hartmann, 2000, ACCOUNT ORGAN SOC, V25, P451	0.000670541	19	43	39
Dekker, 2004, ACCOUNT ORGAN SOC, V29,P27	0.000669616	20	73	16

Table 3

Top 20 papers in Citation Count method and their corresponding ranks in TAMRR model

Title	Citation		TAMRR
Title	Count	Rank	Score	Rank
Chenhall, 2003, ACCOUNT ORGAN SOC, V28, P127	228	1	0.001461785	5
Bourne, 2000, INT J OPER PROD MANAGE, V20, P754	129	2	0.000678503	18
Ittner, 2003, ACCOUNT ORGAN SOC, V28, P715	105	3	0.000928469	13
Malmi, 2008, MANAGE ACCOUNT RES, V19, P287	98	4	0.000219981	124
Behn, 2003, PUBLIC ADM REV, V63, P586	98	5	0.000697421	17
Gunasekaran, 2004, INT J OPER PROD MANAGE, V87, P333	96	6	0.00020498	151
Henri, 2006, ACCOUNT ORGAN SOC, 31, P529	96	7	0.000361391	59
Neely, 1999, INT J OPER PROD MANAGE, V19, P205	94	8	0.000851006	14
Bisbe, 2004, ACCOUNT ORGAN SOC, V29, P709	93	9	0.000399881	53
Gunasekaran, 2001, INT J OPER PROD MANAGE, V21, P71	93	10	0.00040189	52
Chenhall, 2005, ACCOUNT ORGAN SOC, V30, P395	86	11	0.000429645	49
Neely, 2000INT J OPER PROD MANAGE, V20, P1119	83	12	0.000374696	57
Beamon, 1999, INT J OPER PROD MANAGE, V19, P275	81	13	0.000227425	120
Ittner, 2003, ACCOUNT REV, V78, P725	77	14	0.001257553	7
Neely, 2005, INT J OPER PROD MANAGE, V25, P1264	75	15	0.000157055	223
Dekker, 2004, ACCOUNT ORGAN SOC, V29, P27	73	16	0.000669616	20
Widener, 2007, ACCOUNT ORGAN SOC, V32, P757	73	17	0.000333231	68
Ferreira, 2009, MANAGE ACCOUNT RES, V20, P263	72	18	0.000172032	198
Poister, 1999, PUBLIC ADM REV, V59, P325	70	19	0.001140942	10
Abernethy, 1999, ACCOUNT ORGAN SOC, V24, P189	69	20	0.000791608	15

5.2 Evaluation of prestige based on both inter-network and intra-network

If a certain publication satisfies the following three conditions: 1) it is cited by prestigious publications, 2) it is written by prestigious authors, and 3) it is published in prestigious venues, then the publication under consideration itself will tend to be a prestigious one. Yet how do the time-aware factors, citation count, and distribution of contributions affect scientific prestige via inter-network and intra-network considerations? In the following, we elaborate on this issue.

5.2.1 Weighted PageRank differentiates prestige

In the standard PageRank algorithm, edges between any two publications are considered to be the same. However, in general, a more prestigious paper is more likely to be cited. Table 4 lists the top 20 papers returned by the standard PageRank algorithm (PR Rank). Moreover, the ranks of these 20 papers in the weighted PageRank (WPR Rank) algorithm, considering the time interval from the time of publication to the time of first citation, are used for comparison. Most of the analyzed publications have the same status in both algorithms. Specifically, the ranks of five grey-shaded publications for the weighted PageRank algorithm are much lower than those for the standard PageRank algorithm. This likely occurs because publications are not cited immediately following their publication. For example, the paper Abernethy, 1999, ACCOUNT ORGAN SOC, V24, P189 was published in 1999 but was cited in 2003, with the citation interval of four years.

Table 4
Ranking of publications using PageRank and weighted PageRank with the time factor

Title PR Rank Time Rank

Neely, 1999, INT J OPER PROD MANAGE, V19, P205 1 2

Melkers, 1998, PUBLIC ADM REV, V58, P66 2 5

Chenhall, 2003, ACCOUNT ORGAN SOC, V28, P127 3 1

Banker, 2000, ACCOUNT REV, V75, P65 4 6

Poister, 1999, PUBLIC ADM REV, V59, P325 5 8

Bourne, 2000, INT J OPER PROD MANAGE, V20, P754 6 7

Gunasekaran, 2001, INT J OPER PROD MANAGE, V21, P71 7 4

Lambert, 2001, J ACCOUNT ECON, V32, P3 8 3

Indjejikian, 1999, J ACCOUNT ECON, V27, P177 9 19

Behn, 2003, PUBLIC ADM REV, V63, P586 10 13

Perrin, 1998, AM J EVAL, V19, P367 11 26

Meer-Kooistra, 2000, ACCOUNT ORGAN SOC, V25, P51 12 16

Ittner, 2003, ACCOUNT REV, V78, P725 13 9

Abernethy, 1999, ACCOUNT ORGAN SOC, V24, P189 14 90

Beamon, 1999, INT J OPER PROD MANAGE, V19, P275 15 41

Hartmann, 2000, ACCOUNT ORGAN SOC, V25, P451 16 18

Ittner, 2003, ACCOUNT ORGAN SOC, V28, P715 17 12

Eddy, 1998, HEALTH AFF, V17, P7 18 10

Ittner, 2001, J ACCOUNT ECON, V32, P349 19 44

Davila, 2000, ACCOUNT ORGAN SOC, V25, P383 20 29

Title	PR Rank	Time Rank
Neely, 1999, INT J OPER PROD MANAGE, V19, P205	1	2
Melkers, 1998, PUBLIC ADM REV, V58, P66	2	5
Chenhall, 2003, ACCOUNT ORGAN SOC, V28, P127	3	1
Banker, 2000, ACCOUNT REV, V75, P65	4	6
Poister, 1999, PUBLIC ADM REV, V59, P325	5	8
Bourne, 2000, INT J OPER PROD MANAGE, V20, P754	6	7
Gunasekaran, 2001, INT J OPER PROD MANAGE, V21, P71	7	4
Lambert, 2001, J ACCOUNT ECON, V32, P3	8	3
Indjejikian, 1999, J ACCOUNT ECON, V27, P177	9	19
Behn, 2003, PUBLIC ADM REV, V63, P586	10	13
Perrin, 1998, AM J EVAL, V19, P367	11	26
Meer-Kooistra, 2000, ACCOUNT ORGAN SOC, V25, P51	12	16
Ittner, 2003, ACCOUNT REV, V78, P725	13	9
Abernethy, 1999, ACCOUNT ORGAN SOC, V24, P189	14	90
Beamon, 1999, INT J OPER PROD MANAGE, V19, P275	15	41
Hartmann, 2000, ACCOUNT ORGAN SOC, V25, P451	16	18
Ittner, 2003, ACCOUNT ORGAN SOC, V28, P715	17	12
Eddy, 1998, HEALTH AFF, V17, P7	18	10
Ittner, 2001, J ACCOUNT ECON, V32, P349	19	44
Davila, 2000, ACCOUNT ORGAN SOC, V25, P383	20	29

Table 5 lists the top 20 authors proposed by the PageRank algorithm, for different weighting conditions. The PR Rank measure is the rank based on the standard PageRank algorithm. Similarly, Count Rank is ranking by the PageRank algorithm weighted by citation counts, while Contribution Rank is ranking that is weighted by the distribution of contributions. First, compared with the standard PageRank algorithm, the ranks of Larcker D F, Banker R D, Bourne M, and Platts K are higher in Count Rank. Taking Bourne M as an example, the total number of citations is 924 in our dataset, and it far exceeds the numbers of citations for Behn R D and Sturman M C, whose ranks are lower in Count Rank but higher in PR Rank. Then, comparing the results of Count Rank and Contribution Rank for both of them, consider the credit distribution of contributions. Therein, the ranks of Eddy D M, Perrin B, Sturman M C and Behn R D increase to some extent, according to their high contributions to the papers. For example, Behn R D was cited 198 times and in all these cases as an independent author. Except these cases, the ranks of authors listed in Table 5 decline inordinately. For example, Srinivasan D is ranked 14 by the Count Rank method but ranked 119 by the Contribution Rank method. This author is the third co-author on all his cited papers, with overall three co-authors, which means that his contributions are smaller compared with the other two co-authors.

Table 5

Author ranking according to the PageRank algorithm with different weights

Name	PR Rank	Count Rank	Contribution Rank
Neely A	1	1	2
Chenhall R H	2	2	1
Willoughby K	3	3	30
Melkers J	4	4	14
Ittner C D	5	5	9
Eddy D M	6	9	3
Poister T H	7	7	15
Larcker D F	8	6	22
Streib G	9	12	38
Perrin B	10	17	7
Gunasekaran A	11	13	17
Patel C	12	14	36
Banker R D	13	11	50
Srinivasan D	14	15	119
Potter G	15	16	90
Sturman M C	16	20	6
Behn R D	17	23	4
Bourne M	18	8	37
Tirtiroglu E	19	22	91
Platts K	20	10	78

The PageRank algorithm is widely used to measure the prestige of journals. However, a certain journal can be cited more than one time by the same journal, which is substantially different from the publication citation network. Thus, the number of citations should be considered when measuring the prestige of a journal. The more times a journal is cited by other journals, the more prestige it inherits from those other journals. Weighted PageRank is a modification of PageRank algorithm weighted by the number of citations, and its output is listed in Table 6. Because the present study is exploratory, we considered only top 20 journals. In general, the results of PR Rank and Weighted Rank were similar, and for some entities the ranks differed slightly. For example, the rank dropped to 41 for the journal Organization Science owing to its relatively small number of citations by prestigious journals, while Journal of Public Administration Research and Theory is ranked ninth considering its number of citations. Owing to the limitations presented by the field of study that we selected to analyze, many citations and references were out of our dataset; thus, our results capture prestige measurements for one given field, without considering potential impacts of other fields.

Table 6

Ranking of journals using PageRank and PageRank weighted by the number of citations

Journal Title	PR Rank	Weighted Rank
Accounting Organizations and Society	1	1
Health Affairs	2	4
International Journal of Operations and Production Management	3	2
Journal of Portfolio Management	4	11
Accounting Review	5	5
Journal of Financial Economics	6	10
Journal of Accounting and Economics	7	7
Public Administration Review	8	3
Medical Care Research and Review	9	15
Management Accounting Research	10	6
Review of Financial Studies	11	12
Journal of Finance	12	14
Contemporary Accounting Research	13	8
Journal of Financial and Quantitative Analysis	14	26
Journal of the Operational Research Society	15	13
Strategic Management Journal	16	23
Journal of the British Sociological Association	17	36
Journal of Accounting Research	18	25
Organization Science	19	41
Journal of Public Administration Research and Theory	20	9

5.2.2 Ranking results considering the inter-network and intra-network

To increase the effectivity and feasibility of our ranking methods, we considered the effect of heterogeneous network, which reflects the notion that more prestigious publications tend to be written by more prestigious authors and published in more prestigious venues. In Table 7, results are shown for a method in which journals and authors were accounted for while ranking publications. Therein, PR Rank shows the ranking results for the standard PageRank algorithm. Time Rank shows the corresponding ranking results for the time weighted PageRank algorithm. A comparison between the results yielded by PageRank and Time Rank has been elaborated on before. To account for the prestige inheritance from journals and authors, we propose a TAMRR model with mutual reinforcement. The results of ranking are listed in Table 7. Most of the results are similar to those obtained using PR Rank and Time Rank, indicating that the approach is feasible to some extent. Specially, Gunasekaran, 2001, INT J OPER PROD MANAGE, V21, P71 and Beamon, 1999, INT J OPER PROD MANAGE, V19, P275 drop significantly in TAMRR ranking compared with their ranks according to both the PR Rank and Time Rank algorithms. Further analysis reveals that the authors of Gunasekaran, 2001, INT J OPER PROD MANAGE, V21, P71 are ranked 17, 36, and 91, respectively, and the authors of Beamon, 1999, INT J OPER PROD MANAGE, V19, P275 are ranked 53 and 109. Although the published venue International Journal of Operations and Production Management is prestigious, the rankings of these two papers show relatively significant drops. The paper Abernethy, 1999, ACCOUNT ORGAN SOC, V24, P189 is ranked 15 according to the TAMRR ranking method, which is similar to its ranking according to the PR Rank method; however, the paper is ranked 90 by Time Rank. Its publication venue is at the top in our dataset, as shown in Table 6, and its first author is ranked ninth in our dataset. The prestigious publication venue and authors make this publication a prestigious one by TAMRR method. Regarding Ittner, 2001, J ACCOUNT ECON, V32, P349, its sole author is ranked third among all the 6330 authors in our database. Thus, this publication is ranked seventh according to the TAMRR ranking method. In general, the TAMRR model quantifies prestige according to the inherited prestige of journals and authors, indicating that papers written by more prestigious authors and published in more prestigious venues tend to be more prestigious, and vice versa.

Table 7
Ranking results by different methods with inter-network and intra-network

Title PR Rank Time Rank TAMRR Rank

Neely, 1999,INT J OPER PROD MANAGE, V19, P205 1 2 14

Melkers, 1998, PUBLIC ADM REV, V58, P66 2 5 6

Chenhall, 2003, ACCOUNT ORGAN SOC, V28, P127 3 1 5

Banker, 2000, ACCOUNT REV, V75, P65 4 6 2

Poister, 1999, PUBLIC ADM REV, V59, P325 5 8 10

Bourne, 2000, INT J OPER PROD MANAGE, V20, P754 6 7 18

Gunasekaran, 2001, INT J OPER PROD MANAGE, V21, P71 7 4 52

Lambert, 2001, J ACCOUNT ECON, V32, P3 8 3 1

Indjejikian, 1999, J ACCOUNT ECON, V27, P177 9 19 3

Behn, 2003, PUBLIC ADM REV, V63, P586 10 13 17

Perrin, 1998, AM J EVAL, V19, P367 11 26 28

Meer-Kooistra, 2000, ACCOUNT ORGAN SOC, V25, P51 12 16 12

Ittner, 2003, ACCOUNT REV, V78, P725 13 9 7

Abernethy, 1999, ACCOUNT ORGAN SOC, V24, P189 14 90 15

Beamon, 1999, INT J OPER PROD MANAGE, V19, P275 15 41 120

Hartmann, 2000, ACCOUNT ORGAN SOC, V25, P451 16 18 19

Ittner, 2003, ACCOUNT ORGAN SOC, V28, P715 17 12 13

Eddy, 1998, HEALTH AFF, V17, P7 18 10 37

Ittner, 2001, J ACCOUNT ECON, V32, P349 19 44 9

Davila, 2000, ACCOUNT ORGAN SOC, V25, P383 20 29 22

Title	PR Rank	Time Rank	TAMRR Rank
Neely, 1999,INT J OPER PROD MANAGE, V19, P205	1	2	14
Melkers, 1998, PUBLIC ADM REV, V58, P66	2	5	6
Chenhall, 2003, ACCOUNT ORGAN SOC, V28, P127	3	1	5
Banker, 2000, ACCOUNT REV, V75, P65	4	6	2
Poister, 1999, PUBLIC ADM REV, V59, P325	5	8	10
Bourne, 2000, INT J OPER PROD MANAGE, V20, P754	6	7	18
Gunasekaran, 2001, INT J OPER PROD MANAGE, V21, P71	7	4	52
Lambert, 2001, J ACCOUNT ECON, V32, P3	8	3	1
Indjejikian, 1999, J ACCOUNT ECON, V27, P177	9	19	3
Behn, 2003, PUBLIC ADM REV, V63, P586	10	13	17
Perrin, 1998, AM J EVAL, V19, P367	11	26	28
Meer-Kooistra, 2000, ACCOUNT ORGAN SOC, V25, P51	12	16	12
Ittner, 2003, ACCOUNT REV, V78, P725	13	9	7
Abernethy, 1999, ACCOUNT ORGAN SOC, V24, P189	14	90	15
Beamon, 1999, INT J OPER PROD MANAGE, V19, P275	15	41	120
Hartmann, 2000, ACCOUNT ORGAN SOC, V25, P451	16	18	19
Ittner, 2003, ACCOUNT ORGAN SOC, V28, P715	17	12	13
Eddy, 1998, HEALTH AFF, V17, P7	18	10	37
Ittner, 2001, J ACCOUNT ECON, V32, P349	19	44	9
Davila, 2000, ACCOUNT ORGAN SOC, V25, P383	20	29	22

5.3 Correlation analysis

Correlation analysis is among widely used methods for evaluating the relationships between different features or methods. Five widely adopted methods were used to rank the publications in our dataset, and Table 8 shows the results of the Spearman correlation analysis applied to these rankings to explore the relationships between the existing models and our proposed method. Owing to the limitations associated with the selected field of study, not all citations and references were present in our dataset. Thus, our analysis and its results are limited to a certain domain. As shown in Table 8, PageRank-like algorithms are highly correlated with Citation Counts. Bothhomogeneous methods and heterogeneous method internal show a comparative higher correlation such as HITS algorithm and PageRank algorithm, and MutualRank model and TAMRR model. Based on the analysis of correlations, we conclude that the proposed model is feasible and efficient.

Table 8
Spearman’s coefficients of the relative ranks in various comparative methods

Citation count HITS PageRank P-Rank Mutual Rank TAMRR

Citation count 1 0.9110 0.9569 0.9466 0.6343 0.5764

HITS 0.9110 1 0.7992 0.7911 0.5439 0.5162

PageRank 0.9569 0.7992 1 0.9874 0.6334 0.5634

P-Rank 0.9466 0.7911 0.9874 1 0.6399 0.5779

Mutual Rank 0.6343 0.5439 0.6334 0.6399 1 0.8068

TAMRR 0.5764 0.5162 0.5634 0.5779 0.8068 1

	Citation count	HITS	PageRank	P-Rank	Mutual Rank	TAMRR
Citation count	1	0.9110	0.9569	0.9466	0.6343	0.5764
HITS	0.9110	1	0.7992	0.7911	0.5439	0.5162
PageRank	0.9569	0.7992	1	0.9874	0.6334	0.5634
P-Rank	0.9466	0.7911	0.9874	1	0.6399	0.5779
Mutual Rank	0.6343	0.5439	0.6334	0.6399	1	0.8068
TAMRR	0.5764	0.5162	0.5634	0.5779	0.8068	1

6 Conclusions and future work

AI and computer science methods are very useful for measuring the prestige of scientific papers. In particular, recursive PageRank-like methods provide intelligent ways to rank scientific papers according to their prestige rather than their popularity. However, many factors should be considered for ranking; thus, new methods have been proposed to account for time factors [7], sentiment similarity [32] and altmetric factors [37]. This study proposed the TAMRR model to further complement the currently existing PageRank-like methods, in the following aspects. First, our proposed model considers time effects by factoring in the time at which publications are cited by others; the shorter the time interval is, the more remarkable the paper is. Next, the credit allocation of different co-authors based on their contributions is considered. Furthermore, citation counts are used to weight the edges of the journal citation network. Finally, to account for the impact of journals and authors on publications, an AI algorithm, modified HITS algorithm, was used to mutually reinforce the prestige of publications, after the integration with the outputs of different weighted PageRank algorithms.

Based on the results of our experiments, we conclude that: prestige and popularity are two different concepts; thus, a more prestigious publication does not imply higher popularity, and highly popular publications are not necessarily prestigious. There are four principal situations: 1) papers have high prestige and high popularity, 2) papers have high prestige and low popularity, 3) papers have low prestige and high popularity, and 4) papers have low prestige and low popularity. By comparing our weighted PageRank with standard PageRank, it is found that factors such as the time interval, the distribution of contributions, and the number of citations, significantly affect ranking of publications, authors, and journals; in addition, the mutual reinforcement which indicates the interactive impact of authors and journals plays an important role while measuring the prestige of papers in the heterogeneous network. Ranking of publications using AI algorithms is promising for recommendation of publications in a more intelligent way. To increase the efficiency of ranking methods, more effective AI techniques and additional factors should be taken into account. In our future research, we will focus on additional factors such as the age of journals and the growth rate of citations during different time periods. Scenarios that add more information that is likely to influence the prestige of papers, authors, and journals should be explored.

Footnotes

Acknowledgments

This work has been supported by National Social Science Foundation of China (No.17NDJC168YB), National Natural Science Foundation of China (No. 51875503, No.51475410), and Zhejiang Natural Science Foundation of China (No. LY17E050010).

References

Teixeira

A.A.

, Vieira

P. C.

and Abreu

A. P.

, Sleeping beauties and their princes in innovation studies, Scientometrics 110(2) (2017), 543–580.

and Raan

V.J.

, Sleeping beauties in science, Scientometrics 59(3) (2004), 467–472.

Raan

A.F.J.V.

, Sleeping beauties cited in patents: is there also a dormitory of inventions? Scientometrics 110(3) (2017), 1123–1156.

Singh

P.A.

, Shubhankar

and Pudi

, An efficient algorithm for ranking research papers based on citation network. In Proceedings of 3rd Conference on Data Mining and Optimization, Putrajaya, Malaysia, June 28–29, 2011, pp. 88–95.

Cuccurullo

, Aria

and Sarto

, Foundations and trends in performance management. A twenty-five years bibliometric analysis in business and public administration domains, Scientometrics 108(2) (2016), 595–611.

, Pan

, Zhen

, Ma

, Yuan

, Guo

, Yu

, Ma

and Wu

, Prestige Rank: a new evaluation method for papers and journals, Journal of Informetrics 5(1) (2011), 1–13.

Fiala

, Time-aware PageRank for bibliographic networks, Journal of Informetrics 6(3) (2012), 370–388.

Lian

, Ye

, Zhu

, Liu

, Xie

and Xiong

, Mutual reinforcement of academic performance prediction and library book recommendation, In Proceedings of 16th IEEE International Conference on Data Mining (ICDM), Barcelona, Spain, 2016, pp. 1023–1028.

Walker

, Xie

, Yan

K.K.

and Maslov

, Ranking scientific publications using a model of network traffic, Journal of Statistical Mechanics: Theory and Experiment P06010 (2007), 10.1088/1742-5468/2007/06/P06010.

10.

, Wang

, Zhang

and Liu

, A multiple-link, mutually reinforced journal-ranking model to measure the prestige of journals, Scientometrics 111(1) (2017), 521–542.

11.

Zhou

, Orshanskiy

A.S.

Zha

and Giles

C.L.

, Co-ranking authors and documents in a heterogeneous network. In proceedings of 7th International Conference on Data Mining, Omaha, USA, October 28–31, 2007, pp. 739–744.

12.

Yan

, Topic-based PageRank: toward a topic-level scientific evaluation, Scientometrics 100(2) (2014), 407–437.

13.

Yan

and Ding

, Weighted citation: an indicator of an article’s prestige, Journal of the American Society for Information Science and Technology 61(8) (2010), 1635–1643.

14.

Yan

, Ding

and Sugimoto

C.R.

, P-rank: an indicator measuring prestige in heterogeneous scholarly networks, Journal of the American Society for Information Science and Technology 62(3) (2011), 467–477.

15.

Ayala-Gómez

, Daróczy

, Benczúr

, Mathioudakis

and Gionis

, Global citation recommendation using knowledge graphs, Journal of Intelligent and Fuzzy Systems 34(5) (2018), 3089–3100.

16.

Galyani-Moghaddam

, Jafari

and Sattarzadeh

, Publications by faculty members indexed in Science Citation Index and Scopus: an Iranian case study, The Electronic Library 35(6) (2017), 1247–1258.

17.

Qiao

, Wang

and Liang

, A value evaluation method for papers based on improved Page Rank algorithm. In Proceedings of 2nd International Conference on Computer Science and Network Technology, Changchun, China, December 29–31, 2012, pp. 2201–2205.

18.

Sayyadi

and Getoor

, FutureRank: ranking scientific articles by predicting their future PageRank. In Proceedings of SIAM International Conference on Data Mining, Nevada, USA, April 30–May 2, 2009, pp. 533–544.

19.

Hirsch

J.E.

, An index to quantify an individual’s scientific research output, Proceedings of the National Academy of Sciences of the United States of America 102(46) (2005), 16569–16572.

20.

Kleinberg

M.J.

, Authoritative sources in a hyperlinked environment, Journal of the ACM 46(5) (1999), 604–632.

21.

Zhang

, Zhang

, Sun

, Yao

and Yao

, Semantically linking events for massive scientific literature research, The Electronic Library, 35(4) (2017), 724–744.

22.

Egghe

, Theory and practise of the g-index, Scientometrics 69(1) (2006), 131–152.

23.

Bras-Amorós

, Domingo-Ferrer

and Torra

, A bibliometric index based on the collaboration distance between cited and citing authors, Journal of Informetrics 5(2) (2011), 248–264.

24.

Dunaiski

, Visser

and Geldenhuys

, Evaluating paper and author ranking algorithms using impact and contribution awards, Journal of Informetrics 10(2) (2016), 392–407.

25.

Kurtz

M.J.

Henneken

E. A.

, Measuring metrics-a 40-year longitudinal cross-validation of citations, downloads and peer review in astrophysics, Journal of the Association for Information Science and Technology 68(3) (2017), 695–708.

26.

Kosmulski

, A new hirsch-type index saves time and works equally well as the original h-index, ISSI Newsletter 2(3) (2006), 4–6.

27.

Lefebvre

, Applied stochastic processes, Springer Science and Business Media, (2007).

28.

Nykl

, Ježek

Fiala

and Dostal

, PageRank variants in the evaluation of citation networks, Journal of Informetrics 8(3) (2014), 683–692.

29.

, Guan

and Zhao

, Bringing PageRank to the citation analysis, Information Processing and Management 44(2) (2008), 800–810.

30.

, Hagen, Harmonic allocation of authorship credit: source-level correction of bibliometric bias assures accurate publication and citation analysis, PLoS ONE 3(12) (2008), 1–7.

31.

Chen

, Xie

, Maslov

Redner

, Finding scientific gems with Google’s PageRank algorithm, Journal of Informetrics 1(1) (2007), 8–15.

32.

Kazi

, Patwardhan

and Joglekar

, Towards a new perspective on context based citation index of research articles, Scientometrics 107(1) (2016), 103–121.

33.

Davis

M.P.

, Eigenfactor: does the principle of repeated improvement result in better estimates than raw citation counts? Journal of the American Society for Information Science and Technology 59(13) (2008), 2186–2188.

34.

P.S.

, Li

and Liu

, On the temporal dimension of search. In Proceedings of the 13th international World Wide Web conference on Alternate track papers and posters, New York, USA, May 17–22, 2004, pp. 448–449.

35.

Behnam

and Cagliano

, Be sustainable to be innovative: an analysis of their mutual reinforcement, Sustainability 9(1) (2016), 17–34.

36.

Brin

Page

, The anatomy of a large-scale hypertextual web search engine, Computer Networks and ISDN Systems 30(1–7) (1998), 107–117.

37.

Banshal

S.K.

Singh

V.K.

, Kaderye

, Muhuri

P.K.

and Sánchez

B.P.

, An altmetric analysis of scholarly articles from India, Journal of Intelligent and Fuzzy Systems 34 (2) (2018), 3111–3118.

38.

Bititci

U.S.

, Suwignjo

and Carrie

A.S.

, Strategy management through quantitative modelling of performance measurement systems International Journal of Production Economics 69(1) (2001), 15–22.

39.

Jiang

, Sun

, Yang

, Zhuge

and Yao

, Exploiting heterogeneous scientific literature networks to combat ranking bias: evidence from the computational linguistics area, Journal of the Association for Information Science and Technology 67(7) (2016), 1679–1702.

40.

Liu

X.Z.

Fang

, Fairly sharing the credit of multi-authored papers and its application in the modification of h-index and g-index, Scientometrics 91(1) (2011), 37–49.

41.

Ding

, Yan

, Frazho

and Caverlee

, PageRank for ranking authors in co-citation networks, Journal of the American Society for Information Science and Technology 60(11) (2009), 2229–2243.

42.

Liu

, Huang

, Wei

and Mao

, Tri-Rank: an authority ranking framework in heterogeneous academic networks by mutual reinforce. In Proceedings of 26th IEEE International Conference on Tools with Artificial Intelligence (ICTAI), Limassol, Cyprus, November 10-12, (2014), 493–500.