Abstract
A review of the literature on organizational rankings across management, sociology, education, and law reveals three perspectives on these complex evaluations—rankings are seen as a form of information intermediation, as comparative orderings, or as a means for surveillance and control. The information intermediation perspective views rankings as information products that address information asymmetries between the ranked organizations and their stakeholders; the comparative orderings perspective views them as representations of organizational status and reputation; and the surveillance and control perspective emphasizes their disciplining power that subjects ranked organizations to political and economic interests. For each perspective, we identify core contributions as well as additional questions that extend the current body of research. We also identify a new perspective—rankings entrepreneurship—which has been overlooked to date but presents significant opportunities to extend our understanding of the production and consumption of rankings.
Keywords
Organizational rankings, which are “publicly available comparative orderings of organizations, based on evaluation criteria determined by a ranking organization” (Martins, 2005: 701), have proliferated in the last three decades (e.g., Hazelkorn, 2014). The general media (e.g., Fortune’s “Most Admired Companies”), speciality media (e.g., Working Mother’s “100 Best Companies for Working Moms”), social monitors (e.g., Human Rights Campaign’s “Best Places to Work”), and governments (e.g., the European Commission’s “U-Multirank”) all publish rankings of organizations that establish their relative standing on various dimensions of performance. A decade ago, the Reputation Institute identified 183 public evaluations of companies in 38 countries (Fombrun, 2007), and that number has grown considerably since. Within university education alone, there are more than 50 since U.S. News & World Report published the first national ranking in 1983 (Yudkevich, Altbach, & Rumbley, 2015). Social media and user-generated content have recently hastened the production of rankings and expanded their reach (Scott & Orlikowski, 2012). As such, rankings have become important elements of organizations’ environments that significantly influence their interactions with stakeholders (Rindova & Fombrun, 1999; Rindova & Martins, 2012). Scholars have described their pervasiveness as an “audit culture” (Strathern, 2000), “audit explosion” (Power, 1994), “ranking regime” (Gonzales & Núñez, 2014: 3), and “ranking mania” (Rhoads, Li, & Ilano, 2014: 31).
The growth of rankings has been accompanied by scholarly interest across multiple disciplines. Yet, a comprehensive review of this growing literature remains unavailable. Within management research, for example, reviews on legitimacy (Deephouse & Suchman, 2008), reputation (Clardy, 2012; Lange, Lee, & Dai, 2011; Mariconda & Lurati, 2014; Sabate & Puente, 2003; Walker, 2010), status (Piazza & Castellucci, 2014; Sauder, Lynn, & Podolny, 2012), decoupling (P. Bromley & Powell, 2012), corporate governance (Aguilera, Desender, Bednar, & Lee, 2015), corporate social responsibility (CSR; Pope & Wæraas, 2016), corporate social performance (Peloza, 2009), and higher education (Thornton, 2014) have touched on rankings, but no systematic review of research on rankings and their various effects has been undertaken. 1
To address this gap, we review the literature across the four disciplines that have shown the greatest interest in studying rankings—management, sociology, 2 education, and law. Our results reveal that prior research examined rankings from one of three perspectives—as a form of information intermediation, as comparative orderings, or as a means to exercise surveillance and control. Each perspective emphasizes a different function of rankings and attends to different consequences. The information intermediation perspective views rankings as information products that reduce information asymmetries either by making private information public or by making public information more accessible and comprehensible to stakeholders (Rindova & Fombrun, 1999). As a result, its primary focus is on the quality of information that rankings provide and on their efficacy in measuring purported outcomes. The perspective on rankings as comparative orderings emphasizes how they stratify organizational fields and alter interactions between organizations and their audiences (Sauder & Espeland, 2009), with some studies using rankings as a proxy for status, defined as the relative social standing of a focal organization (e.g., Sauder et al., 2012), and others using them as measures of unobservable firm quality and, therefore, reputation (e.g., Fombrun & Shanley, 1990). Finally, a diverse group of scholars, writing from various social science disciplines, have begun to draw attention to the “dark side” of rankings as instruments of surveillance and control. Much of this research highlights how rankings reshape the power dynamics in organizational fields (e.g., Déjean, Gond, & Leca, 2004), and it stresses the distorting effects that they have on the missions of organizations such as business schools (Martins, 2005) and law schools (Sauder & Espeland, 2009).
Although published work tends to espouse one of the three perspectives, our review suggests that a better understanding of rankings emerges in considering the complex interactions among them. For example, Rao (1998: 936, 953) suggests that rankings provide “impartial information to consumers to make rational purchasing decisions” but also serve as “social control mechanisms that institutionalize distrust.” The fact that rankings organizations stipulate the criteria and data generation procedures, however, tends to muddy these relationships (e.g., Daniel, 1998; Déjean et al., 2004; Dichev, 1999; Rao, 1998). Sauder and Espeland (2009: 74) characterize rankings as “engines of status anxiety” that pressure organizations to disclose information but also incentivize them to focus their efforts on improving their relative position rather than investing in their own strategic visions. The conceptualizations of rankings as catering to bottom-up demands for information and top-down pressures for control are therefore related (Wedlin, 2006), and future research would benefit from exploring their interconnectedness. To capture the interconnections and tensions across perspectives, Figure 1 presents an overarching framework that shows how current research has viewed the links between audiences and ranked organizations that are created by rankings. We further highlight the importance of understanding rankings entrepreneurship—the initiation and production activities of organizations that publish rankings—which we identify as a missing perspective from current research.
Integrated Model of Research on Rankings
Based on our literature review, Figure 1 represents rankings as outcomes of informational needs of consumers, reputation and status aspirations of organizations, and surveillance and control interests of political and economic elites. The model highlights the influence of design characteristics on the quality and utility of rankings, as well as the effects of organizational and extraorganizational factors on organizations’ relative positions. Once established, organizational inclusion (or exclusion) and position in a ranking affect various outcomes. These effects depend on the extent to which rankings reduce qualitative distinctions among organizations to quantitative differences deemed precise, objective, and rational by organizations and their stakeholders. We highlight key processes (i.e., quantification, commensuration, reactivity, and disciplining) that contribute to the simplification and routinization of decision making that rankings encourage. Additionally, we identify characteristics of rankings, organizations, and stakeholders that moderate the effects of rankings. We also highlight that the mechanisms through which rankings affect outcomes have not been sufficiently examined in the literature.
Figure 1 also emphasizes the centrality of the producers of rankings in this complex system. This suggests the need for more systematic attention to the entrepreneurs who produce rankings to understand the links between the information demands that motivate the production of rankings and their effects on the audiences that consume them. While we found some references to rankings entrepreneurs in our review (e.g., Sauder & Fine, 2008), there is a clear gap in the literature regarding the production and consumption of rankings. We therefore propose rankings entrepreneurship as a distinctive perspective that requires further research and extends work in the area beyond the current discussion, which has tended to center on the rankings themselves and their effects. While such work has helped us understand how rankings are endowed with institutional power despite questions about their quality, it is also important to acknowledge the human origins of rankings and examine the entrepreneurial processes that shape rankings and their effects. Specifically, we identified a lack of attention to two areas of rankings entrepreneurship. First, the motives of “rankings entrepreneurs” (Chandler & Rindova, 2008), those who initiate new rankings, are not well understood. Although some research has focused on motives behind the production of rankings (e.g., Déjean et al., 2004; Rao, 1998), we know little about how the needs, aspirations, and interests of various field-level actors generate entrepreneurial opportunities for individuals and organizations to create new rankings. Second, the processes through which rankings entrepreneurs mobilize various resources and develop the measurement systems they deploy remain poorly understood. To the degree that they are involved in a quantification enterprise, the quality of the measurement system is critical to the value of the quantification effort (T. M. Porter, 1995). Relatedly, how rankings entrepreneurs work with and appeal to their stakeholder audiences and whether they benefit the decisions and actions of the audiences whose interests they purport to represent have not been considered in extant research. While anecdotal evidence exists that audiences use rankings once they are in place, we know little about whether the audiences understand or agree with the criteria used to generate them and whether rankings provide audiences with information that improves their decisions and choices. At the end of our review, we elaborate on these core issues that are brought to the foreground when rankings are viewed through the rankings entrepreneurship perspective.
As rankings become more prominent and scholars devote more attention to them, a comprehensive review of what we know and do not know provides a strong foundation for future research. Furthermore, a systematic understanding of rankings has significant implications for policy and practice (Colyvas, 2012). To this end, in addition to our comprehensive review, we offer some practical implications for ranking and ranked organizations.
Methods
Consistent with prior reviews (e.g., Narayanan, Zane, & Kemmerer, 2011; Piazza & Castellucci, 2014), we conducted an extensive search of the literature for published work on rankings. First, we searched the EBSCO and PsycINFO databases to find relevant articles on rankings in leading management journals (e.g., Academy of Management Journal, Academy of Management Review, Administrative Science Quarterly, Journal of Applied Psychology, Journal of Management, Organization Science, and Strategic Management Journal). Second, we searched in specialized journals known to publish research on rankings (e.g., Academy of Management Learning & Education, Corporate Reputation Review, Journal of Business Ethics, and Scientometrics). Third, given the multidisciplinary nature of research on rankings, we searched the ERIC and Sociological Abstracts databases, focusing on articles published in the disciplinary journals of education (e.g., American Journal of Education, Higher Education, and European Journal of Education), sociology (e.g., American Journal of Sociology, American Sociological Review, European Journal of Sociology, and Social Anthropology), and law (e.g., Indiana Law Journal and Law & Social Inquiry). To make the review manageable, we focused on articles published since 2010 in these journals, but we used the reference lists of the papers that we reviewed to identify additional prior relevant articles, book chapters, and books. From the more than 4,000 articles identified in our search, we excluded those using the term “ranking” with different connotations (e.g., rank of an employee) and those focusing on rankings based solely on objective performance indicators (e.g., the Fortune 500). Our final sample consisted of 436 articles—247 in management and 189 in cognate disciplines.
This literature review led us to identify three analytically distinct but interrelated perspectives. First, researchers working from strategy and economics perspectives tend to study rankings as a form of information intermediation that reduces asymmetries between ranked organizations and their stakeholders. Second, scholars working from sociological and management perspectives tend to study rankings as comparative orderings that structure organizational fields. Third, researchers from sociology and education perspectives tend to study rankings as a means for surveillance and control over ranked organizations. In what follows, we articulate the focal themes that emerge within these three perspectives, highlight their distinctive contributions, and identify gaps that suggest opportunities for future research.
Rankings as Information Intermediation
A large body of research views rankings as information products that either make private information public or make public information more accessible and comprehensible (Porac, Ventresca, & Mishina, 2002; Rindova & Fombrun, 1999). This research focuses primarily on the information utility and quality of rankings.
Information Utility
Rankings are seen as addressing stakeholder problems of information gathering and decision making by aggregating information and the perceptions of different audiences on preselected performance criteria and by making the aggregated data available in comprehensible comparative orderings (Rindova, 2004). In markets where quality assessments are difficult or subjective and products and services are either “experience goods” or “credence goods” (Darby & Karni, 1973), such as gastronomy (Rao, Monin, & Durand, 2005) and education (Tutterow & Evans, 2016), rankings mediate exchanges between producers and consumers. They purportedly reduce technical uncertainty, which refers to the degree to which quality can be inferred according to known performance criteria, and performance standard uncertainty, which refers to establishing “the standards of desirability” (Graffin & Ward, 2010: 331).
In terms of utility for ranked organizations, rankings supply definitions of success, which establish the criteria and standards of desirability (Rindova & Fombrun, 1999) and feedback regarding performance on those criteria (Elsbach & Kramer, 1996; Martins, 2005). For example, rankings such as the Great Place to Work Institute’s “100 Best Companies to Work for in America” provide firms with measures of their workplace quality and human resource practices (Edmans, 2012). As such, rankings are a source of comparative data that inform strategic planning (e.g., Hou, Morse, & Chiang, 2012), facilitate competitive benchmarking (Carpenter-Hubin & Crisan-Vandeborne, 2016), and motivate strategic change (e.g., Locke, 2014; Martins, 2005). By prompting organizations to address stakeholder concerns (McGrath, 2007), rankings also stimulate the adoption of ethical and socially responsible practices (Waddock, 2008), potentially orienting organizations toward new forms of value creation (Chandler, 2015).
Although rankings are seen as important sources of information by audiences, researchers have questioned their utility. Research suggests that rankings are less likely to influence well-informed stakeholders (Vidaver-Cohen, 2007), such as investors, as they lack new information (Abraham, Friedman, Khan, & Skolnik, 2008). Stakeholders ignore them on dimensions that they do not care about, such as CSR (Pope &Wæraas, 2016), and supplant rankings with more credible information signals, such as advertising (Thornbury & Brooks, 2010). Organizations, too, vary in their reliance on rankings (Peloza, 2009; Piro & Sivertsen, 2016) relative to other information sources, such as customized research, financial performance indicators, and media coverage (Kitchen & Laurence, 2003). Beyond decision utility, the cost of providing the data in required formats and interpreting the results tend to outweigh the information benefit for the ranked organization (e.g., Carpenter-Hubin & Crisan-Vandeborne, 2016). Rankings are seen as “backwards-looking” and pointing to areas of improvement without guidance to the required actions (Brønn & Brønn, 2005: 46). When rankings are seen as irrelevant to identity, they have little impact on strategy or organizational change (e.g., Elken, Hovdhaugen, & Stensaker, 2016).
Furthermore, some studies suggest that rankings actually have harmful effects, as they are too narrow (McCormack, Lynch, & Hennessy, 2015), ignore variation in stakeholder preferences (e.g., Carpenter-Hubin & Crisan-Vandeborne, 2016; Glick, 2008), force comparisons among organizations with different missions (Carpenter-Hubin & Crisan-Vandeborne, 2016; Martins, 2005), and ultimately redefine rather than establish standards of excellence and value (e.g., Power, 1994, 2004; Power, Scheytt, Soin, & Sahlin, 2009; Wedlin, 2006). They therefore shift attention to new, though not necessarily better, performance criteria (e.g., Biesta, 2009).
Some scholars take a different point of view, arguing that, with reasonable expectations, rankings can facilitate decision making, especially if they are designed as “multi-dimensional transparency tools” that factor in the multiplicity of stakeholder interests (Van Vught & Westerheijden, 2010). Researchers (e.g., Iacobucci, 2013; Robinson-García, Torres-Salinas, López-Cózar, & Herrera, 2014) suggest that stakeholder audiences are best served by multiple rankings and the recognition that some rankings are appropriate for short-term planning while others can inform medium- and long-term decision making (Hou et al., 2012).
Informational Quality
Information quality refers to the extent to which information is “intrinsically good, contextually appropriate for the task, clearly represented, and accessible” (R. Y. Wang & Strong, 1996: 6). The question of information quality, therefore, is largely seen as a question of the soundness of measurement criteria, and researchers have expressed a great deal of concern about the methodologies used to construct rankings (e.g., Adler & Harzing, 2009; Safón, 2013). First, the choices of normalization, weighing, and aggregation schemes have been criticized as arbitrary (e.g., Adler & Harzing, 2009; Billaut, Bouyssou, & Vincke, 2010). Scholars have pointed out that mixing input, throughput, and output attributes and using “weight and sum” assumptions lead to problematic interpretations (e.g., D. Bromley, 2002; Eccles, 2002; Rindova, Williamson, Petkova, & Sever, 2005; Soh, 2015). For college rankings, issues of intercorrelations, redundant indicators, and discrepancies between assigned and attained weights have been reported (e.g., Soh, 2012a, 2015). Similarly, assigning fixed and exogenous weights across organizations ignores their diversity (De Witte & Hudrlikova, 2013). The relatively arbitrary aggregation of noisy information by rankings has been blamed for fluctuations not explained by actual changes in performance (e.g., Bookstein, Seidler, Fieder, & Winckler, 2010; Dichev, 1999; Tofallis, 2012). In response to these problems, researchers have suggested solutions that include utilizing objective weights obtained from data analysis (Freyer, 2014), complementing objective weights with subjective information based on theoretical frameworks or expert opinion (J. Ding & Qiu, 2011), replacing scaling relative to the best performer by a fixed gains approach (Docampo & Cram, 2014), as well as using more sophisticated statistical techniques (Benito & Romera, 2011; Billaut et al., 2010; Jeremic, Bulajic, Martic, & Radojicic, 2011; Tofallis, 2012).
Second, the choice of audiences surveyed has been argued to limit the validity of rankings, rendering them idiosyncratic measures that reflect the interests of specific stakeholders who may not be interested in attributes that others value (e.g., Clardy, 2012; Gioia & Corley, 2002). Such criticisms are partially countered by arguments that stakeholder audiences may have convergent interests, such as stock analysts and peer executives in Fortune’s “Most Admired Companies” rankings (Love & Kraatz, 2009), as well as industrial buyers and trade journalists in Journal du Textile’s ranking of fashion houses (Godart, Maddux, Shipilov, & Galinsky, 2015).
Third, scholars have raised questions about the appropriate unit of analysis for rankings. For example, Adler and Harzing (2009) argue that the aggregation of campus-level data to compute overall university rankings is arbitrary and meaningless. Others have suggested that organizational-level rankings may be less appropriate than those for individual departments (Lazaridis, 2010) or areas of specialization (López-Illescas, de Moya-Anegón, & Moed, 2011).
Fourth, scholars have examined the extent to which rankings of the same organizational populations converge. While rankings of environmental performance (e.g., Elsayed, 2006), law schools (Hitt, Bierman, Shimizu, & Kochhar, 2001), MBA programs (Iacobucci, 2013), and universities (e.g., Boliver, 2015; Claassen, 2015) are highly correlated across multiple rankings within their fields, Dichev (1999) finds that contemporaneous changes in rankings of business schools are uncorrelated. Also, Fortune rankings of social performance have been found to be inconsistent with the Kinder, Lydenberg, and Domini index that captures firms’ environmental, social, and governance performance (Liston-Heyes & Ceton, 2009), and Chan, Fung, Fung, and Yau (2016) show that research rankings of schools differ with the criteria used. These studies cast significant doubt on rankings as reliable measures of performance.
Future Research Opportunities
Research on rankings from an information intermediation perspective indicates mixed support for the core assertion that rankings facilitate an assessment of unobservable quality that stakeholders value. We propose that research in the following directions may address this problem. First, specifying the mechanisms through which rankings influence market exchanges (see Figure 1) may begin to shed light on what ranking attributes address what type of information needs. For example, information intermediaries create value by increasing the availability and quality of information (Baker, Choi, & Gulati, 2006; Feddersen & Gilligan, 2001), but the substantive effects of rankings on audiences appear to derive from their framing effects based on their form as simple quantitative comparisons (Chandler & Rindova, 2008). Consistent with this argument, Luca and Smith (2013) show that when ranked schools were presented in alphabetical order, rankings had no causal impact on applicant decisions. This finding suggests the importance of research that distinguishes between the informational and framing effects of rankings and investigates them systematically.
Second, we see a need for more studies that illuminate the distinctive nature of rankings relative to other social measures of organizational performance, such as benchmarks, ratings, certifications, and consecrations/awards (Graffin & Ward, 2010). Rindova (2004) proposes three kinds of assessments: reputational rankings generated through opinion polls, expert ratings based on audits, and awards based on corporate contests. Graffin and Ward (2010) distinguish between rankings as relative assessments based on comparison with other organizations and ratings and accreditations as absolute assessments based on standards. Systematic analysis of these forms of evaluation will give us a more complete picture of how audiences gauge unobservable organizational attributes and how that affects market consequences.
Rankings as Comparative Orderings
The perspective of rankings as comparative orderings views them as a “public, stable system of stratification comprised of unstable positions” that affects how organizations and their stakeholder audiences interact in an organizational field (Sauder & Espeland, 2009: 79). Within this perspective, some studies use rankings as a proxy for status (i.e., an organization’s relative standing), while others use them as measures of reputation (i.e., unobservable organizational quality). Espousing the reputation view, Fombrun and Shanley (1990: 234) characterize rankings as “signals about firms’ activities, achievements, and prospects” that crystalize reputations into visible comparative orderings (Clardy, 2012; Lange et al., 2011; Mariconda & Lurati, 2014; Sabate & Puente, 2003). Espousing the status view, Sauder and colleagues (2012: 269) define rankings as not “simply an affirmation of underlying differences in the desirable qualities.” By gathering evaluations on criteria that may have little to do with audience preferences, rankings create and transform perceptions of organizations. 3 Rankings therefore do not just reflect market exchanges but form part of the structural conditions that shape them (Wedlin, 2006). They define the structure and the basis of the hierarchy in a field (Sauder, 2006). Despite differences in emphasis on the extent to which rankings reflect quality, scholars working from this perspective share similar concerns about the antecedents and consequences of rankings.
Antecedents of Comparative Orderings
The lion’s share of research into the antecedents of comparative orderings has focused on the effects of prior financial performance on rankings via a “halo effect” (e.g., Brown & Perry, 1994). Several studies have considered the effects of firm strategies, such as diversification (Williams, Schnake, & Fredenberger, 2005), competitive market actions (Basdeo, Smith, Grimm, Rindova, & Derfus, 2006), norm-conforming behaviors (Philippe & Durand, 2011), CSR (Martínez-Ferrero, Banerjee, & García-Sánchez, 2016), and the use of mission statements (Erwin, 2011). Leadership and governance attributes, such as CEO succession (Flatt, Harris-Boundy, & Wagner, 2013), director’s foreign work experience (Lau, Lu, & Liang, 2016), and presence of outside and female directors on boards (Bernardi, Bosco, & Columb, 2009; Zhang, Zhu, & Ding, 2013), have been found to have positive effects on ranking positions. In contrast, contested governance practices, such as “poison pills” (Bednar, Love, & Kraatz, 2015) and layoffs (Flanagan & O’Shaughnessy, 2005; Love & Kraatz, 2009), tend to have negative effects. 4
Consequences of Comparative Orderings
Individual-level consequences
Rankings have consequences for organizations and the individuals affiliated with them. Employees in highly ranked organizations receive larger pay raises (Bidwell, Won, Barbulescu, & Mollick, 2015) and more attention from executive search firms (Hamori, 2010) while enjoying more job security (Ahmadjian & Robinson, 2001). Former CEOs of highly ranked firms enjoy more legitimacy when they join a new firm (Graffin, Carpenter, & Boivie, 2011), and managers with degrees from highly ranked universities are more influential in top management teams (Barsade, Ward, Turner, & Sonnenfeld, 2000). Security analysts at highly ranked banks are more respected for their advice (Giorgi & Weber, 2015) and have greater autonomy in issuing ratings (Hayward & Boeker, 1998). Scientists at highly ranked universities have more opportunities to join corporate advisory boards (W. W. Ding, Murray, & Stuart, 2013), have higher research impact (e.g., Podsakoff, MacKenzie, Podsakoff, & Bachrach, 2008), and are more likely to develop new schools of thought (Ofori-Dankwa & Julian, 2005). PhD graduates of highly ranked universities obtain more prestigious jobs (e.g., Podsakoff et al., 2008), produce more impactful research (e.g., Judge, Weber, & Muller-Kahle, 2012), and have better career options (Bedeian, Cavazos, Hunt, & Jauch, 2010). Among all graduates, this effect holds despite evidence that they do not receive a better education (e.g., Navarro, 2008; Rubin & Dierdorff, 2009) and may have lower learning satisfaction (Huang, Chen, & Chien, 2015).
Organizational-level consequences
At the organizational level, higher rankings increase survival rates (Rao, 1994) and improve performance (e.g., Cho & Pucik, 2005; T. Porter & Miles, 2013), even during market crises (Sánchez, Sotorrío, & Díez, 2012). High rankings are also linked to better stock market performance (e.g., Aaron, McMillan, & Cline, 2012; Edmans, 2012; Lyon & Shimshack, 2012), moderate the effects of investments in CSR (Servaes & Tamayo, 2013), and reduce the effect of negative earnings surprises (Pfarrer, Pollock, & Rindova, 2010).
Rankings are particularly influential in higher education, where an organization’s rankings are positively related to recruiters’ demand for its graduates (Rindova et al., 2005), the size of its endowment (Rutherford, Parks, Cavazos, & White, 2012), the rate of its technology licensing (Sine, Shane, & Gregorio, 2003), as well as other financial indictors, including research funding, alumni donations, and tuition (Bastedo & Bowman, 2011). MBA program rankings affect the ability to attract faculty (D’Aveni, 1996) and prospective students (e.g., Alter & Reback, 2014; Bowman & Bastedo, 2009; Luca & Smith, 2013), especially high-quality applicants (Horstschräer, 2012). Highly ranked firms are also more likely to be benchmarked and to influence other firms (Still & Strang, 2009), show a more substantive commitment to sustainability (Perez-Batres, Doh, Miller, & Pisani, 2012), and be more sensitive to the unethical acts of partner firms (Sullivan, Haunschild, & Page, 2007).
Future Research Opportunities
Our review finds that while the comparative ordering perspective comprises work that treats rankings as indicators of status and reputation, empirical research examines primarily the antecedents and consequences of rankings for ranked organizations and their stakeholders. This convergence is not surprising, given the commonalities between status and reputation as social approval assets based on stakeholder perceptions and evaluations (Pfarrer et al., 2010; T. G. Pollock, Lee, Jin, & Lashley, 2015). Furthermore, as Fombrun and Shanley (1990) argued, rankings reflect “reputation judgments” that anticipate future relative performance and “status judgments” that locate the firm in a comparative order (see also Bitektine, 2011). However, in light of a growing interest in the differential effects of status and reputation (Ertug & Castellucci, 2013; George, Dahlander, Graffin, & Sim, 2016; T. G. Pollock et al., 2015), future research should seek to separate the effects of ranking attributes that indicate status or reputation.
One possible avenue of progress rests on the differential stability of status, characterized as “stable, unitary, and relational,” and reputation, seen as “less stable, multidimensional” and based on perceptions of quality and performance (Chandler, Haunschild, Rhee, & Beckman, 2013: 217). Future research, therefore, can explore the use of positional stability rather than actual rank as an indicator of status. Indeed, studies find evidence that the stickiness in rankings (e.g., Ang & Wight, 2009; Iacobucci, 2013; Schultz, Mouritsen, & Gabrielsen, 2001) generates the “Matthew effect” that becomes self-reinforcing and amplifies the advantages to higher-ranked organizations (e.g., D’Aveni, 1996; Sine et al., 2003). A different interpretation of the self-reinforcing effects of rankings suggests that prior rankings anchor the assessments of survey respondents (Bastedo & Bowman, 2010; Bermiss, Zajac, & King, 2014; Bowman & Bastedo, 2011), irrespective of actual status differentials. In either case, the stability of rankings is important to understand their status effects, as the “circular effects” may solidify a position that, once achieved, “merely requires ordinary prudence” to maintain (Marginson, 2009: 32).
In contrast, other studies find significant annual variability in the ranking position of some organizations (Piro & Sivertsen, 2016), especially for performance on specific attributes (Carter & Ruefli, 2006; Iacobucci, 2013). Such variation is attributed to statistical noise (e.g., Bookstein et al., 2010), changes in rankings criteria and methodology (e.g., Tofallis, 2012), and exogenous events (e.g., Piro & Sivertsen, 2016). While some studies find changes to be “transitory and reversible” (Dichev, 2001: 237), others note that changes vary across status tiers and that significant changes at the top are possible only with substantial and sustained resource commitments (Gnolek, Falciano, & Kuncl, 2014). Collectively, these findings indicate that fine-grained analysis of the relative stability of ranking positions and the consequences for firms with stable and unstable positions will help refine the understanding of reputational and status effects.
Further advances may be made by systematic consideration of the different tiers within the comparative orderings produced by rankings, as well as the simple effect of being included or excluded. For example, Guyer (2010: 123, 126) suggests that rankings “tend to produce a parabolic curve: very few at very high value at the top, descending in a curve to very many of very low value at the bottom.” She notes that fewer than 20 hospitals were included in the U.S. News & World Report’s “Honor Roll,” while the remaining 5,000+ hospitals in the same category were relegated to “near-irrelevance.” From a reputation point of view, inclusion in rankings is a valuable signal, whereas absence equates to a lack of quality. As such, future research should seek to understand the differential effects of status and reputation by comparing actions and outcomes across ranked tiers, because prior work suggests that status effects may operate at the very top, whereas reputation effects may matter more at lower levels.
In seeking to disentangle the effects of status and reputation in rankings, future research should also compare the potential status effects of rankings with the effects of more unambiguous measures of status, such as network position (Benjamin & Podolny, 1999; Podolny, 1993). Research comparing differential network effects suggests a negative effect of high status and a positive effect of reputation. For example, Chandler et al. (2013: 217) show that “high-status firms have networks that are higher in partner quality but are less diverse and contain fewer opportunities to bridge structural holes than the networks of high-reputation firms.” Similarly, Ertug and Castellucci (2013) show that the reputation of resource providers has a stronger effect on an organization’s product quality, but their status has a stronger effect on revenues. These studies suggest that future research should consider more systematically the different effects of rankings on performance in light of the broader research on status and reputation.
Another area that has attracted surprisingly little attention is the strategic actions that organizations can take to influence their ranking positions. Because many rankings are based on the subjective assessments of stakeholders, organizations could influence the constituents involved. Although some research has shown how management “fashion setters,” such as the business media, influence evaluators’ attention to specific indicators (Bermiss et al., 2014), we know little about how organizations seek to influence stakeholder audiences directly in an effort to improve their positions in rankings. Evidence suggests that respondents to rankings surveys are influenced by identity concerns in ways that favorably reflect their individual and social selves (Peters, Daniels, Hodgkinson, & Haslam, 2014) and that organizations can strategically craft communication utilizing different message genres to appeal to potential survey respondents (Courtright & Smudde, 2009; Geppert & Lawrence, 2008). Future studies could examine how such symbolic management is instrumental in influencing performance in rankings.
Finally, future research should study more systematically the effects of negative rankings by identifying the factors that influence inclusion and position in rankings of “contemptible” companies, such as 24/7 Wall St.’s “The Worst Companies to Work For.” Experimental studies on firm familiarity, for example, suggest that familiar firms are more likely to appear on rankings of the worst companies (Brooks, Highhouse, Russell, & Mohr, 2003). Negative rankings appear to have some poorly understood positive effects, such as firms ranked among the worst enjoying positive abnormal stock returns, possibly due to the resulting publicity or to contrarian investors who expect such firms to be likely to reform (Johnson, Ellstrand, Dalton, & Dalton, 2005).
Rankings as Means of Surveillance and Control
The research reviewed within this perspective comprises a diverse mix of studies from different disciplinary and theoretical perspectives that share a concern about rankings as manifestations of larger sociopolitical processes, such as commensuration (Espeland & Stevens, 1998), quantification (Espeland & Stevens, 2008; T. M. Porter, 1995), and auditing (Power, 1994; Shore & Wright, 2015). While authors vary in their implied stance toward rankings, with some seeing them as part of increasing accountability and oversight and others seeing them as a “technology of surveillance,” a form of distributed power that “disciplines” organizations by reducing their autonomy and differentiation, they share at least two fundamental notions. First, the quantification underlying rankings is less about facilitating market selection and more about serving a political purpose—“not so much about trust as it is about power and control of political and economic elites” (Fligstein, 1998: 326). Second, the naturalization of rankings is attributed to societal discourse embedded in a neoliberal framework emphasizing effectiveness and efficiency (Amsler & Bolsmann, 2012; Lynch, 2015; Raptis, 2012). We summarize this work in two subthemes: rankings as surveillance technologies and resistance to rankings.
Rankings as Surveillance Technology
Commensuration, quantification, and control
Researchers working from this perspective link the origins and spread of rankings across fields to wider historical processes of surveillance and control (Power, 2004). They view rankings as a form of soft law (P. Bromley & Powell, 2012), or normative control, that inhibits organizations’ strategic actions by conferring relative advantages to those that conform (Wilkins & Huisman, 2012). Rankings in this perspective are understood as technologies of oversight that constitute, rather than merely represent, the social world that they claim to assess.
Espeland and colleagues (Espeland, 2002; Espeland & Stevens, 1998, 2008; Espeland & Vannebo, 2007; Stevens & Espeland, 2004) have drawn attention to the role of commensuration, the “expression or measurement of different entities according to a common metric” (Stevens & Espeland, 2004: 375), in this constitutive dynamic. Commensuration, they argue, reduces all qualitative differences to quantitative ones and enables the measurement and comparison of complex and intangible attributes. It reflects changes in larger societal processes of valuation, trust, and control, offering the technical advantages of simplifying decision making and the political advantages of enabling control. However, it comes with the high costs of values being objectified, entities being decontextualized, and social relations being distorted (Espeland & Stevens, 1998). The effects of commensuration are reinforced by quantification. Providing a cultural and political account of quantification, T. M. Porter (1995: 74) explains that “strict quantification, through measurement, counting, and calculation, is among the most credible strategies” for addressing an “ideal of objectivity,” which is “a political as well as a scientific one.” The proliferation and power of rankings therefore stem from the relentless demand for mechanical objectivity across fields (T. M. Porter, 1995).
This line of research brings to the forefront the social implications of such mechanical objectivity. “Reactivity,” defined as the tendency of organizations to attempt to succeed in the evaluation criteria of rankings rather than pursue autonomous and differentiating value creation strategies, has been identified as one unintended consequence. The simplified cognition and unreflective decision making that commensuration engenders are argued to be mechanisms underlying this reactivity to rankings (Espeland & Sauder, 2007; Hansen & Porter, 2012; Sellar, 2015). In the extreme, rankings can generate pressures on managers to meet higher expectations that lead organizations to engage in illegal behaviors (Mishina, Dykes, Block, & Pollock, 2010).
Building on the idea of reactivity, Sauder and Espeland (2009: 64) conceptualized rankings as disciplinary practices that exercise power over organizations “through incentives that are simultaneously seductive and coercive.” They draw on Foucault’s (1977) idea that power in modern society operates in subtle ways through freely adopted disciplinary practices that constitute how social actors perceive, behave, and relate to one another. Disciplinary power, according to Foucault (1977: 177), “sustains itself by its own mechanism” of continuous surveillance and normalization. Sauder and Espeland’s (2009) study of law schools shows how the two central processes of discipline—surveillance and normalization—changed how law schools made sense of their organizational field. Furthermore, they found that resistance by deans and leaders of professional organizations amplified the influence of rankings. Overall, a key insight from this work is that quantification removes and disincentivizes individuation and increases conformity (Sauder & Espeland, 2009). 5
Effects of oversight on fields and society
Another large body of work on the effects of rankings-based oversight focuses on higher education, highlighting the “odd combination of marketization on one hand, and centralization of control on the other” (Apple, 2005: 382) that rankings create. Unlike the critiques within the information intermediation perspective that focus on the consequences of inadequate quality of the measurement systems that rankings employ, this research highlights the changes in rationality norms and professional ethics that rankings promote (Gewirtz & Cribb, 2013). In this view, by shifting the locus, terms, and mechanisms of accountability, rankings produce a “low trust regime” (Perryman, Ball, Maguire, & Braun, 2011: 181) that leads to “marketization” and erodes traditional academic values and institutional commitments, with the loss of faculty autonomy, academic rigor, and critical discipline-based knowledge (e.g., Collyer, 2013; Erkkilä & Piironen, 2014; Lo, 2014; Zell, 2005).
The pressures for conformity discussed earlier lead organizations to copy top-ranked counterparts, resulting in the rapid diffusion of practices (Paradeise & Thoenig, 2013; Stake, 2006), uniformity in research methodologies (Harmon, 2006; Murillo & Vallentin, 2016), homogenization of knowledge (Gonzales & Núñez, 2014), and the “commoditization of an otherwise unique product,” such as education (Birnik & Billsberry, 2008; Martins, 1998: 296). By forcing other organizations to benchmark themselves against top-ranked ones, rankings inhibit change and renewal. In essence, the “homogenizing imperative,” as Thornton (2014: 27, 30) highlighted in a review of legal education, “exercises a centripetal pull,” even when new organizations create unique sources of competitive advantage.
Resistance to Rankings
The majority of the work on rankings as means for surveillance and oversight stresses that organizations have internalized rankings (Power et al., 2009; Sauder & Espeland, 2009). Depending on their history, resources, local contexts, and ranking position, however, organizations also resist or manipulate rankings (Amsler, 2014; Locke, 2014). For example, some universities are adopting alternative benchmarks that focus on regional settings (Alperin, 2013) and upholding universal values (Tan & Goh, 2014). Rather than passively subjecting themselves to external evaluation, organizations interpret rankings to serve their need to benchmark performance, convey credentials, and manage stakeholder expectations, thereby questioning rankings’ utility and weakening their influence via competing narratives (O’Connell & Saunders, 2012). Some scholars have called for alternative rankings to contain the distorting effects of any one ranking (Billaut et al., 2010).
Whether an organization responds to rankings with impression management and identity claims (Elsbach & Kramer, 1996), alterations to product offerings (Durand, Rao, & Monin, 2007) and prices (Askin & Bothner, 2016), or strategic change (Martins, 2005), researchers agree that rankings are pervasive. As rankings become a “hegemonic measure of competitiveness,” organizations find themselves forced to “play the game” with little ability to quit, protest, or win (Corley & Gioia, 2000: 320, 323). Espeland and Vannebo (2007: 40) find that law schools now see rankings as a “permanent if unwelcome part of the landscape, and resistance has shifted from trying to stop them to constructing more credible rankings.” While some scholars advocate for a temporary moratorium (e.g., Adler & Harzing, 2009), others have argued that such resistance is impractical (Worrell, 2009) and may only spur greater interest and legitimacy (Glick, 2008).
Some scholars have suggested political and ideological responses to the hegemony of rankings (Gilbert, 2015). Shore and Wright (1999: 572), for example, observe that audit systems acquire “functional legitimacy” by relying largely on the “fear, expectations of compliance and a lack of imagination regarding the possibility of alternatives.” As such, one response is to use “political reflexivity”—action that requires the alternative semantics of accountability and a knowledge of power to criticize the rankings rationality. Amsler and Bolsmann (2012: 295) argue that resistance is unlikely to have “transformative effects” as long as the “logic of ranking itself remains uncontested,” and they call for “problematising the practice of ranking itself.” In the same vein, management scholars (e.g., Walsh, 2011) allude to radical alternatives in which education and research are not driven by rankings’ “consequentialist thought” but by an alternative worldview based on “self-conceptions, identities, and proper behavior” (March, 2003: 205-206). Collectively, these ideas point out that efforts to resist, irrespective of efficacy, at least denaturalize the hegemony of rankings (Amsler, 2014).
Future Research Opportunities
Due to its multidisciplinary depth, the surveillance and control perspective on rankings offers rich avenues for future research. Although management scholars have characterized the ranking process as a “very powerful control system” that shapes firm action (Martins, 1998: 295), the implications of this view have yet to be fully incorporated into management research.
First, research on power and surveillance has emphasized the systemic nature of the power of rankings but has not fully considered the moderating influence of context. For example, Sauder and Espeland (2006) found that the effects of rankings in fields with multiple rankings are different from their effects in fields with one dominant ranking. Also, the extent of diffusion and responses to social measures varies across regulatory and competitive environments (Sharkey & Bromley, 2015). Significant opportunities therefore exist to extend this line of inquiry by taking into account the effects of influential institutional actors (Rao, 1994) and competition among rankings (Chatterji, Durand, Levine, & Touboul, 2016).
Second, our review suggests that the literature would benefit from studying variation in the nature and extent of organizational resistance to rankings (e.g., T. Wang, Wezel, & Forgues, 2016). The goal here is to understand “whether, where, when and how rankings serve as an incentive for change” and examine “not just the differences in the degree of change, but also variations in the nature of that change” (Locke, 2014: 80; italics in original). One way to examine this is to unpack the varied legitimacy expectations from external constituents that firms face. Although some research suggests that an organization’s rank affects the extent of public scrutiny that it draws (e.g., Evans, Treviño, & Weaver, 2006), understanding how that affects the organization’s discretion to resist such pressures remains to be examined.
Third, our review reveals significant opportunities for research to explore the limits of rankings and the quantification and commensuration processes on which they rely. An intriguing study by Hirschman, Berrey, and Rose-Greenland (2016: 298) shows that the authority that quantification accords to rankings is contestable. They studied the demise of the quantitative system for admissions at the University of Michigan and found that its transparency, contentiousness, and substitutability (with an alternative qualitative process based on a holistic review of all applicants) contributed to its weakening. Understanding when systems of quantification are “neither inevitable nor irreversible” is essential.
Fourth, institutional theory may provide interesting avenues to explore the range of responses available to ranked firms and the conditions under which they can better manage pressures to conform. Although institutional theory has been used to understand the effects of rankings (Wilkins & Huisman, 2012), more recent research on firm responses to varying institutional pressures (e.g., Chandler, 2014) and managing institutional complexity (e.g., Greenwood, Raynard, Kodeih, Micelotta, & Lounsbury, 2011) has yet to be applied to rankings. For example, Rasche and Gilbert (2015) argue that the presence of prominent rankings by media and the absence of equally influential specialized rankings create situations of competing institutional pressures that jeopardize the promotion of responsible management education. Given the growing interest in institutional pluralism (Kraatz & Block, 2008), exploring how firms manage multiple, often competing, expectations while remaining true to their underlying values (Kraatz, Ventresca, & Deng, 2010) would further our understanding of how organizations can protect their autonomy from the homogenizing pressures of rankings.
Summary of Findings and Contributions
We uncovered three distinct perspectives in research on rankings—as forms of information intermediation, comparative orderings, and means of surveillance and control. Within the information intermediation perspective, rankings have been analyzed for the validity of measurement and the quality of quantification procedures employed. The comparative orderings perspective has emphasized how rankings stratify organizations and has identified a range of consequences that such orderings have for individuals and organizations. Finally, the surveillance and control perspective stresses that rankings are tools utilized by political and economic elites to “discipline” organizations to conform to dominant sociopolitical discourses. Table 1, in the online supplement, summarizes the key contributions of the three perspectives.
Among the three perspectives, rankings as information intermediation and comparative orderings have received the most empirical attention across disciplines. Within management, scholars have focused on five areas: (1) theorizing information asymmetry and decision making in markets that stimulate new rankings; (2) the effectiveness of rankings as measures of organizational phenomena of interest, such as unobservable quality; (3) the resource flow implications for organizations of rankings’ comparative social evaluations; (4) the negative effects on organizations and fields of stable status orderings; and (5) the validity of rankings in the wake of incontestable normalization. In terms of methods, while management research on rankings continues to develop across the three perspectives via theoretical and qualitative analyses, quantitative studies have largely been limited to examining the quality of rankings as informational products and their effects as comparative ordering mechanisms.
Taken together, these perspectives emphasize that rankings perform a technical function (provision of information), a social function (conferring reputation and/or status and patterning resource exchanges), and a political function (control and power redistribution). Furthermore, whereas research has suggested that rankings compel organizations to become more strategic by providing them with institutional feedback, it has also revealed and emphasized extensively the negative consequences that arise from poor quality of measurement and information provision, means-end decoupling, and the psychological consequences of simple quantitative comparisons. Although rankings are seen as contributing to the emergence, legitimacy, and transformation of fields, research has emphasized the adverse implications of marketization, stratification, and homogeneity. Furthermore, while critiquing rankings for their paradoxical displacement of trust and perpetuation of inequality, the literature has highlighted the futility of resistance to rankings.
A Missing Perspective: Rankings Entrepreneurship
In addition to identifying important gaps in research on organizational rankings conducted from the three dominant perspectives, our review clarified that an important perspective is largely missing from the current research agenda on rankings. In contrast to the hundreds of studies that focus on the consequences of rankings, there is a dearth of published research on the production of rankings (cf. Chandler & Rindova, 2008). We refer to this missing perspective as rankings entrepreneurship to highlight the need to focus on the actors that create rankings and the processes that they use to do so (Mehrpouya & Samiolo, 2016) as well as the audiences they serve. Our review suggests that establishing this research stream would significantly enrich rankings research because our understanding of rankings will remain incomplete without a more systematic analysis of the agency involved in their production. A focus on rankings entrepreneurs may also lead scholars to attend to entrepreneurs’ motives for financial gain, legitimacy, and power that could constitute the basis for launching new rankings.
Several qualitative studies provide some early evidence about the practices of rankings entrepreneurs (e.g., Déjean et al., 2004; Jeacle & Carter, 2011; Mehrpouya & Samiolo, 2016; Orlikowski & Scott, 2014; N. Pollock & D’Adderio, 2012; Sauder, 2008; Sauder & Fine, 2008). They demonstrate that the capacity of rankings to exert influence in an organizational field is shaped by the interplay of a range of social practices in the field and that the power of rankings entrepreneurs is constituted and constrained by the actions of the ranked organizations and their stakeholders. While these studies provide valuable insights into some aspects of the inner workings of the production of rankings, for this perspective to develop further, we need systematic analysis of the motives and agendas of rankings entrepreneurs and the resources, capabilities, and the processes that they mobilize to produce, legitimate, and disseminate rankings.
The importance of these issues is vividly illustrated in the experience and testimony of the entrepreneur who created one of the most successful organizational rankings to date—John Byrne, creator of BusinessWeek’s business school rankings. The story he tells about his initial motivations for launching the rankings fits with the information intermediation perspective: The reformer in me said a couple of things, “Business schools believe in markets, so I want to create a real market, where there’s information that people can act on.” . . . At the time, there was no Internet in 1988, and there was very little information, other than what you could get in a brochure from a single school. There wasn’t a lot of comparative data among and between the different programs that was readily available to the public. . . . We were gathering data that had never been gathered before and putting it into the marketplace to make the market really work. (John Byrne, quoted in AACSB, 2016)
Less fitting with the information intermediation perspective, however, is his account of the resources that he accessed and deployed in responding to the perceived market need. Discussing the evolution of the BusinessWeek rankings, he recounts, I’ve got to tell you that I literally printed out the surveys on my printer. I folded them on paper, I put them in envelopes, licked the stamps, and sent them out to thousands of students and the schools themselves. Then, when they came back, I literally had to input all that data. There were 35 questions in the original graduate survey. They all had to be inputted into a data sheet, into a computer. I did all of that. It was a labor of love. I had to convince my employer at the time, BusinessWeek, that this was a worthy project and something that I thought was really important. Why did they think it was important? I’ll tell you why because in every journalist, there’s a little bit of a reformer. (John Byrne, quoted in AACSB, 2016)
Even more shocking is his reflection on the outcome: I’m the first to admit I created a monster, and the monster is uncontrollable. The monster roams the earth doing all kinds of damage to many schools. . . . It’s a monster because everyone has decided to rank schools. There is no perfect way to really rank business schools or any schools. What you have is a lot of imperfect and flawed methodologies. Some of them are frankly journalistically mindless, where people are measuring things that have nothing to do with quality and may even have to do with political correctness. . . . Yeah, I think I created . . . a monster because most of the methodologies are flawed. They’re imperfect. They’re even intellectually dishonest. (John Byrne, quoted in AACSB, 2016)
As the example illustrates, the enterprise of ranking creation is ridden with complexity and is poorly understood. Given the consequential influence of rankings over organizational actions and audience decisions, developing an understanding of the entrepreneurs who produce rankings and how they produce and establish them would help address important missing pieces of the puzzle. First, what motivates different types of actors to produce rankings? Second, how do entrepreneurs produce their rankings—what are the practices, resources, and activities involved in various aspects of the process, ranging from identifying stakeholder audiences to represent and target to collecting, processing, and distributing data? In what follows, we outline some potentially generative approaches to addressing these questions.
Motives of Rankings Entrepreneurs
While rankings entrepreneurs routinely use rhetoric that reveals their motives, limited research exists to discern between the stated and actual motives behind rankings. It is quite possible, for example, for an entrepreneur to set out to achieve a goal that is different from the stated goal. Here, research on legitimacy and decoupling provides theory that can be tested in a rankings context. The literature on rankings has noted a gap between stakeholders’ needs, aspirations, and interests that rankings actually address and those they purport to address. Shore (2008: 292), for example, observes that the extension of an audit culture, including rankings, into hospital and university settings is “not simply a self-evident response to problems of governance and accountability; rather, it is about the politics of regulation and managerial control.”
The three perspectives that we reviewed imply a view of rankings organizations as actors who identify and exploit opportunities to address demands for information, produce comparative orderings, and impose surveillance and control. In the extended example, John Byrne states his motives in terms that are consistent with the information intermediation perspective. In contrast, the comparative ordering perspective would suggest that rankings entrepreneurs may respond to bottom-up demands to delineate an elite group of organizations from others in a field. Efforts in the 1970s by Columbia University researchers to rank business schools based on a survey of deans, by Georgia State University professors to list the leading American schools based on academic publications, and by Columbia University students to rank schools based on the employability of graduates all indicate such stratification-focused work (Daniel, 1998). Furthermore, the surveillance and control perspective is likely to point to rankings entrepreneurs who enact political ideologies to create changes in organizational fields consistent with their beliefs, as implied by the founders of “Beyond Grey Pinstripes,” an alternative ranking of global b-schools: Business school rankings, with their narrow emphasis on incoming achievement scores and outgoing starting salaries, only tell a small part of the story. They certainly do not tell us if graduates are being trained to manage the complex, increasingly interdependent social and environmental challenges present in a global economy. . . . Beyond Grey Pinstripes 2001 fills the gap left by rankings. . . . The report spotlights schools and faculty at the forefront of incorporating social and environmental stewardship issues into the fabric of their MBA programs. (World Resources Institute, 2001: 3)
As this anecdote suggests, the different perspectives that we identify point to different organizing logics, by which entrepreneurs themselves conceive of the rankings enterprise. However, the portrayal of motives behind the production of rankings based on these three perspectives alone is clearly incomplete. Again, John Byrne’s recollections provide a glimpse into the broader range of motives that can drive entrepreneurs to act, such as the creation of revenue streams: Back in 1988, when I created the . . . rankings for BusinessWeek, . . . I was concerned about who would read BusinessWeek in the future. I felt we had to do something on a regular basis, that was big and substantial, to attract the next generation of readers. (John Byrne, quoted in Boyd, 2010)
As this example illustrates, rankings entrepreneurs may need to walk a fine line between their self-interested motives and the broader stakeholder interests that they purport to represent. In theorizing and empirically investigating the types of actors that emerge as rankings entrepreneurs, we believe that the literatures on stakeholders and publics can be instructive. As Chandler and Rindova (2008) point out, the “situational theory of publics” proposed by Grunig (2005) recognizes that some stakeholders are actively involved in pursuing their interests vis-à-vis organizations, whereas others, while generally sympathetic to an issue, are less likely to take action. According to this theory, active stakeholders are (a) more likely to identify a problematic situation arising from organizational actions and perceive the need to address the problem and (b) less likely to be stalled by constraints. Furthermore, Rowley and Moldoveanu (2003: 211) distinguish between “interest-driven stakeholder groups,” whose collective actions are motivated by common discontent and pragmatic urgency, and “identity-driven stakeholder groups,” who value the common identity conferred by group membership. Interest-driven stakeholders are motivated by “end utility” rather than the “symbolic utility,” whereas the actions of identity-driven actors are “not instrumental but expressive” (italics in original). Understanding these distinctions can facilitate the analysis of which actors in a field become entrepreneurs, what their motives are, and how they engage with rankings production.
Specifically, we propose that these two types of stakeholder groups are likely to perform different roles in the rankings production process. First, it is interest-driven stakeholders who are more likely to recognize a problem in a field—for example, information asymmetry between stakeholders and organizations—and address it by taking actions that are consequential. They are therefore more likely to perceive rankings as an opportunity for progress, even if incremental, and to devise compromise, consensus-seeking solutions such as engaging with target actors and developing processes for accessing and gathering data. In contrast, identity-driven stakeholders are more likely to create rankings that reflect an ideological framework. Such a framework may enable them to gain legitimacy with relevant stakeholder audiences and to promote the prominence of their rankings in the absence of other resources for doing so. While these two types of stakeholder groups have different approaches to problem solving, both contribute to the processes of rankings production in important ways. It is possible that identity-driven stakeholders provide the ideological basis for rankings that increase their relevance and appeal to stakeholder audiences, whereas interest-driven stakeholders identify practices and foster relationships with the targets of rankings to enable implementation. Rankings therefore may emerge from the complementary interaction among these different stakeholder groups; the rankings process, however, may also become a focal point for developing such interactions, as they may otherwise be impeded by the differences between the two groups highlighted.
Beyond differences in stakeholder perspectives, we currently know little about the type of actors who are likely to become rankings entrepreneurs. As noted throughout, a variety of organizations, including the media, watchdog groups, and governmental organizations, have published rankings. Additionally, focal organizations in a field may initiate a ranking, such as the business school at the University of Texas at Dallas creating a ranking of business schools based on research productivity. An understanding of the types of actors that produce rankings can contribute not only to rankings research but also to broader research in institutional theory as it contributes to understanding agency and change in an organizational field.
Practices and Resources of Rankings Entrepreneurs
The production of rankings is a complex enterprise that involves a technical modality of representation, a social modality of evaluation of relative merit, and a political modality of control. As Norman, Roux, and Bélanger (2009) detail, entrepreneurs must resolve a number of epistemic and normative issues to create a ranking, including determining the underlying measurement dimensions, criteria for inclusion, approach to data, and modes of publication. Yet, we know very little about the rationale behind the selection of dimensions of performance, the procedures for data collection and aggregation, and the logic behind the presentation of rankings (e.g., Fortune’s “Most Admired All-Star” list includes 50 firms, while Forbes’s “The World’s Most Innovative Companies” list comprises 100). Whereas some research shows that management fashions influence the social construction of rankings (Bermiss et al., 2014), a comprehensive investigation of how rankings entrepreneurs select and combine criteria, including who is involved in these decisions, is sorely needed. Attempts by the creators of rankings to describe their guiding rationales provide useful inputs to contextualize the information made available (e.g., Chan et al., 2016), but these complex production decisions would benefit from more systematic research.
Beyond the logistics of production, scholars can investigate the institutional work that rankings entrepreneurs perform. Research on institutional work focuses on the agency involved in the processes by which institutions are created, maintained, and changed (Lawrence, Suddaby, & Leca, 2009). Commensuration is a key process in ranking production that can benefit from such analysis, as this technical and social process involves considerable effort, expertise, and capital (Espeland, 2002). It comprises three core dimensions that vary with context—a value dimension that defines the common metric, a technical dimension that establishes the precision and efficacy of the measurement system, and a cognitive dimension that identifies the categories that align with new metrics (Levin & Espeland, 2002). The practices and processes to integrate disparate values, implement precise measurement, and create new categories to make sense of the world could be thought of as commensuration work. Questions arise: Who performs this work? Are different people as well as different institutional actors involved in the different dimensions? Is additional work required to sell the commensuration (and subsequent rankings) to stakeholder audiences? How does the work involved in creating rankings differ from that involved in maintaining them? Tackling such questions illustrates the tensions in the production of rankings that results from the dual roles of “faithful representation and intervention” with which rankings entrepreneurs are tasked (Mehrpouya & Samiolo, 2016: 24).
More broadly, the literature has been slow to address how rankings entrepreneurs gain credibility and power in times when the number of rankings assessing a single phenomenon (e.g., CSR) have proliferated (Pope & Wæraas, 2016). Similarly, scholars have yet to consider the extent to which rankings entrepreneurs themselves are subject to reputational risk (Power et al., 2009). Once-dominant rankings can sometimes lose their luster (Raptis, 2012), which should encourage further research on the legitimacy challenges faced by “legitimating agencies” such as ranking organizations (e.g., Durand & McGuire, 2005). Our review suggests that the literature would benefit from studies that elaborate the role of stakeholder interests in shaping why some rankings gain traction, such as U.S. News & World Report in education, while others do not, for example, Aspen Institute’s “Beyond Grey Pinstripes” (Rasche & Gilbert, 2015; Sauder & Fine, 2008). Similarly, it would be useful to know why new entrants, such as the “Academic Ranking of World Universities” (Jeremic et al., 2011), become legitimate status arbiters in the presence of dominant incumbents, such as U.S. News & World Report (Marginson, 2009).
Related questions include the following: What resources allow entrepreneurs to produce and promote rankings? What are the constitutive elements necessary for production to occur? What business models are commonly employed, with what implications for value creation and capture? How do entrepreneurs gain legitimacy for new rankings? Does the extent to which the comparative orders of rankings correspond to preexisting hierarchies influence the legitimacy of rankings? How can rankings entrepreneurs overhaul the existing social order in a field? Do stakeholders perceive differences among competing rankings? Do rankings flourish more when introduced by actors that are central or peripheral to the field? In other words, when are rankings perceived to address stakeholders’ interests, and when are they deemed “wholly irrelevant” (Sampson, 2015: 82)?
These questions highlight how little we know about how rankings entrepreneurs work with the audiences that they seek to represent and the approaches that are most impactful. There is also a lack of analysis of how rankings entrepreneurs interact with one another and how this might affect the foci of their rankings. Rao’s (1998) study of consumer watchdog organizations suggests the value of cultural analysis to examine how entrepreneurs combine unconnected beliefs, norms, and values to produce rankings. Research distinguishing between the cultural and political processes used to produce, promote, and naturalize rankings would also be particularly valuable (Jarzabkowski & Kaplan, 2015; Kellogg, 2011; Rao, 1998).
Additionally, it is important for management research to explore the accountability implications of rankings in an age of social media as rankings entrepreneurs increasingly leverage online technologies (Jeacle & Carter, 2011). As emerging research has shown, rankings that use social media (e.g., TripAdvisor) potentially reconfigure organizational accountability by enabling the collection and aggregation of ratings from millions of distributed actors (Scott & Orlikowski, 2012), while the “criteria for global orderings” that underlie these rankings are “often not at all transparent, possibly hidden and shifting” (Staeheli, 2012: 242). By collecting information from and providing information to consumers, an Internet-mediated ranking system “shifts the originating source of assurance from the certified professional expert to consumers,” “creates the conditions by which everyone is potentially an auditor on behalf of everyone else,” and “amplifies the reputational risk” to ranked organizations (Power, 2011: 325).
Research suggests that consumer-generated rankings may not agree with the evaluations generated by expert intermediaries and so pose an identity threat to ranking organizations (T. Wang et al., 2016). Thus, user-generated Internet-based rankings merit further research as new forms of social measures that are likely to have unique mechanisms and consequences. Such an examination may require conceptual tools, such as a practice-based lens that draws attention to rankings as “neither given nor socially constructed but as constituted in practice” and focuses “specifically on the material-discursive practices through which valuations are produced” (Orlikowski & Scott, 2014: 872).
In sum, our review reveals significant opportunities for research to theorize and test the production of rankings as complex technical, cultural, and political phenomena. A broad swath of theoretical perspectives appears relevant, such as stakeholder theory, institutional theory, cultural analysis, a practice-based view, and information aggregation. Research on rankings entrepreneurship, therefore, is an important and promising area for future rankings research.
Implications for Management Practice
The extant literature on rankings suggests different implications for managers, intended audiences, producers, and the institutional actors who could benefit from using these complex evaluations. For managers, rankings can variously serve as feedback, distraction, and external sources of conformity pressures. Given the documented power of rankings (e.g., Lynch, 2015; Power et al., 2009; Wedlin, 2006) and the futility of overt resistance to them, at least in some contexts (e.g., Sauder & Espeland, 2009; Shore & Wright, 1999), the literature suggests caution to managers in their engagement with rankings. As rankings are incorporated into the compensation packages of senior executives, the potential for distortion of the organization’s values to “play to the rankings” has grave implications—an effect already identified in universities (Corley & Gioia, 2000; Espeland & Sauder, 2007; Kraatz et al., 2010). However, managers can use rankings to achieve strategic goals (O’Connell & Saunders, 2012), generate change (Gioia & Corley, 2002; Locke, 2014; Martins, 2005), and highlight favorable performance to influence key constituents (Fombrun, 2007; Sauder & Espeland, 2006).
For the intended audiences of rankings, our review suggests caution in relying on the information that they provide to guide decisions. While consumers may be attracted to the simplification of complex consumption choices, the comparative orderings and oversight and control perspectives suggest that it is essential to keep clear consumption goals in mind to avoid being led astray by the expediency rankings afford. Indeed, commensuration processes may create illusions of comparability that reinforce existing status hierarchies and distort final decisions due to the poor quality of information that can be presented (Espeland & Sauder, 2007; Espeland & Stevens, 1998; Levin & Espeland, 2002).
It is abundantly clear from the literature that the producers of rankings wield tremendous power, which makes it imperative that ethical concerns are considered as they build and refine their ranking methodologies. To this end, researchers have recommended using sound measurement and statistical techniques (e.g., Palocsay & Wood, 2014), qualitative profiling of organizations using relevant indicators (e.g., Soh, 2012a), integrating information across rankings (e.g., Claassen, 2015), constructing rankings of rankings (e.g., Stolz, Hendel, & Horn, 2010), and promoting alternative systems, such as badges of quality (Özbilgin, 2009), ratings (e.g., Rapoport, 1999), and user-driven scoreboards (e.g., Tijssen, Yegros-Yegros, & Winnink, 2016). Additionally, producers of rankings could increase the transparency of their data collection and aggregation processes (e.g., Piro & Sivertsen, 2016; Van Vught & Westerheijden, 2010; Walsh, 2011). Whereas increased transparency may reduce their power, it could also increase the utility of rankings such that ranked organizations and intended audiences accord them greater credibility. The producers of related rankings could also benefit by collaborating with one another and their constituents to establish common principles and guidelines (Hägg & Wedlin, 2013).
As sources of external control of organizations, the literature suggests that rankings can serve the interests of other institutional actors. Depending on their agendas, these actors could select rankings that serve their objectives, using their own institutional positions to amplify the rankings and their influence over the relevant set of organizations (O’Connell & Saunders, 2012). Thus, rankings can provide an important source of leverage to institutional actors in advancing their agendas relative to a specific set of organizations.
Conclusion
We reviewed the management and cognate literatures to assess the current state of research on organizational rankings. We identified three distinct extant perspectives on rankings as information intermediation, comparative orderings, and means of surveillance and control, and we proposed a new perspective focused on rankings entrepreneurship. This new perspective opens up an exciting avenue for future research, including how the production of rankings represents stakeholder interests. Our observations resonate with the “puzzle of measurement” that “numbers are powerful and fragile, simple and qualified, trusted and distrusted” proposed by Power (2004: 779). It is evident from the literature that addressing this puzzle in the context of organizational rankings is a vibrant research area with great promise for future development.
Footnotes
Acknowledgements
This article was accepted under the editorship of Patrick M. Wright.
Supplemental material for this article is available with the manuscript on the JOM website.
Notes
References
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
