Taylorizing business school research: On the ‘one best way’ performative effects of journal ranking lists

Abstract

The article critically examines how work is shaped by performance measures. Its specific focus is upon the use of journal lists, rather than the detail of their construction, in conditioning the research activity of academics. It is argued that an effect of the ‘one size fits all’ logic of journal lists is to endorse and cultivate a research monoculture in which particular criteria, favoured by a given list, assume the status of a universal benchmark of performance (‘research quality’). The article demonstrates, with reference to the Association of Business Schools (ABS) ‘Journal Guide’, how use of a journal list can come to dominate and define the focus and trajectory of a field of research, with detrimental consequences for the development of scholarship.

Keywords

journal lists knowledge development performance measurement performativity research evaluation Taylorization work culture

Introduction

The creation of ‘journal quality lists’ in the field of business and management in the UK has coincided with the growing importance and formalization of national research evaluation exercises (Geary et al., 2004; Keenoy, 2005; see also Gendron, 2008).¹ The compilers and advocates of these lists say that their intention is to provide an objective measure of the comparative esteem of journals using a standardized quality metric, thereby overcoming information asymmetries associated with the use of ‘insider’ knowledge (e.g. Rowlinson et al., 2011).² Their use, it is further suggested, can correct the biases ascribed to evaluators of research quality (see, for example, Taylor, 2011). However, when the lists are used as a standard to calculate the equivalent of an exchange value of outputs (e.g. journal articles) and of authors on the academic hiring, promotion and transfer markets, such justifications largely disregard the extent to which lists contribute to, and have further potential to promote, a commodification of academic labour and a narrowing of scholarship (Bryson, 2004; Harley and Lee, 1997; Scandinavian Journal of Management, 2012; Van Fleet et al, 2011; Willmott, 1995).

The pressures upon business school academics are particularly intense where these schools have become amongst the largest of university departments, with corresponding implications for institutional funding and reputation. The significance and influence of journal lists increases as competition between institutions for resources, symbolic as well as material, intensifies. As journal quality lists (e.g. those created by the Financial Times and the Association of Business Schools) become influential for processes of recruitment, promotion and the selection of staff/outputs for submission to evaluation exercises, they come to shape the nature, structure and conditions of academic work (Espeland and Sauder, 2007; Sauder and Espeland, 2009). Such performative effects are, of course, greatest when they weaken or marginalize alternative criteria and processes of evaluation. Examining the use and effects of journal lists is therefore important not simply for better understanding, or refining, how such metrics are devised (see Truex et al., 2011 for a critical review) but also, and more significantly, for appreciating and questioning their constitutive role in defining and policing the focus and direction of research activity.

Regardless of the particular method or algorithm used to compute journal lists (see Morris et al., 2009 for a typology), their design shoehorns horizontal diversity of research and scholarship into a single, seemingly authoritative vertical order. By valorising the ‘research agenda’ that has become institutionalized in the topics, methods and perspectives favoured by ‘A’ category journals, the use of journal lists to assess the quality of research sends out a strong ‘market signal’: it privileges the agenda pursued in those journals; and, conversely, it devalues research published elsewhere, irrespective of its content and contribution. When an article’s place of publication, as indicated by its ranking in a journal list, becomes more significant or valued than its scholarly content, faculty find themselves increasingly in receipt of the following kind of ‘advice’ from Deans, research directors and senior colleagues:

If you wish to be counted as ‘research active’ and so be submitted to the next evaluation exercise or to improve your promotion prospects, your work should be designed, shaped and honed to emulate the genre of research published in journals most highly ranked in the prescribed journal list used to assess your eligibility for submission to the exercise. Failure to demonstrate this competence risks staying on probation/ not being counted as research-active/ not being considered for promotion/ being moved to a teaching only contract. Whatever its intended purpose, the journal list has become a potent instrument of managerial decision-making whose use, we will argue, has the performative effect of homogenizing, in addition to commodifying and individualizing, research activity.

This concern complements a number of other objections levelled against journal lists, which range from issues about the technicalities of their construction which include consideration of processes of rating their quality (Hussein, 2010), through criticisms of their neglect or devaluation of other kinds of publication (e.g. monographs), to their exclusion of ‘research that matters’ and their obsessive or ‘fetishised’ use (Őzbilgin, 2009: 113; see also Clarke et al., 2012; Harzing and Metz, 2012; Keenoy, 2005; Knights et al., 2011; Willmott, 2011; Worthington and Hodgson, 2005). Our focus here, in contrast, is upon the performative effects of a “one size fits all” logic of research evaluation. (see also Nkomo, 2009). To illustrate these effects, our analysis examines in some detail the development, justification and application of the Association of Business Schools’ (ABS) Academic Journal Quality Guide. Our example is taken from the UK context where the use of journal lists is probably most widely and deeply embedded. But, of course, their use has been widespread, and seems set to become more influential. Downloads of the ‘Journals Guide’ from the ABS website are reported to have been, in one year (2010), ‘90,000…from nearly 100 countries’ (Rowlinson et al., 2011: 443).

We begin by considering the squeeze on heterogeneity by the ‘one size fits all’ philosophy enshrined in the compilation of journal lists – a restrictive process that is increasingly reinforced by reliance upon citation counts and impact factors. To underscore the homogenizing influence of journal lists, we draw a parallel between the ‘one best way’ design of industrial production advocated by Frederick Taylor (see Kanigel, 1995) and the ‘one size fits all’ design philosophy enshrined in journal lists. Detailed consideration is given to the establishment and use of the ABS ‘Quality Guide’ before we assess claims that its use brings cultural and economic benefits.

Measuring scholarship

The UK has been in the vanguard of national assessments of research quality. The first assessment was undertaken in 1986 and it has been repeated every 5–7 years with the most recent exercise completed in 2008 (see Gillies, 2008). The development of business school journal lists has been associated with increasingly selectivity in these exercises and the rapid expansion of business schools within universities. Before the exercises and the appearance of journal lists, academics in UK universities were expected to publish where they found a receptive audience; and if their work appeared in a widely read and respected journal so much the better. An initial effect of introducing a formal system of research quality evaluation, we submit, was to make the playing field of business and management research more level (see Morris, 2011:39–40) as it was self-defeating to exclude heterodox or questioning scholarship (e.g. critical accounting or ‘soft operational research/systems’) on ideological grounds. During the 1980s and 1990s, the national assessment exercises tended to raise the profile and expand the space for innovative and heterogeneous scholarship, and so nurtured the development of comparatively diverse research cultures across UK business schools.

Without succumbing to misty-eyed nostalgia, management and business scholarship during the late 1980s and 1990s pushed the boundaries of disciplines, and substantially opened up the intellectual landscape (e.g. Hopwood and Miller, Jackson and Keys, 1989; 1994). Subsequently, these tendencies have slowed in many areas (see Piercy, 2000 for a trenchant critique), aided inter alia in recent years by the use of journal lists to evaluate the quality of research (and researchers) (see Hopwood, 2008). This slowing of innovation has coincided with increasing pressures to ‘game the system’ (Lee, 2011; Macdonald and Kam, 2007, 2008) as national research exercises have become both more critical and more divisive in their distributional effects.³ Central to such game playing are efforts to define and legitimize preferentially rewarded research space (see Bedeian, 2004; Marsden, 1993). So, for example, to the extent that a list elevates an established, currently North American-dominated, set of journals as the benchmark of ‘top-notch’ research, a particular, neo-positivist research agenda is deemed to yield the highest ‘quality’ publications (see, for example, Lee and Elsner, 2008).

One size fits all: The citation and impact metric

A distinguishing feature of journal rankings is the application of a ‘one size fits all’ logic, which necessarily assumes a single, shared basis for specifying and comparing the quality of different journals.⁴ Each ranking metric comprises elements of (i) measures of impact derived from numbers of citations in selected journals and (ii) some moderation of these measures by forms of peer evaluation which, as Peters et al. (in press) note, can be a mixed blessing. That is because there is evidence that such moderation, which in the case of the ABS list is acknowledged but not explicated, will tend to reflect the ‘personal concerns’ (Peters et al. in press) of raters (e.g. their prior publication in journals that they rate highly). The resulting rank order is often then hierarchically divided into ‘top’ (A/world elite), ‘middle’ (B/international) and ‘bottom’ (C/national) categories. For illustrative purposes, we take up the ABS ‘Quality Guide’ as our example, but a comparable analysis and critique is, in principle, applicable to other journal quality lists.

Citation indices, it is widely assumed, provide the most reliable indicator of journal quality as they are blind to any specific values or criteria that may ‘bias’ processes of peer review. Yet, arguably, reliance upon citation counts and associated indices necessarily privileges particular kinds of journals, irrespective of their distinctive contents: namely, journals that are long established, which publish topics that are widely researched, use methodologies that are frequently deployed and/or engage familiar theoretical frameworks, etc. (see McWilliams et al., 2005; Mingers and Xu, 2010; Parker and Thomas, 2011). Journals with high citation indices tend to be well resourced and benefit from influential sponsors. For example, members of the (American) Academy of Management receive the equivalent of a season ticket, in the form of subscription to a selection of its journals. Like globally branded sports teams, these journals are best placed to attract star players – the editors and board members who command the greatest intellectual capital, and who act as a magnet for authors whose work is already well known, and which is therefore more likely to be cited. This is significant because journals with high citation scores tend to rely upon a few ‘big hitters’ (Baum, 2011). Their citation counts effectively enable others who publish in top-tier journals, but whose work is infrequently cited, to ‘free-ride on a small number of highly-cited articles, which are principal in determining journal Impact Factors’ (Baum, 2011: 465). Reliance upon citation indices in the construction of journal lists conceals the ‘big hitters’ effect, and it also marginalizes journals dedicated to less popular / emergent areas of research and methodologies.

Compilers of journal lists invariably rely upon journal impact factors (JIFs), based upon a two, three or even a five year period. Using this time frame also has significant but seldom acknowledged implications. Reliance upon them may perhaps be more justifiable in fast-moving areas such as the natural sciences, but their use is of dubious relevance for the social sciences, including the field of management and business studies. A study by Mingers (2008) of 600 papers published in 1990 in six well-known management science journals showed that citations for these papers did not reach their peak until six years after publication. Many papers were still being cited fifteen years after publication. And it was not possible to predict the eventual number of citations received by a paper from citations in the early years. ‘Shooting stars’ are highly cited immediately, perhaps because they catch a fashionable wave of work, and then fade into obscurity. But ‘sleeping beauties’ may be so innovative that their importance is only appreciated, and is correspondingly influential, much later on.

Subject to the reservations entered above, and within the important exception of articles that are cited ritualistically or repeatedly demonised as examples of unwelcome scholarship, there is a strong case for identifying articles of the highest quality as those that continue to be cited many years after their publication – that is, the ‘sleeping beauties’, not the ‘shooting stars’. A perverse performative effect of ascribing the highest value to ‘shooting stars’, as happens when JIFs are based upon five or fewer years, is to incentivise the production of comparatively safe papers that contribute to established or fashionable topics and issues, rather than encouraging innovative scholarship that, potentially, has a longer lasting relevance (see also Hopwood, 2008). In sum, in the field of management and business, there are major problems and limitations of relying upon citation counts to determine journal quality (see also Truex et al., 2011).⁵

Heterogeneity at risk

The image of sporting league tables, with their organization into divisions and top clubs, is frequently invoked to characterize and legitimize journal rankings. The aptness of the parallel is, however, limited and can be misleading. In competitive sports, team managers have broadly the same objective: in many ball sports, for example, the aim is to score goals without conceding. Despite intensifying ‘isomorphic pressures’ (Rowlinson et al., 2010: 167), which are compounded by the use of journal lists, the objectives of journal editors are, in contrast, comparatively diverse. That is to say, they have diverse ambitions that are not reducible to an emulation of journals that are in the premier division of the rankings. Across the extensive and variegated field of management and business studies, editors and contributors participate in diverse epistemic communities (Knorr-Cetina, 1999) where divergent views are harboured about what is ‘the best’ or the ‘most valuable’ research. When establishing a new journal, there is often an ambition to address emergent areas, develop new approaches and/or engage innovative perspectives. The ‘one size fits all’ logic of quality assessment does not, and cannot, acknowledge or value this heterogeneity. It can only devalue and/or ignore it.

At the same time, it would be naive to deny that editors are subject to reputational and commercial pressures, which are intensified by the competitive use of citation scores and JIFs, by publishers as well as prospective authors, as performance indicators. These pressures can induce editors to emulate the genre of scholarship published in top-tier journals. As Hopwood (2008) comments, editors of new or lowly-ranked journals may bow to pressures to ‘replicate the biases’ of top tier journals instead of seeking to carve out a specialism of their own. This herd mentality may then extend to ‘second tier’ journals where editors come under pressure from commercial publishers to ‘up’ their JIFs. Often this deferential strategem has the paradoxical consequence of a journal’s lesser status being confirmed by an ‘almost feudal form of behaviour with the lowly genuflecting to the more highly placed’ (Hopwood, 2008: 90; see also Grey, 2010). The likelihood of such an outcome is increased when citation scores influence adoption decisions by libraries, and when authors seek out highly cited/ranked journals in which to publish in an effort to raise their personal citation rates (Judge et al., 2007; Sharplin and Mabry, 1985). These scores can then be ‘cashed in’ by authors in the promotion and transfer markets, and in the selection process for submission of outputs to research evaluation exercises.

Marginalizing innovation

In this highly competitive milieu, it would be surprising if many journal editors were unable to resist all temptation, including material incentives, to game the system in order to increase their JIFs – for example, by favouring articles that are assessed to have ‘shooting star’ potential (see above), or by inviting (aka requiring) authors to consider including reference to articles that have been published in the journal. What happens, then, if a specific area of scholarship, or a favoured approach to a field of investigation, does not share the particular research agenda and associated set of values privileged in the most highly cited or ranked journals? Journals dedicated to peripheral areas and approaches are often invisible or marginal to citation indices. As a consequence, they tend to be poorly rated within, or excluded by, ranking lists. Well-documented examples include the areas of sustainability (see Wells, 2010) and innovation studies (see Rafols et al., 2011). But a similar fate befalls a wide range of work that addresses non-mainstream topics and embraces heterodox methodologies and perspectives in journals that lack the profile and institutional support enjoyed by those published by the (American) Academy of Management, for example. This disadvantage extends to much multidisciplinary, interdisciplinary and transdisciplinary work, which is not readily accommodated within disciplinary silos (see Meriläinen et al., 2008).

The marginalization of heterodox research is illustrated by the editorial policy of the (US-based) journal Operations Research, widely regarded as one of the top two journals in its field. Operational Research (OR) has mathematical roots but since the 1970s, especially in the UK, the adequacy of mathematical modelling of complex real-world problems has been questioned, resulting in the development of a new area of OR, known as ‘soft OR’ or ‘soft systems’ (Checkland, 1981). Yet, no papers on soft OR have ever been published by Operations Research, or in the other ‘top’ (also US-based) journal, Management Science. When challenged by a letter in OR/MS Today (Ackerman et al., 2009), signed by academics from around the world, the response from the Editor was that, as far as Operations Research was concerned, non-mathematical OR was not OR and therefore not publishable. Since Operations Research and Management Science are the only two journals in the ABS list to be ranked 4* for OR, soft OR academics are, as far as the ABS list is concerned, not able to produce work of the highest quality, at least in the name of OR. OR may provide a ‘limit’ case but an equivalent restrictive logic applies across other specialist fields of management (e.g. International Business), and this is despite repeated exhortations in ‘top-tier’ editorials to develop a more expansive and inclusive research agenda (see also McWilliams et al., 2009). At best, publishing research in highly ranked journals that does not fit their established mould is conditional upon shaping and revising submissions in a manner that tends to emulate a model of rigor ascribed to the physical sciences (Thomas and Wilson, 2011). Even when the data are qualitative – and when there is no explicit hypothesis testing of propositions; no reference to internal, external or construct validity; and no preoccupation with the operationalization, measurement and statistical analysis of variables – the ethos of positivism, in the numerous guises of neo-positivism, tends to hold sway (e.g. Gibbert et al., 2008; Scandura and Williams, 2000). To the extent that a journal list broadly reflects, endorses and reproduces the hegemony of this neo-positivist tradition of scholarship, its most potent (performative) effect is to devalue and marginalize, if not exclude, heterodox forms of scholarly contribution, and thereby induce a homogenization of research activity.

One best list: Taylorizing business school scholarship

For a majority of senior managers – from research directors, through deans to vice-chancellors – the ranking of their university’s departments, including its business school, in national evaluation exercises is the key indicator of its status in relation to other universities. The position of departments in the resulting league table also indicates a rise or decline of their standing since the previous exercise. In turn, the outcome of the evaluation is an indicator of the stewardship of senior managers in preparing for the exercise - with regard to staff recruitment and retention as well as to fostering a supportive research environment. The development and widespread adoption of the ABS ‘Journal Guide’ stems, we suggest, from the performance anxiety of university managers.

Performance anxiety

Determining whether the outputs of staff are of adequate quality to be submitted to research evaluation exercises and, if so, which publications to select, presents university managers with a major challenge, especially when the reputation of the university and the business school is research-based, and so is heavily dependent upon the outcome. When faced with such troublesome decisions, the availability of a journal list is a seductive decision-making aid as it purports to provide an objective benchmark for assessing the quality of research published by staff. Its appeal resides in seeming to provide an impersonal, objectively computed basis for making and justifying difficult and divisive decisions. This appeal is especially strong in contexts of diversity where there are multiple paradigmatic differences over what counts as quality – differences that can be as acute and delicate within business schools as they are between schools.

In the UK, in the run-up to the 2008 evaluation exercise, university managers were wrestling with consequential, and politically charged, decisions concerning the selection of staff and their outputs. Their decision-making was informed by the knowledge that outputs judged to be of the highest quality (i.e. as 4*, see Exhibit 1) – were very likely to attract a significantly greater (but unspecified at the submission stage) weighting for funding purposes than outputs judged to be 3* or 2*. There were therefore strong material inducements, in addition to symbolic benefits, to submit outputs with the best chance of being judged as 4* or at least 3* quality; and, conversely, there was a concern to minimize the number of staff with outputs that risked being evaluated as 2* or below.

Exhibit 1

Research Assessment Exercise (RAE) definitions of quality

4* Quality that is world-leading in terms of originality, significance and rigour.

3* Quality that is internationally excellent in terms of originality, significance and rigour but which nonetheless falls short of the highest standards of excellence.

2* Quality that is recognised internationally in terms of originality, significance and rigour.

1* Quality that is recognised nationally in terms of originality, significance and rigour.

Unclassified Quality that falls below the standard of nationally recognised work. Or work which does not meet the published definition of research for the purposes of this assessment.

Faced with this challenge, how might research directors and deans of business schools hope to minimize the risks and maximize the rewards of the exercise, thereby securing their personal, collegial and corporate reputations? Might the assessment process and associated procedures offer any pointers to a defensible stratagem? The public record for the previous (2001) national evaluation exercise stated that the Business and Management Panel read ‘15–30 per cent of outputs with some reading as much as 75%’ (Bessant et al., 2003: 53). The Panel was noticeably silent on the question of how it assessed outputs that it did not read. It was unlikely that the volume of outputs for the 2008 exercise would be smaller, or that the size of the Panel would be significantly larger (or its operation as a body would become highly unwieldy). So, an obvious question presented itself: how would Panel members find the time to base their assessment on reading the outputs? And, if they couldn’t, what would they do? Would they not have to rely upon some other, proxy measure of quality?

Some indications of how the 2008 Panel would operate were provided in its statement of Working Methods, published in January 2006 (see Exhibit 2). It this statement, use by Panel members of any journal lists to evaluate submitted outputs were explicitly excluded: ‘the assessment will be one of expert review based on professional judgement’. A lingering question remained, however, about how practically this undertaking could be fulfilled in the time available (a few months) by a Panel comprising less than twenty members. Closer consideration of the 2008 Panel’s statement of its Working Methods revealed a commitment ‘to collectively examine in detail at least 25% of the submitted outputs’ (emphasis added). So, how were the remaining outputs to be assessed?

Exhibit 2

RAE working methods

‘The assessment will be one of expert review based on professional judgement. Each sub-panel member will be expected to form a view on all submissions. Sub-panel members with relevant specialist knowledge will assess cited work from all submissions…They will focus detailed examination on work which has not undergone peer review, or is published in new and less familiar media, or which is deemed to be potentially of the very highest standards. In conjunction with specialist advisers and members of other sub-panels to which work is cross-referred, the sub-panel expects to collectively examine in detail at least 25% of the submitted outputs.’

Extracted from paras 47 and 49 of http://www.rae.ac.uk/pubs/2006/01/docs/i36.pdf

It could be inferred from its Working Methods statement that the Panel would rely upon its ‘professional judgment’ to assess the remaining outputs; and that it would do this without reading the outputs ‘in detail’. This inference pointed in the direction of Panel members’ (expedient) reliance upon some other indicator(s) of the quality of not-read-in-detail outputs - outputs which, it could be anticipated, would contain a very high percentage of journal articles. The most obvious proxy indicator of the quality of such outputs is the publication in which they appear. Whatever the explicit and repeated pronouncements of the Panel, its huge workload suggested that its members might well resort to a journal list in order to get the job done.⁶

As the 2008 exercise approached, members of research committees, research directors and deans of business schools chewed over the vexed question of who and what might be submitted. A possible remedy for their performance anxieties (see above) presented itself in the form of the indicator of publication quality provided by a journal rankings list.⁷ There was, however, an irksome difficulty. In 2006, when the Working Methods of the 2008 evaluation Panel were published, numerous journal lists were in circulation, including those constructed by several UK business schools, such as Aston, Cranfield, Durham, Imperial, Kent and Warwick (see also http://harzing.com/). Supposing that Panel members would indeed be obliged to resort to a list when assessing up to 75 per cent of outputs that would not be read ‘in detail’, there was no way to know, or deduce, which list or combination of lists they might use, individually or collectively, to assess those outputs. Faced with this uncertainty, a shared strategic objective of deans, especially those heading up research-based business schools, was, we conjecture, collectively to overcome the vertigo of uncertainty by establishing, endorsing and promoting a preferred journal list.

The creation of the ABS list was sponsored by the deans of UK business schools through the ABS which hosts their regular meetings and activities. The publication of Version 1 of the ABS list in January 2007⁸ coincided with, and drew legitimacy from, the endorsement of journal ranking lists in other fields by influential bodies such as the Science and Technology Committee of the House of Commons.⁹ With the appearance of a list sponsored and endorsed by deans, the relevance of other ‘local’ lists diminished. As the creators of the ABS list claim, and not without justification, it ‘has been widely adopted as a policy tool in UK business schools and indeed in business schools in many parts of the world’ (Kelly et al., 2009: 2). They also note that, ‘In preparing for the RAE 2008, many university and business school managers…made use of the ABS guide in planning their submissions’ (Kelly et al.: 2). Given the widespread application of the ABS list, and its continuing use in preparations for the 2014 evaluation exercise, it is relevant to consider further how its construction and justification has contributed to exerting an influence over the range and direction of business and management research.

Managing by numbers: The ABS ‘Journal Quality Guide’

In the run-up up to the 2008 evaluation exercise, the ABS ‘Journal Quality Guide’ became adopted as the ‘de facto standard’ across UK business schools (Mingers et al., 2012: 3). Its relevance for this purpose was unsubtly signalled by Version 1 of the ABS list in which journals were grouped using a five point scale that directly mimicked the scale used by the Panel. As Rowlinson et al. (2010: 157) acknowledge, the scale was adopted, ‘in anticipation of the rating system to be used in RAE 2008’, a view that finds confirmation in the Introduction to Version 1 of the ‘Guide’ (see Exhibit 3).

Exhibit 3

Quality rating and meaning from ABS Journal List

4 * A top journal in its field. Publishes the most original and best executed research papers. Journals typically have high submission and low acceptance rates. Papers are heavily refereed and the journals have high citation impact factors in their sub-field.

3 * A highly regarded journal in its field. Publishes original and well-executed research papers. These journals typically have good submission rates and are very selective in what they publish. Papers are heavily refereed and the journals have fair to good citation impact factors.

2 * A well regarded journal in its field. Publishes original research of an acceptable standard. Papers are fully refereed and the journals have modest citation impact factors or do not carry one at all.

1 * A recognised journal in its field. Publishes research of a modest standard or has yet to establish a reputation by virtue of being launched recently. Few journals in this category have an impact factor.

0 * A journal not recognised as an authentic research publication. Journals aimed at practitioner audiences which attract academic contributors and which do not generally rely on peer review. (Morris et al., 2009: 1448)

In place of any principled articulation or defence of the ABS list – for example, by indicating how it might contribute to promoting more innovative research and scholarship – its architects simply anticipate and openly commend its managerial use. The preparation of the list is warranted on the grounds that, ‘people do not always read all that they are expected to read’, from which it follows, apparently, that ‘it is surely a good thing if a systematic method of determining journal quality, like the ABS guide, is used’; the alternative being ‘the unsystematic and imprecise methods that might prevail in the absence of ranking journal titles’ (Morris et al., 2009: 1449). In other words, the list is commended as a handy, expedient tool for those charged with making onerous decisions about their colleagues’ careers, yet who are disinclined to prioritize time for reading and assessing the work itself, or seeking advice from subject specialists. To the extent that this commendation is followed, the performative effect of journal lists is to frame, guide and justify decisions on what counts as ‘quality research’ and where resources are allocated.

Performative effects of the ABS list as a policy tool

The conflation of the scales used in the ABS list with the 2008 assessment exercise, and the calculation that the Panel would make expedient use of a journal list, make it probable, as the architects of the ABS list claim, that the ABS list has been ‘widely adopted as a policy tool in UK business schools’, and used ‘in planning their submissions to evaluation exercises’ (Kelly et al., 2009: 2). Moreover, the findings of a modelling exercise, the architects of the ABS list argue, demonstrate ‘a high degree of congruity between the judgments reached by the 2008 Panel and the journal quality rankings of the ABS guide’ (Kelly et al. 2009: 3). By noting this ‘congruity’, it seems to be implied that savvy university managers made use of the ABS list when selecting staff and outputs for the 2008 exercise and that, despite the evaluation Panel’s repeated protestations to the contrary, its members were heavy closet users of the list. In order to assess such claims it is important to take a close look at the results of the 2008 evaluation exercise.

Mingers et al. (2009) examined the overall results for each business school submission to the Panel. Their analysis necessarily relies upon available, aggregated information on the outcomes of the 2008 exercise profiled as the percentage of outputs awarded a particular grade from 4 to 1 and ungraded (see Exhibit 1). The profile is compared to the actual journal-based outputs submitted by each business school or management department in order to determine whether the results could be recreated through an imputed ranking of the journals. The main findings of interest to the present analysis are:

Finding. A much greater concentration of outputs as journal articles and correspondingly fewer books or reports than in previous years – in 1996, 69 per cent were journals, whereas in 2008 the figure was 92 per cent.

Comment. This increase cannot be attributed to the ABS list per se but it does perhaps indicate increasing risk-avoidance behaviour to which the significance assigned to journals by the ABS list contributes, leading to the conclusion that ‘at least you are safe with a refereed-journal paper, especially if it’s in ABS’ as there was no equivalent list for any other type of output. The implication is that academics should direct their research activity primarily to what is publishable in journals rather than through other media (e.g. monographs).

Finding. The number of journal titles submitted increased from 1275 in 1996 to 1639 in 2007. Of the journals submitted to the 2008 exercise, only 50 per cent appeared in the ABS list.

Comment. This shows the vast range of research carried out within business and management that does not have the ‘ABS stamp’ of recognition. This remarkable diversity remains to be adequately acknowledged by the ABS list. We are concerned that this range will be narrowed if, in future, the (ABS) list becomes even more hegemonic in defining where management research is published.

Finding. Evidence that the ABS list was used in making submissions – for instance comparing the ABS journals that were submitted with those that were not, 45 per cent of those not submitted were ABS 1*, while only 4 per cent were ABS 4*. An article appearing in a journal ranked as 1* or 2* in the ABS list was generally not submitted even though it might be assessed to make a substantial and well received contribution to its field.

Comment. This tendency militates against newly developed journals and innovative work. Mingers et al. (2012) conclude that ‘there is evidence of extensive selectivity in submissions’ guided by the pecking order of journals on the ABS list. They suggest that the presence and position of a journal on the ABS list was a ‘possible bias’ with regard to the judgments of Panel members. This does not imply that Panel members directly consulted a list but, rather, that a ‘list mentality’ may have been acquired by some Panel members, predisposing them to value outputs published in journals ranked comparatively highly on the ABS list.

Finding. Evidence of an association between the proportion of a department’s submission appearing in ABS journals and the evaluation it achieved. In other words, in general, departments with a greater concentration of ABS journals did better.

Comment. There is not necessarily a direct causal effect here – members of a department in which well-regarded research is undertaken will tend to publish in journals which are included and highly ranked in the ABS list. However, some submissions were virtually 100 per cent ABS journals, pointing to a high degree of managerial selectivity based upon the use of the ABS list.

Finding. Mingers et al. (2012) analysis indicates

differences [between the ABS ranking and RAE grades] for particular journals with some being two or even three grades apart’ (Mingers et al. 2012: 25). There is evidence of dispersion around the grade of a journal, i.e. papers from the same journal being awarded different grades, as the RAE Panel expected a priori, and claimed a posteriori (e.g. for the journal Industrial Relations – ‘4* 53.6; 3* 0.0; 2* 8.8; 1* 36.7 O*0.0)*. (Mingers et al. 2012: 20–22)

Comment. For research managers seeking to ‘game the system’, this indicates a downside of reliance on a list when selecting outputs for submission, rather than making peer-reviewed judgments about their merit.¹⁰

The actual grades awarded to individual outputs are not published. It is therefore necessary to underscore that it is impossible to know definitively whether the 2008 Panel members assigned a high grade to some articles in journals that were not ranked highly on the available ABS list; or, indeed, if a low grade was given to articles published in more highly ranked journals. Taylor (2011) has used regression analysis to assess the extent to which similar aggregate results could have been obtained by simply using the ABS list together with two other variables measuring the size of the submission, and membership of university groupings such as the Russell Group (a grouping of top UK universities). Taylor found that he could explain around 90 per cent of the variation in mean score per department using these variables, which certainly suggests a high degree of correlation, at least in aggregate, but it does not shed light on differences at particular grade levels – e.g. percentage of 4*.

Such differences are, however, potentially illuminated by Mingers et al. (2012). They show the following proportions – 4* 15 per cent, 3* 31 per cent, 2* 37 per cent and 1*17 per cent – for publications entered in the 2008 exercise that appear in the ABS list. The equivalent percentages which their study estimated from the results of the exercise were – 4* 18 per cent, 3* 31 per cent, 2* 28 per cent, 1 per cent 22 per cent and 0* 2 per cent. These results suggest that the Panel awarded more 4* (if repeated across all entries this would have been about 377 extra 4* outputs) as well as more 1* (and 0*). In other words, the Panel was more positive or generous in its assessments than the ABS list at the upper 4* end. These differences may not seem great but the weights for awarding funding were strongly skewed towards 4* (4*:9; 3*:3; 2*:1; 1*:0), and so have a significant financial effect for business and management research across the UK.

Discussion: The ABS list and the Taylorization of business school research

The claim by architects of the ABS list that ‘by promoting a broader consensus...researchers will benefit collectively both culturally and economically’ (Morris et al. 2009: 1446, emphases added) rather echoes Taylor’s assertions when introducing his principles of scientific management to rationalize industrial production. The consensus claim has been challenged elsewhere (Willmott, 2011), and we have questioned the claimed material benefits of the list. In this section, we confine our comments to how the architects and sponsors of the ABS list have responded to expressions of dissensus, or cultural heterogeneity, including those which point to the adverse effects of its use upon research activity.

A specific, documented example is the misgivings voiced by the UK Committee of Professors in OR (COPIOR). They wrote to the ABS to communicate their concerns about the use of the list in certain business schools where, for example, early career academics are under pressure to publish in an ABS 4* journal before they can pass probation. In its response to COPIOR, the ABS replied that it simply produced the list, and that how the list is used is not its responsibility (email communication). Rather like Frederick Taylor, any problems associated with using the list are attributed to its imperfect or misdirected application; or, more recently, such problems have been associated with the ‘predelictions and prejudices’ of journal editors and referees, which the ABS list merely ‘reflects’ and renders more ‘visible’, and so makes more available to ‘challenge’ (Morris et al., 2011: 563). The architects and sponsors of the ABS list willingly take credit for what they extol as its individually and managerially beneficial effects (e.g. widening access to previously restricted information and assisting managers seeking ‘a reliable means of assessing the achievements of their academic staff’ (Harvey et al. 2008: 1). But, to date, they have resisted being ‘held responsible for at least some of the reasonably predictable effects of their actions’ (MacIntyre, 1999: 312), and so they effectively deny, or only selectively acknowledge, their moral agency. Is this perhaps a case of wilful blindness?

For the construction and/or use of a journal list to be contemplated and justified, its performative effects in narrowing, impeding or inhibiting diversity must be downplayed or simply ignored. In the case of the ABS list, its architects first acknowledge that the ranking of journals courts ‘the danger that highly original work fails to make a significant contribution to the field because it is damned by the name of the publication it appears in’ (Morris et al., 2009: 1449). They also concede that work appearing in a journal ranked highly in a list can be seen in a better light than it would otherwise merit. But, they then push such considerations aside in order to establish and refine a single list which aspires to be definitive, and of universal applicability. In order to create a metric with general applicability or universality, its particularity must be obscured – a particularity that unavoidably privileges the values of certain research traditions while it marginalizes others. The particularity of the ABS list is further disguised, as noted earlier, by the adoption of the 1* to 4* grading scale devised for the 2008 UK evaluation exercise. Quality criteria (see Exhibit 1) used by the Panel to assess research outputs are superficially indistinguishable from criteria used to rank journals in the ABS list. Use of the same scale is, at best, confusing and misleading. Whereas the 2008 Panel’s evaluation criteria explicitly rely upon the exercise of ‘professional judgment’ in evaluating each output on its own merits (albeit not in detail, see http://www.rae.ac.uk/pubs/2006/01/docs/i36.pdf), the ABS list assigns quality to outputs based upon place of publication whose status in the ranking is determined primarily by citation counts, impact factors, etc., moderated by the judgments of subject experts whose identification is unexplained, whose evaluation processes are undeclared, and whose inter-rater reliability, for those who believe in the meaningfulness of such checks, is apparently untested.

Policy implications

If the logic of applying the ABS list is taken to its ultimate conclusion, the 22 ‘world elite’ journals identified in its most recent edition (Version 4) should become the benchmark for business and management scholarship in the UK. Such a move is openly advocated by those, including a member of the 2008 Panel, who contend that the use of such a benchmark is the way to make UK management and business research genuinely ‘world leading’ (Saunders et al., 2011; Saunders and Wong, 2011). In Saunders and Wong’s judgment, the most rigorous metric is the University of Texas at Dallas list. Had this view been shared by the 2008 Panel members, a very small amount of funding would have been directed to a tiny number of institutions (e.g. London Business School, see Saunders et al., 2011: 412) – that is, to the few business schools in the UK that command the resources and reputation sufficient to attract researchers with the capability and inclination to confect papers compliant with the requirements of such ‘world leading journals’. At a stroke, funding would have dried up for all forms of research that challenges the hegemony of the dominant model exemplified in the content of the elite, predominantly US, journals.

If diversity of research is to flourish rather than be smothered by the homogenizing logic of journals lists, proactive interventions are required. In the UK context, we suggest that the Business and Management Panel for any future evaluation exercises might:

Reiterate the exclusion from the evaluation process of all use of journal lists.

Actively welcome the submission of research monographs and other forms of output, thereby mitigating fears that such outputs are intrinsically too weak or risky to submit.

Encourage submission of outputs pursuing a broad research agenda and adopting a wide range of research styles.

Affirm the importance of research that addresses the problems that the world faces, and not just the managerially defined problems identified by executives and/or the narrow topics and use of scientistic methods which currently dominate top-tier journals.

Comment directly upon the contribution of journal lists to the development of the field.

The use of a journal list to evaluate research quality, we have argued, induces a narrowing of scholarship and so acts surreptitiously to ‘limit academic freedom’ (Tourish, 2011: 369). Accordingly, our chief policy recommendation is a moratorium on their use. Unsurprisingly, the architects of the ABS list are hostile to this proposal. Objections to the use of journal lists to assess ‘research quality’ have also been widely ignored, resisted or rejected by university managers and narrowly careerist academics who welcome the availability of a performance metric that mitigates uncertainties about what makes an academic a more desirable and highly priced commodity – in terms of being recruitable, promotable, eligible for submission to evaluation exercises, and so on. Writing about ‘Poppleton’ (a pseudonym) Business School during the run-up to the 2014 evaluation exercise, Parker (in press) observes how, when making appointments or promotions,

the only publications that mattered were stated to be those in the (ABS list) three or four star journals, and books or chapters were treated as effectively irrelevant…The Research Dean expressed the aim of [Poppleton] for the 2013 exercise in terms of number of ABS four star publications, and a report was compiled which measured the success of each group in achieving on this measure.

What is particularly revealing about the objection by architects of the ABS list to a moratorium on its use is their assumption that this would require ‘policing’ or ‘enforcement’ (Rowlinson et al. 2011: 444), presumably by some managerial authority, rather than an agreement amongst self-regulating peers to suspend, and so effectively abolish, its use. The lead given by the 2008 and 2014 evaluation panels could be followed by collectively agreeing to return the malevolent genie to the bottle. It should be obvious enough that the call for a moratorium places no ‘curbs on academic freedom to analyse publicly available data and publish findings’ (Rowlinson et al., 2011: 444). Such data analysis can continue but our hope is that a fuller appreciation of the dubious basis and adverse effects of the use of journal lists to evaluate research quality would accompany such analysis and so accelerate their demise.

If our moratorium proposal is considered excessively radical or censorious, then the ABS or an equivalent body might develop numerous lists as a way of recognizing and encouraging scholarly diversity, innovation and multi-disciplinarity. A similar proposal has been made independently by Peters et al. (in press) who arrive at it by a different route as their concern is principally about the reliability of rater evaluations in processes of list construction rather than the effects of their use. Lists could be developed for heterogeneous forms of research – on particular topics, within specific fields or sub-fields, adopting a variety of methods and/or guided by different theoretical approaches. This alternative to a moratorium, we readily acknowledge, multiplies rather than eliminates the ‘one-size fits all’ logic that is common to all journal lists. But the resulting profusion of lists would surely serve to mitigate the homogenization of research resulting from ascribing authority to One Best List (see also Adler and Harzing, 2009: 90–1). If there were 20 or 50 lists, each covering journals relevant to a particular topic, sub-field, method or theoretical orientation, the effect would be to subvert the effect of relying upon one list to judge scholarly quality. That said, a moratorium on the use of journal lists for making decisions on recruitment, promotion and research evaluation submission is more consistent with our analysis, and so is the preferred policy recommendation. The plea to ‘develop and use [evaluation] systems that are reliable and valid and justifiable to all major stakeholders, internal and external’ (Hitt and Greer, 2012) is doubtless well-intentioned and as desirable as Apple Pie but, in our view, it is politically naïve. Its pursuit is likely to perpetuate the shortcomings of using a journal list to evaluate research quality that it is intended to rectify.

Conclusions

It is hoped that the evidence and arguments presented in this article will stimulate further discussion of the pros and cons of the use of journals lists. We have examined in some detail the application of the ABS list in the UK where the use of this list is widespread and, to their credit, its architects have been somewhat transparent about its design and on-going revisions. Rather than focus upon its distinctive and idiosyncratic features, we have stressed its typicality with regard to its construction and effects. Accordingly, our analysis and conclusions are broadly applicable to the use of the ABS list, or equivalents, elsewhere in the world.

Paying close attention to the proclaimed virtues and latent vices of journal lists might seem to be a rather esoteric and perhaps self-indulgent activity when compared with scholarship that addresses topical issues of management practice or which aspires to advance theory in much vaunted areas like strategy or finance. In our view, this judgment incompletely appreciates how the entire field of management scholarship and research can be significantly influenced and directed by evaluation criteria and metrics, such as those enshrined in journal lists. Paying attention to the effects of such quality and performance measures, as exemplified in Parker’s (in press) reflections on their Taylorian use at Poppleton Business School, is important because, rather like the time and motion techniques developed by Taylor, or the infamous evaluations produced by credit ratings agencies prior to the 2008 financial meltdown, they exert potent and pervasive performative effects in commending and resourcing some forms of (research) activity whilst devaluing and impeding others. By providing a means of adjudication equivalent to Taylor’s ‘scientific’ calculations and prescriptions, the ABS list offers a technically elegant fix for a troublesome managerial problem as it disregards its unintended consequences, including the homogenization of research.

The claim of architects of journal lists is to have invented the equivalent of ‘a better mousetrap’ – in the form of a superior device for identifying the quality of research. The numerically based, rationally calculated, centrally administered, bureaucratically assembled list is commended as a worthy replacement for the time-consuming and comparatively unsystematic process of carefully reading and considering scholarly work. In contesting this commendation, we have highlighted the spurious objectivity of citation metrics and JIFs. The use of journals lists, we have argued, valorises formal comparability to the detriment of substantive contribution and diversity. When a list mentality gains a grip, the key indicator of an academic’s scholarly credentials is no longer the potential or demonstrated capacity to develop original or significant research. Rather, it becomes evidence of the possession of the technical, instrumental competences for shaping and shoehorning research to fit the style of scholarship published in journals highly ranked by the favoured list.

A Taylorist infatuation with precision and systematicity obscures and homogenizes the features of a field which, as the creators of the ABS list themselves acknowledge, has shown considerable resiliance, up until now at least, in ‘resist(ing) normative pressures to coalesce around a set of ontological, epistemological and methodological norms (Tranfield and Starkey, 1998)’ (Morris et al., 2009: 1444; see also Becher, 1989). Research in UK business schools, as they also note, exhibits “differences in values, theoretical reference points, methods, and writing styles” (Morris et al., 2009: 1449). If this diversity is to be prized and revitalized, rather than brushed aside and smothered, then the overall assessment of the ‘quality’ of the contribution of articles that appear in a particular journal cannot convincingly be made by reference to a single, standardising and homogenising measure.

Footnotes

Funding

This research received no specific grant from any funding agency in the public, commercial, or not-for-profit sectors.

Notes

John Mingers is Professor of Operational Research and Information Systems at Kent Business School, UK, and is also the Director of Research. His research interests include research metrics and research performance; the nature of information, meaning and knowledge; the use of systems methodologies in problem situations — particularly the mixing of different methods within an intervention (multimethodology), and critical management studies. He is an Academician of the Academy of the Social Sciences, and is on the Editorial Boards of MIS Quarterly, Systems Research and Behavioural Science and Journal of Mixed Methods Research. He has published in MIS Quarterly, Information Systems Research, Journal of the Operational Research Society, European Journal of Operational Research, Journal of Critical Realism, Sociological Review, and Journal for the Theory of Social Behaviour and has published four books, the latest being Realising Systems Thinking: Knowledge and Action in Management Science (Wiley).

Hugh Willmott is Research Professor in Organization Studies, Cardiff Business School, UK and has held visiting professorships at the universities of Copenhagen, Denmark; Uppsala and Lund, Sweden; Innsbruck, Austria and the University of Sydney and University of Technology, Sydney, Australia. He previously held professorial appointments at the UMIST (now Manchester Business School) and Cambridge, UK. He co-founded the International Labor Process Conference and the International Critical Management Studies Conference. He currently serves on the editorial boards of Academy of Management Review, Organization Studies, Journal of Management Studies and is an Associate Editor of Organization. He has previously served on the board of Administrative Science Quarterly and Accounting, Organizations and Society among other journals. He has contributed to a wide range of management and social science journals and has published over 20 books. Full details can be found on his homepage: .

References

Ackerman

. (2009) The case for Soft O.R. OR/MS Today 36: 20.

Adler

Harzing

A-W

(2009) When knowledge wins: Transcending the sense of and nonsense of academic rankings. Academy of Management Learning and Education 8(1): 72–95.

Baum

(2011) Free-riding on power laws: Questioning the validity of the impact factor as a measure of research quality in organization studies. Organization 18(4): 449–466.

Becher

(1989) Academic tribes and territories: Intellectual enquiry and the cultures of disciplines. Milton Keynes: Open University Press.

Bedeian

(2004) Peer review and the social construction of knowledge in the management discipline. Academy of Management Learning & Education 3(2): 198–216.

Bedeian

Van Fleet

Hyman

(2009a) Scientific achievement and editorial board membership. Organizational Research Methods 12(2): 211–238.

Bedeian

Van Fleet

Hyman

(2009b) Circle the wagons and defend the faith: Slicing and dicing the data. Organizational Research Methods 12(2): 276–295.

Bessant

Birley

Cooper

Dawson

Gennard

(2003) The state of the field of business and management research: Reflections of the Research Assessment Exercise (RAE) Panel. British Journal of Management 14(1): 51–68.

Bryson

(2004) What about the workers? The expansion of higher education and the transformation of academic work. Industrial Relations Journal 35: 38–57

10.

Burgess

Shaw

(2010) Editorial board membership of management and business journals: A social network analysis study of the Financial Times 40. British Journal of Management 21(3): 627–648.

11.

Checkland

(1981) Systems Thinking, Systems Practice. Chichester: Wiley.

12.

Clarke

Knights

Jarvis

(2012) Labour of love: Academics in business schools. Scandinavian Journal of Management 28(1): 5–15.

13.

Espeland

Sauder

(2007) Rankings and reactivity: How public measures recreate social worlds. American Journal of Sociology 113(1): 1–40.

14.

Geary

Marriott

Rowlinson

(2004) Journal rankings in business and management and the 2001 research assessment exercise in the UK. British Journal of Management 15: 95–141.

15.

Gendron

(2008) Constituting the academic performer: The spectre of superficiality and stagnation in academia. European Accounting Review 17(1): 97–127.

16.

Giacalone

(2009) Academic rankings in research institutions: A case of skewed mind-sets and professional amnesia. Academy of Management Learning and Education 8(1): 122–126.

17.

Gibbert

Ruigrok

Wicki

(2008) What passes as a rigorous case study? Strategic Management Journal 29: 1465–1474.

18.

Gillies

(2008) How Should Research Be Organized? London: College Publications.

19.

Grey

(2010) Organizing studies: Publications, politics and polemic. Organization Studies 31(6): 677–694.

20.

Harley

Lee

(1997) Research selectivity, managerialism, and the academic labor process: The future of nonmainstream economics in UK universities. Human Relations 50(11): 1427–1460.

21.

Harvey

Morris

Kelly

(eds) (2008) Academic journal quality guide version 2. Introduction: context, purpose and methodology. Association of Business Schools available at http://www.the-abs.org.uk/files/ABS%20Journal%20Ranking%20Guide%20Introduction.pdf, accessed 20 October 2011.

22.

Harvey

Kelly

Morris

Rowlinson

(eds) (2010) Academic journal quality guide version 4, Association of Business Schools available at http://www.the-abs.org.uk/files//absalphawintro.pdf, accessed 20 October 2011.

23.

Harzing

A-W

Metz

(2012) Practicing what we preach: The geographic diversity of editorial boards. Management International Review DOI 10.1007/s11575–011–0124-x

24.

Hinings

C R

(2010) Thirty years of organization studies: Enduring themes in a changing institutional field. Organization Studies 31(6): 659–676.

25.

Hitt

Greer

(2012) The value of research and its evaluation in business schools: Killing the goose that laid the golden egg. Journal of Management Inquiry 21(2): 236–240.

26.

Hogler

Gross

(2009) Journal rankings and academic research: Two discourses about the quality of faculty work. Management Communication Quarterly 23(1): 107–126.

27.

Hopwood

(2008) Changing pressures on the research process: On trying to research in an age when research is not enough. European Accounting Review 17(1): 87–96.

28.

Hopwood

Miller

(eds) (1994) Accounting as Social and Institutional Practice. Cambridge: Cambridge University Press.

29.

House of Commons (2004) Research Assessment Exercise: A re-assessment. Science and Technology Committee, Eleventh Report of Session 2003–4, HC586, September.

30.

Hussein

(2010) Accounting journals and the ABS quality rankings. British Accounting Review 42(1): 1–16.

31.

Jack

Calás

Nkomo

Peltonen

(2008) Introduction to special topic forum: Critique and international management: An uneasy relationship? Academy of Management Review 33(4): 870–884.

32.

Jackson

Keys

(eds) (1989) OR and the Social Sciences. London: Plenum.

33.

Judge

Cable

Colbert

Rynes

(2007) What causes a management article to be cited – article, author or journal? Academy of Management Journal 50(3): 491–506.

34.

Kacmar

Whitfield

(2000) An additional rating method for journal articles in the field of management. Organizational Research Methods 3: 392–406.

35.

Kanigel

(1995) The One Best Way: Frederick Winslow Taylor and the Enigma of Efficiency. Boston, MA: MIT Press.

36.

Keenoy

(2005) Facing inwards and outwards at once: The liminal temporalities of academic performativity. Time and Society 14(2/3): 303–21.

37.

Kelly

Morris

Harvey

(2009) Modelling the outcome of the UK business and management studies RAE 2008 with reference to the ABS journal quality guide. Association of Business Schools available from http://www.the-abs.org.uk/files//RAE2008_ABS2009_final.pdf, accessed 21 October 2011.

38.

Knights

Clarke

Jarvis

(2011) It’s a bitter sweet symphony, this life: Fragile academic selves, Bristol Business School Working Paper.

39.

Knorr-Cetina

(1999) Epistemic Cultures: How the Sciences Make Knowledge. Cambridge, MA: Harvard University Press.

40.

Lee

(2011) Reflections on assessing academic quality in marketing and the UK REF. European Journal of Marketing 45(4): 477–483.

41.

Lee

Elsner

(2008) Publishing, ranking, and the future of heterodox economics. On the Horizon 16(4): 176–184.

42.

Macdonald

Kam

(2007) Ring a ring o’ roses: Quality journals and gamesmanship in management studies. Journal of Management Studies 44: 640–655.

43.

Macdonald

Kam

(2008) Quality journals and gamesmanship in management studies. Management Research News 31(8): 595–606.

44.

MacIntyre

(1999) Social structures and their threats to moral agency. Philosophy 74: 311–329.

45.

Marsden

(1993) The politics of organizational analysis. Organization Studies 14(1): 93–124.

46.

McWilliams

Siegel

Van Fleet

(2005) Scholarly journals as producers of knowledge: Theory and empirical evidence based on data envelopment Analysis. Organizational Research Methods 8(2): 185–201.

47.

McWilliams

Lockett

Katz

Van Fleet

(2009) Who is talking to whom? The network of intellectual influence in management research. Journal of Applied Management and Entrepreneurship 14(2): 61–81.

48.

Meriläinen

Tienari

Thomas

Davies

(2008) Hegemonic academic practices: Experiences of publishing from the periphery. Organization 15(4): 584–597.

49.

Mingers

(2008) Exploring the dynamics of journal citations: Modelling with s-curves. Journal Operational Research Society 59(8): 1013–1025.

50.

Mingers

Burrell

(2006) Modelling citation behavior in management science journals. Information Processing and Management 42(6): 1451–1464.

51.

Mingers

Watson

Scaparra

(2012) Estimating business and management journal quality from the 2008 Research Assessment Exercise in the UK. Information Processing and Management 00 http://dx.doi.org/10.1016/j.ipm.2012.01.008.

52.

Mingers

(2010) The drivers of citations in management science journals. European Journal of Operational Research 205(2): 422–430.

53.

Morris

(2011) Business and management research in the UK from 1900 to 2009 and beyond. In: Lee

Cassell

(eds) Challenges and Controversies in Management Research. London: Routledge, pp 30–55.

54.

Morris

Harvey

Kelly

(2009) Journal rankings and the ABS Journal Quality Guide. Management Decision 47(9): 1441–1451.

55.

Morris

Harvey

Kelly

Rowlinson

(2011) Food for thought: A rejoinder on peer review and RAE 2008 evidence. Accounting Education: An International Journal 20(6): 561–573.

56.

Nkomo

(2009) The seductive power of academic journal ratings: Challenges of searching for the otherwise. Academy of Management Learning and Education 8(1): 106–112.

57.

Özbilgin

(2009) From journal rankings to making sense of the world. Academy of Management Learning and Education 8(1): 113–121.

58.

Parker

(in press) Reflections: Changing ‘Poppleton’ Business School, Journal of Change Management, forthcoming.

59.

Parker

Thomas

(2011) What is a critical journal? Organization 18(4): 419–428.

60.

Peng

Dess

(2010) In the spirit of scholarship. Academy of Management Learning and Education 9(2): 282–298.

61.

Peters

Daniels

Hodgkinson

Haslam

(in press) Experts’ judgements of management journal quality: An identity concerns model. Journal of Management DOI: 10.1177/0149206311434532

62.

Piercy

(2000) Commentary: Why it is fundamentally stupid for a business school to try to improve its Research Assessment Exercise score. European Journal of Marketing 34(1/2): 27–35.

63.

RAE (2006) Panel criteria and working methods: I38 Business and Management Studies, available at http://www.rae.ac.uk/pubs/2006/01/docs/i36.pdf, accessed 20 October 2011.

64.

RAE (2009) RAE 2008 Subject Overview Reports: I36 Business and Management Studies, available at http://www.rae.ac.uk/pubs/2009/ov, accessed 20 October 2011.

65.

Rafols

Leydesdorff

O‘Hare

Nightingale

Stirling

(2011) How journal rankings can suppress interdisciplinarity: The case of innovation studies in business and management, Working Paper, Science and Technology Policy Research, University of Sussex, available at http://www.sussex.ac.uk/Users/ir28/IDR/Rafols2011-Rankings&IDR.pdf, accessed 27 August 2011.

66.

Rowlinson

Harvey

Kelly

Kestinova

Morris

(2010) Business history in the UK Research Assessment Exercise. European Business History Association, Newsletter 31: 4–10.

67.

Rowlinson

Harvey

Kelly

Morris

(2011) The use and abuse of journal quality lists. Organization 18(4): 443–446.

68.

Rowlinson

Hassard

Mohun

(2010) Research audits and their financial consequences. In: Cassell

Lee

(eds) Challenges and Controversies in Management Research London: Routledge.

69.

Sauder

Espeland

(2009) The discipline of rankings: Tight coupling and organizational change. American Sociological Review 74(1): 63–82.

70.

Saunders

Wong

(2011) Manoeuvring towards research decline: The RAE and the decline of Britain’s international research standing. European Journal of Marketing 45(4): 484–512.

71.

Saunders

Wong

Saunders

(2011) The research evaluation and globalization of business research. British Journal of Management 22: 401–419.

72.

Scandinavian Journal of Management (2012) Special topic forum: Critical scholars in the machinery of publishing: Experiences, reflections, alternatives. 28(3): 205–256.

73.

Scandura

Williams

(2000) Research methodology in management: Current practices, trends, and implications for future research. Academy of Management Journal 43(6): 1248–1264.

74.

Sharplin

Mabry

(1985) The relative importance of journals used in management research: An alternative ranking. Human Relations 38(2): 139–149.

75.

Singh

Haddad

Chow

(2007) Are articles in top management journals necessarily of higher quality? Journal of Management Inquiry 16(4): 319–331.

76.

Starbuck

(2005) How much better are the most prestigious journals? The statistics of academic publication. Organization Science 16(2): 180–200.

77.

Taylor

(2011) The assessment of research quality in UK universities: Peer review or metrics? British Journal of Management, 22(2): 202–217.

78.

Thomas

Wilson

(2011) ‘Physics envy’, cognitive legitimacy or practical relevance: Dilemmas in the evolution of management research in the UK’, British Journal of Management 22: 443–456.

79.

Tourish

(2011) Leading questions: Journal rankings, academic freedom and performativity: What is, or should be, the future of leadership? Leadership 7(3): 367–381.

80.

Tranfield

Starkey

(1998) The nature, social organization and promotion of management research: Towards policy. British Journal of Management 9(4): 341–353.

81.

Treasury (2006) Science and innovation investment framework 2004–2014: Next steps. Budget, March 2006, London.

82.

Truex

Cuellar

Vidgen

Takeda

(2011) Emancipating scholars: Reconceptualizing scholarly output. 7th International Critical Management Studies Conference, Naples available at http://www.organizzazione.unina.it/cms7/proceedings/proceedings_stream_13/Truex_et_alii.pdf, accessed 13 November 2011.

83.

Van Fleet

Kacmar

Griffin

Ford

Duncan

(2011) Towards finding the balance of art and science in management: A market approach to valuing management research. Journal of Applied Management and Entrepreneurship 16(1): 3–27.

84.

Wells

(2010) The ABS rankings of journal quality: An exercise in delusion. Working Paper, The Centre for Business Relationships, Accountability, Sustainability and Society, Cardiff Business School available at http://www.brass.cf.ac.uk/uploads/010610_ABS_Rankings.pdf, accessed 21 October 2011.

85.

Willcocks

Whitley

Avgerou

(2008) The ranking of top IS journals: A perspective from the LSE. European Journal of Information Systems 17: 163–168.

86.

Willmott

(1995) Managing the academics: Commodification and control in the development of university education in the UK. Human Relations 48(9): 993–1028.

87.

Willmott

(2003) Commercialising higher education in the UK: The state industry and peer review. Studies in Higher Education 28(2): 129–141.

88.

Willmott

(2011) Journal list fetishism and the perversion of scholarship: Reactivity and the ABS list. Organization 18(4): 29–44.

89.

Worthington

Hodgson

(2005) Academic labour and the politics of quality. In: Higher Education: A Critical Evaluation of the Conditions of Possibility of Resistance, Critical Quarterly 47(1–2): 96–110.

90.

Worrell

(2009) Assessing business scholarship: The difficulties in moving beyond the rigor-relevance paradigm trap. Academy of Management Learning and Education 8(1): 127–130.