Peer Review or Lottery? A Critical Analysis of Two Different Forms of Decision-making Mechanisms for Allocation of Research Grants

Introduction

It is difficult to say exactly how long human beings have used the drawing of lots to justify and legitimate decisions, but one early historical source revealing this practice is undoubtedly Homer’s epic poems. Consider the following passage from the Iliad: “Hector, Priam’s son, and goodly Odysseus first measured out a space, and thereafter took the lots and shook them in the bronze-wrought helmet, to know which of the twain should first let fly his spear of bronze” (Book III, lines 314-17). Lotteries were not at all uncommon in Greek social life and were used in a variety of decision-making contexts. In Athens, during the classical period, political positions in the governing body were filled through regular rotation of the city’s citizens using a relatively advanced system for drawing lots, a kind of randomization device called a kleroterion. It was thought that a true democracy could best be maintained through the use of a public selection system that was immune to all manner of interpersonal intrigue, power constellations, and special interests. Long after antiquity and even during the modern era, lotteries have been used in, for instance, jury selection, organ donation, military recruitment, sporting contexts, and in allocation of residences and places in educational programs (Aubert 1959; Broome 1984; Budish et al. 2013; Elster 1989; Goodwin 2005; Keren and Teigen 2010; Stone 2011).

What characterizes lotteries more generally and from a scientific perspective is that “they eschew rather than embrace identifiable elements of personal desert or social value; lotteries are driven by chance, not reason” (Kornhauser and Sager 1988, 483). But despite the strength of a lottery as an unbiased selection mechanism, it is often considered much more appropriate to use a proper assessment process based on the ability of human reason to discern, evaluate, and assume responsibility for the decision that is to be made. Naturally, there are in many cases strong ethical and political reasons for using expert knowledge instead of a lottery. The problem, however, is that we often almost instinctively avoid considering drawing lots as a method, even in situations where there is great uncertainty or disagreement as to how a given resource would best be allocated. The reason for this is that there is a basic level of ambivalence and suspicion concerning the consequences of using lotteries. Regarding this suspicion of lotteries, Elster (1989) wrote: “we have a strong reluctance to admit uncertainty and indeterminacy in human affairs. Rather than accept the limits of reason, we prefer the rituals of reason” (p. 37). Many of the assessment processes underlying decision are by necessity limited in their rationality and constitute cases of “bounded rationality” (March and Simon 1958; see also Gigerenzer and Selten 2002). A lottery, on the other hand, can be seen as a “second-order rationality,” which means that reason enters the decision-making process during an earlier phase, then taking a step back to let chance be the determining factor—this being in many cases the most rational approach (Elster 1989, 116). But arguments in favor of a lottery do of course not only concern what is rational or efficient, they also concern fairness and equality in the allocation of scare resources. In fact, the very idea of using a lottery is that random selection aims to be fair. From the perspective of moral philosophy, Broome (1984) has emphasized that: “Random selection…can help to reduce the conflict between fairness and the general good, making it possible to increase one without too much damage to the other” (p. 48). Hence, using chance may be the most just way to deal with the problem of how to distribute scarce resources and opportunities, especially in situations where there are considerable measures of uncertainty and disagreement involved. However, before a community decides to apply a lottery as a method for making decisions, it is always necessary to consider the whole spectrum of pros and cons in order to agree upon its plausibility.

When, we might ask, would a lottery be acceptable and valuable? Are there any new, conceivable cases? The fact is that such a case can be found in the realm of science, more specifically regarding the allocation of government research grants. Today, allocation of public resources occurs regularly through the process of peer review. For decades, this method of selecting which projects get funding has been the standard among research funding agencies worldwide and understanding it is key to understanding the modern conditions of doing research. With the help of panel groups of experienced experts and/or ad hoc reviewers, decisions are made every year on how a limited amount of money should be allocated to future research projects. Peer review is a meritocratic and value-creating mechanism that puts academic judgment to the highest test, as regards discovering potential, identifying quality and defining what is most just. The system of peer review is commonly regarded as a self-regulating procedure within academia in which the professional autonomy is protected against external forces. However, evaluating and comparing different research proposals is not unproblematic. On repeated occasions, the peer review process has been shown to be afflicted by a seemingly unavoidable element of chance, which has been tied to which specific individuals are chosen to make the assessments and what kind of consensus these individuals are able to reach as a group. This streak of randomness has given rise to considerable skepticism among certain parts of the scientific world concerning the peer review system’s reliability and legitimacy (Cole, Cole, and Simon 1981; Graves, Barnett, and Clarke 2011; Mayo et al. 2006). Everything from small seemingly unimportant variations in scoring to strong disagreement concerning scientific quality can have vital consequences as regards promoting certain proposals over others. Previous studies, using different methods, have shown how various kinds of cognitive and institutional bias can affect the review process (Boudreau et al. 2016; Chubin and Hackett 1990; Day 2015, Langfeldt 2006; Lee 2015; Pier et al. 2018; Roumbanis 2017; Travis and Collins 1991; Wennerås and Wold 1997). Professional expertise is accompanied by embodied knowledge, subjective preferences, emotions, and expectations, which operates as an inherent part of sound academic judgments and peer communication. Experts are situated in particular cognitive and social networks, which affects the construction of consensus during the review process in different ways (Huutoniemi 2012; Lamont 2009). Despite the expert reviewers’ joint efforts to make as well-considered and fair decisions as possible, we nevertheless cannot disregard the fact that peer review is based on compromise and risk minimization, the final outcome of which is not always perceived to be completely fair or to promote the most innovative research (Luukkonen 2012). Another issue that scholars have recently highlighted is the fact that many funding agencies and research councils today have begun to ask their reviewers to consider the potential “societal impact” of proposals. This makes the evaluation process to some extent even more complicated as it might generate a range of new tensions in the absence of methods of benchmarking this measure (Derrick and Samuel 2016, 77; see also Holbrook and Frodeman 2011).

If a lottery were used, and chance were allowed to determine the final selection, several of the problems associated with peer review would be completely or largely neutralized. But what consequences would a lottery actually have for the epistemic landscape over time? Letting chance play such an important role in an activity that is marked by fundamental trust in human reason and in basic meritocratic principles would not take place without opposition. Many researchers and politicians would see allocation of research funds using a lottery as something completely inconceivable.

The aim of the present study is to elucidate the notion that a lottery could be an innovative and radically different way of allocating government research grants. The investigation consists of three parts: (i) I will consider general issues regarding the increasing competition, inefficiency, and wasting of time associated with the current method of allocating external funding through peer review. (ii) I will illustrate the uncertainty that is associated with peer review as a selection method, as regards assessment of quality, and that in various ways marks decision-making. (iii) I will argue for the dynamics of a lottery and that it could potentially benefit the epistemic landscape over time by enabling positive random variation and the diffusion of research funds. As a decision-making method, a lottery would be cheaper, more efficient, unbiased, and in several respects fairer and more reliable. What remains to be discussed in the future, however, is the extent to which funds could be allocated through a lottery and how this might be done in relation to an increase in block funding.

Audit Culture, Hypercompetition and a Funding System in Crisis

Drawing lots has long been considered something of a curiosity with regard to allocation of research grants. It is often mentioned in a joking manner, tying to the notion that the peer review process is already a bit like a lottery. But perhaps this joke should be taken more seriously in the future, especially if we consider the increasing specialization of the sciences and the increasing complexity of the epistemic landscape. Two American professors of molecular biology, Ferric C. Fang and Arturo Casadevall (2016), recently wrote an article in which they mounted a frontal attack against the entire peer review system: “The time-honored mechanism of allocating funds based on ranking of proposals by scientific peer review is no longer effective, because review panels cannot accurately stratify proposals to identify the most meritorious ones” (p. 1). There are too many good proposals that cannot be funded, and it is often difficult to determine which researchers are most deserving of the chance to test their research idea. According to Fang and Casadevall, the peer review system has become overstrained in a time of diminishing governmental budgets for basic research; the entire system is inefficient, expensive, and unreliable, and it is often perceived as biased. The experts on the review panels have disproportionally great power to affect the fate of competent researchers. Fang and Casadevall’s counterproposal is that large government funding agencies, such as the National Institutes of Health (NIH) in the United States, should instead be using a modified lottery. As they see it, this approach would solve several problems. Moreover, they have received support for their idea from another researcher, Adrian G. Barnett (2016), who wrote: “we need to work with politicians, the public, and skeptical scientists to demonstrate how lotteries are fairer and less expensive than current funding systems” (p. 1).

However, to truly understand why it would be relevant to consider a lottery at all, it is important to get a broader grasp on the current situation with respect to research funding. In most OECD countries during recent decades, research policies have led to a reorganization of academia. The result of this reorganization is that researchers are under constant pressure to apply for external funding, creating a situation that could be described as one of unsustainable growth and hypercompetition (Fochler, Felt, and Müller 2016). In line with the sweeping reforms made in most major societal sectors since the 1980s, the universities have also been required to meet demands to be more cost-efficient, accountable, and competitive as well as to be driven by an entrepreneurial spirit (Stensaker and Benner 2013). Taken together, this means that each university must strive to cover its own costs, much like a company has to, and that each individual researcher should, in order to succeed, get into the good habit of seeking strategic collaborations, creating networks, identifying interesting funding opportunities, and being highly productive. Philosopher of science Elzinga (2012) wrote the following: “the point is that a capitalist-like behaviour and (contractual) relationships have become rather prominent within the moral economy…of academe. It is a system in which the academic entrepreneur is boosted as a role model” (p. 423; Hackett 1990). However, it is important to underscore that the natural competition that exists among scientists is not necessarily a problem per se; keep in mind, for example, what Merton (1973) wrote decades ago, namely, that we should put aside, “the myth that competition for originality in science is alien to joy in discovery and that the drive for recognition should occasion self-contempt” (p. 341; see also Osmond 1983). The main problem is rather the massive competition for resources that today dominates academic work more than ever before and which unfortunately has many negative effects on science (Chubin and Hackett 1990; Geard and Noble 2010; Hallonsten 2016; Shapiro and Vrana 2015; Sloman 2014; Stephan 2012).

It is in light of this quasi-market orientation of academia and development of the “entrepreneurial university” that the issue of research funding and the peer review system must be considered. All this in combination with an increasing amount of managerial control mechanisms and an audit culture has created new prerequisites for conducting research and for the very meaning of being an academic. In several countries, the tough competition over funding has become very prominent, and its negative effects have begun to be increasingly apparent. According to some commentators, the existing system for allocating grants has slowly begun to break down, and radical reforms are required to save scholarly endeavor from suffering great harm (Barnett et al. 2014; Ioannidis 2011). The pressure on researchers has become too great, and many feel that too much of their time is being spent on writing grant proposals—time they could have instead devoted to actually doing their research.

In a report compiled by The Royal Swedish Academy of Sciences (2010), an estimate was made of the combined time spent on writing the approximately 3,500 grant applications submitted to The Swedish Research Council (SRC) in 2008. Because on average fewer than one in four applications were granted funding, it is possible to calculate the total amount of time spent on grant writing that year that did not have any direct results and that work time amounted to approximately sixty work years. Note that the review work involved in assessing these applications was not included in this calculation—work that “takes up the time of many of the country’s most qualified researchers” (The Royal Swedish Academy of Sciences 2010, 1; my translation). The situation has hardly improved during recent years. In the SRC’s announcement regarding research funding for 2016, the total budget was 3.5 billion Swedish kronor. A total of 6,095 proposals were submitted, but only 17.3 percent of them were granted funding (SRC 2016). This shows clearly the enormous amount of available research time that is wasted and does not include the costs associated with administrating the peer review system year after year. If a lottery system and a simplified application procedure were in place (e.g., a short sketch), much of the time spent on writing, reviewing, and administrating applications could instead be devoted to conducting research.

A similar Australian study estimated that approximately 550 work years were required to submit 3,727 grant applications to The National Health and Medical Research Council of Australia, only 21 percent of which were funded (Herbert et al. 2013). This situation in the United States is similar in many respects. Fang and Casadevall (2016, 2) offered the following description:

Both applicants and reviewers have adapted to the funding crisis in ways that may be counterproductive to science. Applicants have responded by writing more grant applications, which takes time away from their research. As most applications are not funded, this largely represents futile effort. […] In contrast, reviewers are asked to decide between seemingly equally meritorious applications and may respond by prioritizing them in the basis of “grantsmanship,” which has never been shown to correlate with research productivity or innovation.

What Are the Alternatives to Peer Review?

As early as 1985, a special issue of Science, Technology, & Human Values was dedicated to the question of the peer review system’s status as a decision-making mechanism and to possible alternative models for allocating research funds. In one article, material scientist Rustum Roy leveled strong criticism at the peer review system for its poor efficiency and for demoralizing gifted young researchers who often struggle in vain to get funding. Roy’s own suggestion was as follows: funding agencies should instead allocate money to research groups/units based on their qualifications and previous output. This would amount to a performance-based system in which researchers who could demonstrate adequate measurable results (publications, citations, patents, etc.) would automatically be granted more money. In this way, the elite would not have to spend great amounts of time on proposal writing, and resources could simply be allocated through administrative decisions (Roy 1985, 77; Ziman 1983).

The fact is, however, that at present, many countries, including Sweden, use a dual funding system, in which one part of the state budget—block funding—is partly based on performance, while the other part—external funding—is the reason why peer review has come to play a more prominent role (Bégin-Caouette, Schmidt, and Field 2017; Benner and Sörlin 2007). According to sociologist Christine Musselin, the peer review system has become an important factor in the distribution of power, prestige, and influence in the academic world. The great focus now placed on external funding that is allocated through peer review has reinforced in several respects a new kind of academic elite consisting of people who occupy the most important positions on the research councils and at universities: “This overall acceptance of the external peer-review evaluations is one manifestation of the influence of the norms set by the academic elite that reviews proposals” (Musselin 2013, 1170; see also Roumbanis 2018). Peer review is key to preserve the self-regulation and autonomy of the scientific community; it is a method for scientists to ensure that money is spent on what they themselves view as the most innovative and valuable research projects. However, what does academic self-regulation really mean in practice when it is performed by a small elite? In other words, who are the “selves” of the academic self-regulation? Today’s funding system works largely for those who have a permanent position. At the same time, in Sweden, we see an unsustainable growth in the number of researchers who are forced to accept project-based employment and short-term contracts or to settle for part-time employment (Bégin-Caouette, Schmidt, and Field 2017). The lack of permanent employment positions and the general situation of tough competition are part of the crisis that is spreading within academia. This means that allocation of funding is becoming extra delicate, particularly from the perspective of fairness.

What, then, are the alternatives? Martin (2000) described a number of selection methods. In addition to peer review and performance-based decisions, he has mentioned methods based specifically on the notion of fairness. One of these methods essentially entails not making any selection at all but instead allocating the entire budget evenly across all researchers, so that everyone gets a small portion. The other model involves drawing lots, which is based on the principle that all applicants have an equal chance of being funded. The problem with the first model is that the sum would hardly be sufficient for more costly projects in the technical and natural sciences (cf. Vaesen and Katzav 2017). The model could make life easier for some researchers who already have a permanent position, but it would hardly offer a solution for the growing group of researchers who have no established position or who work at smaller university colleges where the block funding cannot support very much research. The most serious objection to both of these decision-making models is that they would result in more money being wasted on researchers who are not particularly productive and, thus, cause more qualified researchers to become highly frustrated (Martin 2000; see also Höylä, Bartneck, and Tiihonen 2016). But what is it that makes certain qualifications more qualifying than others? What is it that makes one research idea better than another, and what is it that actually counts? Having funding enables productivity, which leads to more funding, resulting in stronger positions, without this saying anything about the lasting scientific value of the outcomes of this upward spiral.

Chance and Consensus in Peer Review

As Lamont (2009) explained, peer review is “an evaluative technology that puts connoisseurship and judgment at the center of the evaluation process” (p. 51). Proposals may be evaluated by ad hoc review with no panel meetings, meaning that reviewers are invited to assess particular proposals based on the match between the topic and their special expertise. This form of evaluation is usually conducted via a web-based system. The reason why some funding agencies are using this method is that it is cheaper and, in some respects, more efficient. For example, a study conducted by Fogelholm and colleagues (2012) showed that panel discussions did not improve the reliability of the peer review process. Yet, many funding agencies, including for example the US National Science Foundation and many European research councils, use combinations of ad hoc reviews and panel meetings. Face-to-face meetings are still favored by many organizations because negotiations play an important role in the social creation of trust and legitimacy (Lamont 2009). When reviewers meet in panel groups to jointly identify, negotiate, and make decisions about which grant proposals deserve funding, the issue of fairness and impartiality is crucial. Which projects should be ranked highest, what it is that should be assessed, and how should the reviewers come to an agreement? Much of the review work is done before the panel group meets, when the individual reviewers separately evaluate and score the applications. During the panel group meetings, however, much of the work is concerned with creating consensus, adjusting the scores and jointly making a final ranking of all of the proposals (Langfeldt 2001; Roumbanis 2017). The various combinations of similar or dissimilar epistemic perspectives and quality standards result in different boundaries being drawn that affect the prerequisites for creating consensus in the group. Differences in the intellectual distance between the researcher who has written a grant application and the expert reviewers evaluating it may, for perfectly natural reasons, have significant consequences for the assessment of quality. A reviewer whose own expertise is close to that of the applicant tends to be a tougher critic than a reviewer who is not as familiar with the subject (Boudreau et al. 2016). But sometimes the opposite applies, and this openness to the interpretative space of academic judgment is one source of the variation that can emerge (Kuhn 1977). It is in this dynamic assessment context, where qualities are translated into numerical scores, that an intrinsic separation occurs that has consequences for which scientific projects are or are not given the opportunity to develop in the future.

In a now well-known experimental study published in Science in 1981, Stephen Cole and colleagues showed how an ordinary NSF panel group and an equivalent control group assigned significantly different scores to 150 grant applications. The variations in judgment revealed that not even the highest ranking proposals’ positions were secure: “proposals that were rated in the top quintile by NSF reviewers…would not have been funded had the decision depended on the appraisal of the COSUP reviewers” (Cole, Cole, and Simon 1981, 882). The study revealed the surprisingly great role played by chance, which was tied both to which individual reviewers were chosen to participate in the panel group (“the luck of the reviewer draw”) and to how disagreement affected the creation of consensus. According to Cole and colleagues, these results call for a discussion concerning the extent to which peer review is truly a reliable and legitimate method of selection.

More recent studies have also shown problems associated with variations in assessment and scoring. In one Canadian study, Mayo et al. (2006) revealed relatively great variations in the review process—variations that were directly related to what specific pair of reviewers evaluated the various proposals.

The top rated project in each stream would have failed the funding cutoff with a frequency of 9 and 35%, depending on which pair of reviewers had been selected. Four of the top 10 projects identified by ranking had a greater than 50% of not being funded. (Mayo et al. 2006, 842)

With regard to the majority of grant applications that end up in the middle of the ranking based on their scores, chance always comes into the picture: “These proposals are the most difficult to separate and a slight change in score can push a proposal below or above a funding line” (Graves, Barnett, and Clarke 2011, 2; see also Day 2015). The presence of chance has been shown not only through quantitative experimental studies comparing scoring and measurement variations in assessment retrospectively but also through ethnographic studies in which experts with great experience have reported feeling that chance does make its way into the review process (Lamont 2009; Roumbanis 2017).

Uncertainty, Disagreement, and Scientific Conservatism

According to one recurring criticism, the peer review system, as a selection method, evokes scientific conservatism and risk minimization, which in turn result in low priority being given to genuinely innovative and unorthodox projects (Hackett 1990; Luukkonen 2012). The origins of this tendency can be found in the demand that each panel group should strive to reach a consensus. The rules guiding the review process require the group, while working against the clock, to arrive at a pragmatic solution to the question of allocation. Funding agencies such as the SRC typically claim that the goal is to allocate funds to the most innovative and original research ideas, but in reality, many of these projects are marginalized. The emergence of uncertainty and disagreement is one factor causing reviewers to often choose the more “solid” alternatives that everyone can agree on and that typically fall inside the boundaries of current paradigms. It is essentially never the case that a proposal is funded when only one reviewer gives it the highest score. On the other hand, a proposal can be held down if only one reviewer in a panel group is highly critical. This is obviously a problem given how the peer review system is often organized today, with scoring and rank ordering of proposals: “The concern is that screening with average score tends to eliminate riskier high-return science as there is tremendous variation in perceived magnitude of the values of the research at the time of initiation” (Linton 2016, 2). The numerical scores create a kind of discrepancy between quality and quantity. This discrepancy is related to the overall uncertainty that marks all types of peer review involving evaluation of originality and innovation. The cause of this uncertainty is typically tied to the issue of what it is exactly that characterizes truly innovative research and whether the reviewers involved are able to discern what may potentially be pioneering. In a recent study, Pier et al. (2018) replicated the NIH peer review process by examining forty-three individual reviewers’ ratings and written critiques of the same twenty-five grant applications. The results from the analyses showed surprisingly low levels of agreement among the reviewers regarding the quality of the applications in neither their qualitative nor quantitative evaluations, “Although all reviewers received the same instructions on how to rate applications and format their written critiques, we also found no agreement in how reviewers ‘translated’ a given number of strengths and weaknesses into a numerical rating” (Pier et al. 2018, 1).

In addition to assessment of scientific quality and risk, there is the question of productivity. What can we expect to get for the funds we invest in new research projects? This calculation is largely based on the assessment of previous publications and previously funded grants. A certain kind of success breeds more success. Most people would agree that what prevails today within academia is a “publish-or-perish culture” and that this has led to researchers not always thinking through their findings with sufficient rigor or researchers choosing not to be sufficiently self-critical (Hallonsten 2016; Rekdal 2014). The peer review process tends to reinforce the importance of number of publications, impact factors, and citation indexes. However, actual potential is fundamentally more difficult to assess. It is in fact impossible to predict because it often does not occur as expected or based on established templates. How would, say, Wittgenstein have managed as a researcher in the current British system? Some years ago, philosopher Gillies (2009, 5) wrote:

Wittgenstein published nothing during the last 17 years of employment at Cambridge. On the current RAE system, he would have been classified as research inactive, and have had his research time removed and would perhaps even have been sacked. Yet during these 17 years, Wittgenstein wrote the Philosophical Investigations, which many regard as the best philosophical work of the 20th century. To generalise from this case, we want our research system to allow researchers, if they are so inclined, to spend a long time polishing and re-polishing their works before publication. We know that this strategy can sometimes result in durable masterpieces.

The Hypothetical Effects of a Lottery on the Epistemic Landscape

The most common objection for using a lottery is that it does not take qualifications into account and that this entails the risk of favoring many more poor projects than one would have using peer review (Höylä, Bartneck, and Tiihonen 2016; Martin 2000). However, as I attempted to claim in the preceding section, it is often highly uncertain what it is that determines whether a proposal will end up at the top or bottom of the ranking. Moreover, many of the projects that are granted funding through peer review do not deliver on their promises, and very few generate results that are truly pioneering. This does not mean, of course, that they are poor projects, but it does leave us wondering what other project should have been given a chance instead and if that unfunded project could have produced more interesting research. The content of a grant application certainly has something to say about the researchers who wrote it and their proposed project, but it cannot predict whether the project will lead to important breakthroughs that will advance science in a radically new way (Roy 1985; Sloman 2014).

But can we, in some way, get a hypothetical indication of what could make drawing lots a better decision-making method than peer review? In his dissertation entitled Breaking the Grant Cycle, philosopher of science Avin (2015) presented findings that support the somewhat counterintuitive notion that a lottery has the potential to be a more impartial, efficient, and dynamic way of allocating research funds. To my knowledge, this is the first systematic study to thoroughly theorize around drawing lots as a selection mechanism and to critically compare it to peer review. Avin’s starting point can be seen in part as an attempt to promote the epistemic argument that it would actually be more rational if society were to use a lottery when allocating public resources to research. The methodological snag here is, of course, that there are hardly any funding agencies that employ a lottery, making it impossible to start from larger, comparable empirical data sets. For this reason, Avin instead compared computer simulations of a number of funding allocation models, which he called “old boys,” “best visible,” “lottery,” and “triage.” Yet, despite this seemingly unavoidable limitation, Avin (2015) presented what I consider a solid justification for the method he chose to use in his investigation:

…it may be useful to think of the simulation as a formalised thought experiment. However, unlike the thought experiment, which concretises by loading a hypothetical anecdote with what are taken to be exemplary characteristics, the simulation concretises by assigning numerical parameters to what are taken to be key processes. (p. 148)

Avin explained that, on a static epistemic landscape, “best visible” (which represents peer review) performs better than both “old boys” and “lottery.” On a small-scale dynamic landscape, the three models perform fairly equally. The significant effects, however, are seen on a large-scale dynamic landscape, where a lottery outclasses the other two models: “This is because new avenues on a large landscape are likely to spawn outside the visibility of the agents, where lotto can access them but the other two strategies cannot” (Avin 2015, 144). The clear advantage of the “best visible” model is that, in the shorter term, it can meet the standards of epistemic fitness at any given point in time. But genuine basic research always exists in a high state of tension in relation to the unknown, and for natural reasons, academic judgment is biased toward the current state of knowledge and previous experience. Avin (2015) summarized his study in the following manner:

The simulations showed that on large dynamic landscapes, selecting projects at random outperforms the selection of the highest fitness projects from those which are similar to past projects […] within a set of idealising limitations, a selection of projects that relies on past experiences, such as can be expected from peer review, performs worse than random selection. (p. 181)

Concluding Discussion

The very thought of drawing lots to allocate research funding would certainly seem both provocative and absurd to many researchers and politicians. It would probably be perceived as a way to destroy the legitimacy of modern science. The proposed use of a lottery is at odds with ingrained ideas about the meritocratic principles that govern the sciences today as well as with the predominating notion that funds should be allocated based on competition. However, several studies have revealed the shortcomings of the peer review system, which involves many different kinds of cognitive and institutional bias. Chance has repeatedly been discovered in the review process and shown to affect funding decisions to a relatively great extent. Moreover, the peer review system appears to be rather inefficient in that it takes up a disproportionately great amount of researchers’ time—time that could be spent on actually doing research. A lottery would solve several of the problems tied to the current funding crisis by making the selection process more impartial while better serving the public interest. My primary goal has been to explore the very idea of drawing lots as an alternative selection method on which to base future funding decisions. This idea can be related, though indirectly and from a more general societal perspective, to Luhmann’s (1987) recommendation that, on the whole, we should: “take advantage of the application of chance to create structures in more and more complex societies” (p. 16; from Buchstein and Hein 2009). The old idea of using lottery to allocate desirable resources and values has once again come into the light, and several new proposals have been made concerning when this selection method could be applied (Budish et al. 2013; Landemore 2013; Pluchino et al. 2011).

It is true, to be sure, that a lottery does not only evade unwanted bias and various conflicts of interest but is also blind to obvious differences in qualifications and inherent scientific qualities that could be recognized by an experienced reviewer. But what are these scientific qualities exactly? Isn’t this the fundamental problem of peer review? How can we identify and agree on which of two equally good projects is the better one—how can we do this before the projects have been carried out?

As a group, researchers on review panels have great influence for a number of years over what kind of research projects will or will not have a chance of being funded; they constitute a kind of panel of judges whose opinions are allowed the control the fate of many researchers. One good argument for instead using a lottery is that it can break up the potential power and influence a small group can have over an entire community, just as in the case of democracy in ancient Athens. Moreover, one emotional advantage of drawing lots was pointed out by Elster (1989) with regard to a different historical context: “To be rejected by fortune was less dishonourable than to be rejected by the community” (p. 107). This is particularly relevant in cases where applicants have proposed projects with great potential that were nonetheless rejected in favor of other projects. Regarding such cases, many commentators have suggested that a lottery would seem to be more appropriate, given its fundamental impartiality and fairness. A lottery draws no boundaries based on specific preferences or arbitrariness. Just as Avin (2015) showed in his study, the peer review process is firmly anchored in what is, as regards assessment, “visible” at a given point in time, despite the fact that science is fundamentally much more dynamic, on both the theoretical and social level. Still, the use of random selection in research funding must first be tested and evaluated, so that we may learn more about its effects.

By way of conclusion, I wish to present two deviations that also serves as evidence of present-day use of a modified lottery as an alternative to traditional peer review. In New Zealand, a lottery system has been used for several years where “Explorer Grants” are awarded by the Health Research Council of New Zealand (HRC). This is still done on a relatively small-scale basis, but its extent is growing and, as I see it, it reflects the substantial need for creative reforms in the current organization of research funding. On their website, the HRC (2018) explain the procedure as follows:

Applications for explorer grants are assessed by subpanels within the HRC’s Explorer Grant Assessing Committee to see if they meet the criteria of being both transformative and viable. Unlike with any of our other grants, the assessment process for explorer grant applications is anonymous and all applications that meet the criteria are equally eligible to receive funding. A random number generator prioritises these applications.

This is, as far as I know, the first time the outcome from peer review and a lottery will be compared within the same call. Other research councils and private funding agencies should follow this example and perform their own tests. If there are no substantial differences or even an improvement with the lotteries, the lottery component could be expanded. However, social psychologist Hans-Dieter Daniel has raised some challenges regarding the Volkswagen Foundation’s pilot study. For example, how can we know that the screening performed by the staff is efficient? It would probably be more legitimate if external reviewers from the academic community made the preselection. This is a practical problem that could easily be solved, however. Another, more complex issue which is directly related to the future evaluation of the entire selection process is that it might be “hard or even impossible to find matching pairs of grants for the comparison of outcomes” (Daniel 2018). Although it is important to be aware of these and other challenges when exploring the possible benefits of lotteries, reviewers face the same challenge when they compare the quality of different proposals asking different questions and employing different methods. Still, this type of comparison can at least give us some concrete evidence that could complement the positive results from Avin’s simulation study. A lottery system is the kind of innovation the research community needs, and it would probably have several positive effects. It would reduce costs, free up time for research and increase the heterogeneity of funded projects through the diffusion effects of chance. The system could not be criticized for being biased (owing to nepotism, conflicts of interest, discrimination, etc.), and increased block funding would give research environments more power over the research directions they wish to pursue locally and allow them to take a more long-term approach. In a more complex future, where the epistemic landscape and societal development are increasingly difficult to evaluate from a global perspective, use of a lottery could be an important factor in facilitating the organization of research resources and increase the epistemic diversity, fairness, and impartiality within academia.