Rewarding Replications

Why Do Psychologists Neglect Replication Research?

Strictly speaking, a study is a replication when researchers repeat all the relevant aspects of an original study. This procedure is also called a direct replication (Schmidt, 2009). The odds that any given original empirical finding will replicate are inherently unpredictable (Miller, 2009; Miller & Schwarz, 2011). Researchers should hence conduct direct replications to ensure the reliability of newly published findings. A given researcher is inevitably prone to idiosyncrasies in procedural details or how she or he treats participants (to name two examples). The most compelling direct replications are therefore conducted independently by different researchers than the original study. Direct replications should be distinguished from conceptual replications, which test the hypothesis of earlier studies while using a different study design (Schmidt, 2009; Tsang & Kwan, 1999). This distinction is important because conceptual replications, unlike direct replications, cannot disconfirm an original set of findings (LeBel & Peters, 2011). In this article, we are mainly interested in direct, independent, and explicit replications because these provide the strongest tests of the robustness of an original finding. For the sake of brevity, we refer to the latter studies simply as “replications.”

Most psychologists readily acknowledge the value of replications, at least in principle. In practice, however, editors and reviewers of psychology journals overwhelmingly recommend against the publication of replications (Neuliep & Crandall, 1990, 1993). To illustrate this practice, consider the recent article on extrasensory perception (Bem, 2011) that was published in the Journal of Personality and Social Psychology (JPSP), a flagship journal of the American Psychological Association. When an independent research team submitted a manuscript describing three failed replication attempts to JPSP, the editor refused to send out the paper for review (see Aldhous, 2011, for more details; see also Galak, LeBoeuf, Nelson, & Simmons, in press, for an update). While conceding that these replications were publishable in principle, the editor explained that, “We don’t want to be the Journal of Bem Replication.” Although this example is specific to one journal, other psychology journals have similar publication policies.

Why do psychology journals systematically refuse to publish replications? The answer to this question likely relates to psychology’s historical roots. Before the rise of modern psychology in the late 19th century, the study of human nature was primarily conducted by scholars from religious, philosophical, and literary disciplines. These scholars made use of hermeneutic-interpretive methods, which originate in the work of ancient Greeks and remain in use in the humanities today (Grondin, 1994). Central to hermeneutic-interpretive methods is the assumption that phenomena emerge from a complex and unique combination of historical, cultural, and social influences. Phenomena are irreducible to these influences. Nevertheless, phenomena can be interpreted through a narrative that organizes observations in a new and coherent manner. To researchers who adopt a hermeneutic-interpretive framework, replications are of little value. Indeed, true replications may be considered impossible because phenomena are believed to be inherently unique and unrepeatable.

With the advent of modern psychology, the study of the mind has shifted toward a natural sciences approach. The natural sciences are historically associated with logical positivism, a school of thought within psychology that emphasizes logical deduction and controlled empirical observation (Ayer, 1936; Bergmann, 1957). Positivist philosophy has been severely criticized, and even the natural sciences do not adhere to positivist principles (Cacioppo, Semin, & Berntson, 2004). However, most natural scientists do embrace some form of philosophical realism, which assumes that observed phenomena reflect a more basic ordered reality that exists materially and independently of the mind (Cacioppo et al., 2004; Tsang & Kwan, 1999). Realist philosophies regard true knowledge as an ideal that is worth striving for (Cacioppo et al., 2004). To researchers adopting a realist framework, replications are essential to scientific progress; only replicable findings can be relied upon as law-like regularities (Popper, 1959; Radder, 1996; Schmidt, 2009).

The natural science approach has come to predominate in modern psychology, as evidenced by its widespread use of experimental methods, objective measures, and statistics. Nevertheless, some vestiges of the older hermeneutic-interpretive approach remain. One of these vestiges can be discerned in how psychologists usually report their empirical findings. Psychological journal articles typically embed their data in a meaningful narrative, which not only surveys the relevant literature but also contains broader philosophical, theoretical, and rhetorical arguments (Bem, 1987). The narrative form of psychological articles stands in contrast to most articles in the natural sciences, which are typically briefer and restricted to factual observations.

A narrative approach to scientific publications has many things going for it. Good stories are attention-grabbing and memorable. These are highly favorable qualities that promote the rapid dissemination of scientific findings and ideas. However, a narrative approach has at least one unintended side effect: it works against the publication of replications. Replications are by their nature repetitions of original research. One of the basic rules of effective communication is that speakers should avoid repetitions and redundancies (Grice, 1975). From a narrative perspective, then, researchers who are reporting a replication study are like an uncle who keeps telling the same joke year after year at family events.

A narrative approach to scientific publishing may further distort the true scientific meaning of replication research. From a scientific perspective, replications are worthy of attention regardless of whether their results confirm or disconfirm an original set of findings. As we noted, neither kind of replication is very attractive to begin with from a narrative perspective. Nevertheless, replications that disconfirm prior findings have at least some storytelling value, because they violate an expectancy that has been set up by an original study. Replications that confirm prior findings do not have this narrative advantage, because they merely subscribe to the conclusions of an original piece of research. Scientifically speaking, confirmatory replications can afford a “quantum leap” (Tsang & Kwan, 1999) in increased confidence about a set of findings. However, this increase in confidence offers little suspense and is therefore all but useless from a narrative perspective.

A narrative approach may thus lead researchers to spend inordinate amounts of attention on disconfirmatory replications—to the extent that replications get published at all.

Ironically, this trend is the opposite of the general tendency within psychology (and other branches of science) to publish mostly confirmatory results (Bones, 2012; Fanelli, 2012; Sterling, Rosenbaum, & Weinkam, 1995). Due to this confirmatory publication bias, disconfirmatory replications are more surprising, which renders them even more attention-grabbing than confirmatory replications. As a result, replications are likely to become associated with controversy. This is unproductive and unjustified: Just as a single confirmatory finding gives little certainty a true effect exists, a single disconfirmatory finding is no evidence of a lack of a true effect (Cumming, 2008). The negative perception of replications may carry over to researchers who are engaged in replication research, as they may be perceived as hostile toward the researchers who conducted the original research. Taken together, the narrative approach can easily distort the scientific meaning of replications by fostering perceptions of replications as derivative or inherently hostile toward original research.

Rewarding Replications

Replications are the primary tools to warrant the trustworthiness of psychological science. But how can psychologists be induced to regularly replicate each other’s work? Many psychologists currently have negative attitudes toward replications (Neuliep & Crandall, 1990, 1993). Changing these professional attitudes will be an important first step. One recent initiative that has highlighted the value of replications is the Open Science Framework: a large-scale effort among more than 50 scientists to conduct replication studies of the key effect of the first 30 articles in three high-impact psychology journals that were published in 2008 (Open Science Collaboration, 2012, this issue; see also Carpenter, 2012). The Open Science Framework has developed a structured protocol for designing and conducting replications, which may serve as a blue print for further replication efforts. Initiatives of this kind surely help to put replications on the research agenda.

But there is much more to be done. The current incentive structure within psychology gives rise to a social dilemma in which researchers’ collective interests run in the opposite direction as the interests of individual researchers. The research community collectively benefits if researchers take the trouble to replicate each other’s work, because this improves the quality and reputation of the field by showing which observations are reliable. Yet, individually, researchers are better off by conducting only original research, because this will typically yield more publications and citations and thus ultimately greater rewards in terms of better jobs and more grant money. It seems unrealistic to ask researchers to conduct replications when doing so will hurt their career. Psychologists would thus do well to develop a new incentive structure that strikes a more equitable balance between rewarding original research and rewarding replications.

Professional incentives for researchers are largely derived from their number of scientific publications and the frequency with which these publications get cited. A handful of simple and inexpensive changes in the publication process could go a long way toward creating a favorable incentive structure for replications. The required changes are by themselves straightforward. However, because the steps are directed at changing a complex system, it is important to proceed strategically and with caution. In the following sections, we outline three key incentives for replication research, and provide concrete suggestions for bringing them about.

Incentive #1: Copublications

Replications have little meaning when their results remain unknown to the broader academic community. Thus, if psychologists are to engage in systematic replication research, they should have outlets where they can publish the resulting findings. Limited publication resources function as a bottleneck that keeps replications from getting published (Suls & Martin, 2009).

One way to shatter this bottleneck would be to create separate publication outlets for replication studies. For instance, Hartshorne and Schachner (2012) have proposed an open-access journal dedicated to publishing replication attempts. However, this approach has important disadvantages. Traditional journals have a wide readership that will not easily turn to specialized replication journals. Publishing replications in journals that few people read is only a marginal improvement over the current situation in which replications are hardly published at all.

An alternative would be for existing psychology journals to make room for the publication of replications. Journal space has been traditionally limited by the maximum number of pages that could be printed per issue. However, this limitation no longer applies in the present age of electronic publishing, where journals can publish virtually unlimited amounts of information online (Nosek & Bar-Anan, 2012). Many scientific journals already capitalize on these new electronic resources, for instance, by publishing methodological details about an article as supplementary online materials. The same online resources can be used to publish replications. Replications could be published in a special section of each journal, much like the sections that are currently used for making editorial statements. If journals still appear in print, the print version of a journal issue could contain short abstracts of new replications. The complete manuscripts of replications could be published as supplementary online materials, which become instantly available upon accessing the original report. In this manner, replications would become equally accessible to readers as original research.

Copublishing replications and original research is cost-effective, because it uses the existing infrastructure of psychology journals. This allows replication studies to benefit from an editorial team and a network of reviewers who are knowledgeable in the domain of replication studies, given that they were involved in editing and reviewing the original research. To relieve the burdens on existing journals, restrictions could be applied to the kinds of replications that are eligible for reviewing. First, only explicit, direct, and independent replications may be submitted, because these are most informative regarding the dependability of original findings. Second, editors and reviewers primarily need to verify the methodological rigor of replication, as well as its statistical power to detect an effect. Reviewing replication reports should hence be less time consuming than reviewing reports of original research. Reviewers may further evaluate whether all or only a subset of an original report’s findings were investigated and whether all, none, or some of the attempted findings were successful. These (averaged) evaluations of the reviewers may be published along with replication reports.

Third, existing journals may publish only a fixed number of replications. Generally speaking, the first few replications are the most important in establishing the robustness of an original finding (Tsang & Kwan, 1999). Given the marginal diminishing returns of adding more and more replications, it would make sense for journals to restrict the number of replications they publish. The maximum number of replications can be determined on the basis of a formal analysis that incorporates the expected effect size and desired statistical power of the research (e.g., see Borenstein, Hedges, Higgins, & Rothstein, 2009). Thus, restricting the number of replications would have the added advantage of allocating replications more evenly across different lines of original research.

The accessibility of replications could be further enhanced if journals would publish statistics that summarize the results of replication research. By aggregating the information in replication reports, journals can compute the rate by which replications have confirmed original research findings. The rate of confirmatory replications can be thought of as the “replication factor” of an original research report. Once computed, replication factors can be displayed automatically in electronic databases like Google Scholar and the ISI Web of Science. Hartshorne and Schachner (2012) have provided a detailed account of an automated system for tracking the replication factors of original research. In addition to replication factors, these databases can display the number of replication attempts. Reporting the average effect size of replications will further serve as a continuous measure of replicability, which can serve as input for meta-analyses.

Incentive #2: Cocitations

Although all publications are valuable to a researcher’s career, some publications are more valuable than others. The value of a given publication is commonly determined by counting how often it gets cited in the professional literature. Currently, replications receive few citations, because academics most often cite research that breaks new grounds. When replications are rarely cited, researchers have little to gain from conducting and publishing replication research. Therefore, an additional mechanism is needed to ensure that replications get cited to a degree that is comparable to original research.

One fruitful approach to insuring that replications are cited may be to move toward a more meta-analytic view of scientific knowledge (Cumming, 2008) in which a single study is no longer sufficient to consider a phenomenon as established. In practical terms, this means that researchers may cite multiple replications along with an original report when they are referring to an established phenomenon. Journals may facilitate this practice by compiling replication reports of the same original study together in a single file, which we might call the “Replication File”. Each time the original research gets cited, there could be a cocitation of the Replication File and, thereby, to all the researchers who contributed a replication. In effect, this would mean that citations of original research are extended to its replications. But most researchers would agree that it is still sensible to put a premium on original research. This premium can be achieved by restricting cocitations of replications to a limited period of time, say, the first 5 years after the publication of the original research. This procedure would allow replications to obtain a fair number of citations, while original research would still be rewarded with more citations overall.

Cociting replications has additional advantages. First, cociting replications makes it more attractive for researchers to replicate highly cited original research. Ensuring the trustworthiness of that influential research should have greater priority than ensuring the trustworthiness of obscure lines of research. Cociting replications with original research thus stimulates researchers to conduct replications in precisely those domains where replications are needed most.

A second advantage of cociting replications along with original findings is that it rewards researchers for quickly conducting replications. The sooner a replication gets published, the longer it benefits from the time window during which the replication reaps its cocitations.

A third advantage of the cocitation approach is that it makes the publication of replications more attractive for journals. Just like individual researchers, scientific journals are evaluated in terms of their citation impact. So far, this has meant that publishing replications has been rather unattractive for journal editors because of the low numbers of citations that replications attract. This price tag would diminish if replications are cocited with the reports of original research.

Incentive #3: Collaborative Professional Models

Copublications and cocitations should go a long way toward stimulating replication research. However, these two incentives do not speak to the everyday activities of researchers. The meaningful integration of replications into everyday research practice will further incentivize replications. One way to achieve this would be to make replications part of the academic curriculum. Frank and Saxe (2012) describe how psychology students may conduct replications of recent research as part of their coursework in experimental methods. Replication research may thus become part of an apprenticeship, in which students learn new methodological skills, while simultaneously adding to the scientific knowledge base. By helping to lower the costs of replications, widespread adoption of the apprenticeship model would allow replications to become a standard part of psychological research.

The apprenticeship model does have certain limitations. First, the model seems most useful for replicating relatively simple studies with easily accessible samples (see Frank & Saxe, 2012). Second, peer review of replications remains important because poorly conducted replications can lead to unfounded conclusions and thereby spoil the scientific database. In addition, it may be useful to report how many prior replications were conducted by researchers who report a new replication study, along with the statistical power and “hit rate” of these attempts. For students who are conducting a replication as part of their curriculum, it could be informative to report the track record of their advisor or their academic institution. These statistics may be published together with the average replication factors of the research that the students were trying to replicate, so that they can be properly interpreted.

Even when replication researchers work closely together with the original researchers and take every imaginable precaution, replications can never repeat original research in all respects. (This is not even possible in the natural sciences, where phenomena can often be controlled to a greater degree than within psychology.) Thus, it is always conceivable that a replication yields different findings than an original study because of some influential but unidentified aspect between the two studies. When such contradictory findings arise, the researchers conducting the original research and those who conducted a replication may try to resolve their differences in an adversarial collaboration (Mellers, Hertwig, & Kahneman, 2001).

Adversarial collaborations have so far been rare within psychology, where contradictory findings have typically been ignored or fueled animosity between research groups. Nevertheless, exemplary research by Latham, Erez, and Locke (1988) has shown that different sides of a scientific dispute can join forces in a harmonious and scientifically productive manner. In this case, two research teams with contradictory findings observed each other’s procedures and then jointly designed a series of critical experiments that allowed them to resolve the controversy. Mellers et al. (2001) have laid out some useful ground rules for establishing adversarial collaborations, such as designating a neutral arbiter who collects the data. Adversarial collaboration does not guarantee that researchers will end up fully agreeing with another (see Mellers et al., 2001) Nevertheless, even when disagreements remain, adversarial collaborations can generate important new insights into why different researchers may come up with different empirical findings.

Conclusions and Outlook

Despite its growing methodological sophistication, modern psychology has neglected one of its most fundamental scientific instruments by conducting little systematic replication research. This neglect seems largely due to the fact that psychologists receive hardly any rewards for conducting replications. To improve this situation, we have outlined an incentive structure for replications. First, replications should be published in high-impact journals. Second, replications should get cited. Third, replications should be used as teaching devices in psychology programs and as vehicles for constructive professional dialogue through adversarial collaboration.

The rewards for replication research that we have outlined are technically feasible, financially inexpensive, and involve just a handful of changes in publication policies and educational programs. Moreover, the required policy changes are relatively modest and could be quickly implemented. If a sufficient number of psychologists agree about the utility of rewarding replications, they could advocate policy changes with organizations such as the American Psychological Association (APA) and the Association for Psychological Science (APS). These organizations represent the professional interests of psychologists and are thus well-positioned to spearhead a movement to incentivize replication research. For instance, professional organizations could sponsor programs that require PhD students to conduct replications as part of their doctoral training. Organizations like the APA and APS further own many of the most influential psychology journals. These journals could lead the way in publishing replications and ensuring that they become widely cited.

Conducting replications will inevitably take up some of researchers’ time and resources. In the long run, however, doing so is likely to save time and resources. For one thing, researchers will waste fewer resources using unreliable procedures. Replications further provide information about the effect size of published findings, allowing researchers to optimize their methods and maximize the success rate of their own studies. Because unreliable results will be more rapidly identified, it will become easier for researchers to develop theoretical integrations of existing knowledge. These improvements are also likely to increase psychology’s influence on other disciplines and policy domains. Ultimately, this will make it more attractive for society to devote resources to psychological research.

The benefits of rewarding replications far outweigh the costs. Psychologists would therefore do well to actively discuss, design, and implement a system that makes it rewarding for researchers to conduct systematic replications. Rewarding replications will not solve all of psychologists’ problems. Indeed, psychologists will still face the many challenges that come with unraveling the mind and its complexities. Nevertheless, a psychological science that rewards replications will address these challenges with greater confidence, because it can rely on a robust knowledge base. Rewarding replications is therefore a sure and simple way to improve psychological science.