Short,Sweet,and Problematic? The Rise of the Short Report in Psychological Science

Vanishing effects

Faced with this press toward shorter and faster publications, it is important that we also acknowledge and consider the costs or potential pitfalls that accompany this approach. For example, the much-discussed New Yorker magazine article on the so-called “decline effect”—the observation that many scientific findings shrink in size or disappear entirely across successive replications—placed a spotlight on the problems of publication bias and selective reporting in scientific fields including our own (Lehrer, 2010). The hare approach to publication may be particularly likely to exacerbate these problems for a number of reasons. First, single-study papers are simply more likely than multistudy papers to identify false positives: Mathematically, if a null hypothesis is true, one out of every 20 single-study papers that tests it will erroneously find a significant effect, whereas a replicated result squares that probability to one in every 400. If we acknowledge that many researchers attempt to test a new idea more than once before abandoning it and may include multiple measures to try to “catch” the effect of interest, then that “one in 20” estimate may begin to look more like one in 10 or one in five (see also Ioannidis, 2005, for a discussion of inflated error probabilities in scientific research). In fact, in a simulation study on the effects of “undisclosed flexibility” in the way that psychologists collect and analyze data, Simmons, Nelson, and Simonsohn (2011) found that current practices can produce Type I error rates as high as 60% for a single-study finding—a truly unsettling number to contemplate.

A second way that the hare approach can escalate the problem of vanishing effects is due to confirmation bias: As researchers, we are more likely to follow up false positives than false negatives (see also Smithson, 2011). When taking a slow-and-steady, tortoiselike approach, researchers seeking to publish programmatic packages of studies that replicate and extend an initial finding can be waylaid by a false positive result but will eventually abandon the unreliable finding in favor of one that can be replicated. In the shorter and faster framework, the initial false positive is publishable, and the field follows it up, so that regression to the mean from the initial unusual result becomes a fieldwide rather than individual-level problem.

Moreover, getting results out fast can spur interest in a greater number of possible effects from a wider range of researchers. As noted earlier, this can be good when it increases scientific interaction, but it also has a downside: As the number of researchers testing a (false) hypothesis increases, the likelihood that one or two or several of them will find a false positive result increases as well (Ioannidis, 2005; Pfeiffer & Hoffmann, 2009). It is somewhat discomfiting to contemplate the possibility that there are more researchers currently chasing false leads than in any other time in our field’s history, and a published finding could be replicated simply because of the number of researchers testing it. When combined with the issue of publication bias (the fact that significant findings are published and null results are shunted to a file drawer), this means that what seems like fruitful interaction among researchers can in fact reflect a fieldwide wild goose chase. Presumably, tools such as meta-analysis allow us to figure that out eventually (insofar as they can successfully include unpublished data), but not before we have devoted considerable time and resources to chasing that initial, published false positive result.

Decreasing integration

In addition to exacerbating the issue of vanishing effects, short papers may run the risk of decreasing the extent to which we integrate the research in our field. Insofar as the publication of numerous short papers (versus fewer, longer packages of related studies) tends to flood the field with many apparently novel, disconnected findings, it can be difficult to maintain a coherent picture of the current state of knowledge in different areas of psychology. Moreover, abbreviated introductions can help authors squeeze into tight word limits and save readers from slogging through lengthy literature reviews, but they also surrender the opportunity available in longer introductions to provide a rich retelling of the history of an area and integrate past findings with current ones. Abelson (1997) has emphasized the critical importance of the “lore of a field”—the informal, qualitatively rich, accumulated understanding of what we know in different areas. Part of what sustains this lore is the scholarship that longer articles are more likely to do in terms of linking new findings to past research. Thus, an unchecked and unconsidered adoption of the hare approach to publication can jeopardize our ability to integrate and synthesize results into a rich and coherent understanding of current knowledge in the field.

Likewise, we risk losing the history of our field: Truncated introductions are less likely to reach back to seminal early works in psychology to situate the current research within its historical context. Moreover, insofar as short introductions reduce the necessity for rigorous scholarship, they increase the likelihood that we rediscover what we already know. Thus, if the flood of apparently disconnected findings rises too high without a comparable increase in the amount of synthesis going on elsewhere (e.g., review articles and meta-analyses), we move toward a fractured, piecemeal, and repetitive science rather than a cumulative and incremental one. Awash in disconnected findings, it can become difficult to construct a consensus about where the field has been, where it is going, and what the important questions are.

Focusing on effects

Short-paper formats also may contribute to this disintegration to the extent that they emphasize effects-oriented research to the exclusion of more theory- and process-oriented work. Identifying interesting, novel effects is obviously a critical component of scientific progress in our field. However, in the absence of careful follow-up work to identify the underlying mechanisms that account for an effect, such a finding is unlikely to be integrated into the larger body of knowledge that comprises our science. After all, scientific integration often occurs at the process level as we identify common mechanisms that underlie a range of seemingly disparate effects. Likewise, theoretical innovation depends on identifying and testing process-focused accounts. The specification of underlying mechanisms is critical for transforming an interesting effect into a more abstract theory that can not only explain the effect, but also make novel predictions about other effects and integrate the new effects with prior research.

All of this is more difficult to accomplish within a short-paper format. Process-oriented research often resists brief description, requiring relatively extensive explanations of theories, predictions, methods, and supporting or disconfirming results. In contrast, it is often far more straightforward to describe a study and its outcome: the effect. It is therefore perhaps unsurprising that effects-oriented research is so well represented among short-paper formats. As a consequence, even if each finding provides a potentially interesting avenue for future research, our increasing adoption of the hare approach has yielded a proliferation of stand-alone effects that have not been explained and that are not integrated into the broader base of knowledge.

Posing for the press

One factor that may exacerbate all of these problems is the current trend toward valuing newsworthy findings that are engaging and accessible to the public. As noted earlier, our field has been increasingly emphasizing the importance of effectively communicating our results to those outside the discipline. Features such as the “On the Newsstand” section in the APS Observer and the similar section prominently placed on the society’s website highlight the importance of media coverage, and the new journal Social Psychological and Personality Science was explicitly created to publish reports that are accessible not only to a wide audience within psychology but also to the popular press (SAGE, 2009). But communication is a two-way street: As we have sought to more clearly convey what we do and why it is important, we have also learned what types of research seem particularly engaging to an outside audience. First and foremost (and not surprisingly), outside readers prefer brief accounts of our work that are easily consumed. Moreover, from an outside perspective, eye-catching and counterintuitive findings can be the most immediately appealing. The media often want—and even explicitly ask for—catchy, easy-to-digest, and phenomenon-based findings, so that “newsworthy” ends up meaning flashy rather than necessarily scientifically important (Beck, 2010; Begley, 2010). One of us recently received a media inquiry from a writer who took care to describe exactly the type of work that would be most appealing to his editor, explaining: “I’m hoping that some research you’re currently pursuing—perhaps working with children, or using an fMRI—lends itself to the sort of colorful description that can make an ideas-heavy piece go down more smoothly.” Likewise, research suggests that merely including a colorful image of a brain or mentioning something about brain scans leads people to judge the scientific reasoning in a paper as more convincing, even if the reasoning is objectively poor in quality (McCabe & Castel, 2008; Weisberg, Keil, Goodstein, Rawson, & Gray, 2008), suggesting that flashiness can matter more than logic in guiding nonexpert perceptions of research.

The potential problem, then, is that when we seek to communicate with an audience that values these qualities, that information can trickle back upward and influence what we value in our own research. Indeed, there seems to be a growing sense among researchers that flashy, newsworthy findings are more valued not only by the outside world, but also within our field and that such efforts are more likely to be published than are less sexy efforts focused on cumulative progress (Mischel, 2008; Wager, 2009). At the same time, scientists increasingly view press coverage as an important component of their portfolios, as reflected in the growing emphasis on media attention found on psychologists’ websites and CVs in recent years. All of this would seem to enhance the appeal of pursuing colorful demonstrations of counterintuitive effects rather than more nuanced examinations of basic processes. For the reasons described above, the short-paper format presents a very useful vehicle for the presentation of such effects.

Thus, the trend toward valuing newsworthiness can compound the potential problems of the hare approach. Placing a premium on brief reports of flashy findings can exacerbate the issue of declining effects, as single-study findings that lack a theoretical context or that contradict previous data are especially likely to be spurious (in Bayesian terms, such findings have lower priors and thus require stronger evidence than findings consistent with previous bodies of theory and research). At the same time, the focus on effects rather than processes produces an overabundance of disconnected findings that are isolated from one another and from the field at large. Ironically, then, as we strive to increase outside appreciation of our field’s importance and appeal, we may be risking some of what makes it important and appealing to us in the first place.

Length versus strength: Reducing page quantity while maintaining article quality

As many published short reports already clearly demonstrate, it is possible to pursue shorter and faster publications while maintaining high standards for the strength and reliability of evidence and while valuing scientific importance over simplicity or natural media appeal. For example, short articles can obviously contain more than a single study and can rule out alternative explanations just as carefully as a longer article. A simple replication lends a great deal of confidence to a single-study finding, as discussed earlier, and can be succinctly reported in a paragraph or even a footnote. Additional data or analyses to rule out an alternative explanation can be provided for a careful initial round of reviews and then placed in online supplementary materials rather than the main text of an article (as is the policy at Science, for example), although this strategy will only be effective at balancing length and strength insofar as the supplementary materials are organized and reviewed as rigorously as the rest of the manuscript. Thus, the word count of an article can typically be separated from the amount (and quality) of data reported, and we can increase our confidence in short reports by setting high standards for their data content despite their overall brevity.

Moreover, if we do want outlets for single-study findings (and, as noted earlier, there are reasons to value these), we could employ analytic strategies that better account for the tentative nature of a single finding and the increasing confidence that comes from later replications. For instance, Bayesian analyses can take into account the number of replications supporting a finding so that confidence in the result increases with each successive replication; they can also take into account the prior probability of a hypothesis so that we can require stronger evidence for less likely claims (e.g., a counterintuitive finding that is difficult to link to the existing literature in comparison with an incremental finding that builds on previous research; see, e.g., Kruschke, 2011; Wagenmakers, Wetzels, Borsboom, & van der Maas, 2011, for more on Bayesian methods). We could also offset the potential inflation of Type I error rates in single-study publications by bolstering our field’s ability to self-correct. For instance, Spellman (2010, 2011) has proposed that editors could identify difficult-to- replicate phenomena and commission meta-analyses to more efficiently draw out file-drawer studies and advance our understanding of the topic under investigation. Likewise, journals or societies could create searchable online repositories for data (and perhaps a brief abstract) from null effect studies, so that (a) it would be easy to quickly check whether many studies have failed to replicate an effect before assuming it will replicate in one’s own research, and (b) editors and researchers interested in meta-analyses could quickly and accurately gauge which phenomena seem potentially finicky and ripe for further investigation. To this end, Spellman (personal communication, September 18, 2011) has requested that APS establish a wiki to create a searchable database for replicable and non-replicable experiments (see www.psychfiledrawer.org); likewise, www.openscienceframework.org seeks to provide a new shared infrastructure for storing and searching methods and data.

As a field, we can also take care to separate length from depth. Valuing brevity need not mean that we place greater priority on getting results “out there” than on making sure the results advance our understanding of basic process. We can demand just as much from short articles as from long ones in terms of providing evidence that sheds light on underlying mechanism and contributes to theory development. Clearly, papers can be short without being shallow.

In addition, we can acknowledge that there are two routes to increasing the visibility of our field to the public. We can direct our journals to publish colorful and nonintuitive findings that lend themselves to attractive headlines. However, we also must improve our ability to communicate the value of complex, process-focused work to the outside world. The first route may work in the short run: Catchy findings naturally grab public attention. But in the long run, a string of flashy but disconnected findings cannot build a coherent picture of our field and is unlikely to communicate the importance of what we study. It may be time to turn instead toward the more difficult task of clearly explaining the value of studying basic processes and toward finding ways to communicate our findings to the outside world without relying on the sometimes oversimplifying filter of the media (e.g., Banaji, 2010; Lilienfeld, 2011; Taylor, 2009).

A plea for balance: The tortoise and the hare

Although the issues described above are not inherent to the hare approach, others are more inextricably tied to short-and- fast publications. Theoretical integration and literature reviews take space and are simply more difficult to fit into a short paper than a longer one. As authors, editors, and reviewers, we may want to think more carefully about whether a given set of findings needs or deserves more space than a short format allows.

Just as we can work to minimize the drawbacks of short reports, we can also take concrete steps to minimize the disadvantages of longer papers. One of the major downsides to longer articles is the sheer amount of time and resources they require from authors, reviewers, and editors (e.g., Park, 2009; Taylor, 2009), and a relatively straightforward change that could have an immediate and dramatic impact would be to streamline the review process. For instance, many researchers have advocated the role that an active editor can play in making the review process more efficient by triaging papers as they arrive (i.e., immediately returning papers that are clearly unsuitable for the journal, as is the policy already at a number of journals including Psychological Science; see Cooper, 2009; Hojat, Gonnella, & Caelleigh, 2003; Park, 2009; Schwartz & Zamboanga, 2009), soliciting reviews from two or at most three reviewers (Cooper, 2009; Zanna, 1992), relying on reviewers primarily for new submissions and only occasionally for two rounds of reviews (Schwartz & Zamboanga, 2009; Tsang & Frey, 2007; Zanna, 1992), and allowing authors to forward reviews from a previously rejected submission along with a cover letter detailing the revisions to a new journal (as is the policy at Perspectives on Psychological Science, Social Cognition, the Journal of Research in Personality, and the Neuroscience Peer Review Consortium; see Cooper, 2009).

The lack of space in short reports for theory building also highlights the need for more purely theoretical papers that integrate across the abundance of short reports and help organize the many separate empirical findings generated by a hare approach to publication. Yet we must also acknowledge that when we separate the bulkier part of theory generation and development from empirical research in this way, we impair our ability as reviewers and readers to evaluate the connection between the two. In a theory-rich, longer empirical article, we can judge for ourselves whether the data support the conceptual framework. When we rely instead on purely theoretical papers to integrate purely empirical ones, we are more dependent on the views and judgments of the authors of a particular theoretical paper and are less able to judge this for ourselves. Increasing opportunities for theoretical discussion and expert interaction in the pages of our top journals (e.g., Spellman, 2011) or in online forums may be promising options to provide outlets for empirical and theoretical synthesis without losing the critical back-and-forth that is often so important for theory generation and development.

Finally, it is important to acknowledge that the short report format is simply not built for some things that we want to value and keep as a field. Longer articles are certainly more difficult to write, review, and read, and they can sometimes be unnecessarily lengthy. However, longer articles can contain elegant and programmatic series of studies that make a significant theoretical contribution; test moderators and mediators; and build a highly coherent, nuanced, and compelling empirical story. In the face of the hare approach’s easy appeal, we need to actively protect the value we place on the longer, slower, and more integrative tortoiselike approach. If our students note that they would rather skim through a handful of short reports rather than trudging through a longer article, we need to remind them of what the programmatic work can tell us that an isolated study or two cannot. If we see people bean-counting publications in the hiring, review, or tenure process, we should help clarify which pieces are likely to reflect a greater contribution to the field and encourage them to weight publications accordingly. And when we think about our own research, we should consider which projects deserve a slower and more programmatic treatment before they are ready for publication. The hare has a number of benefits, but our field—and our science—needs the tortoise, too.