Are Review Teams Getting Too Involved in the Revision of Manuscripts?

When I began my academic career in the 1970s as a graduate student the journal review process was focused on determining whether submitted research made a contribution to knowledge and was presented clearly. Rejection rates were very high at the top tier journals, just as today. But as both an author and reviewer, I have noticed some rather dramatic changes over time, especially for empirical articles. Different from today, well-designed research testing important issues could be published even if it was not especially novel or presented counterintuitive hypotheses. Even well-designed replication studies were published (McKinley, 2010). In fact, the Academy of Management Journal's Research Notes and the Journal of Applied Psychology's Short Notes were explicitly for replication studies and those for which the hypotheses were not supported. Also, back then, papers granted revise and resubmit opportunities were typically accepted after one, and in some cases two revisions. It was not until the 1990s that I started noticing that manuscripts along with editor and reviewer comments were getting longer and the number of revisions requested increasing while at the same time becoming more common for revise and resubmit papers to be rejected. ¹ Behind these changes, a clear pattern emerges. Editorial teams today often do not act just as gatekeepers, making sure that accepted research employs valid research designs, makes a value-added contribution to knowledge and is well-presented, but often request major changes in focus and the dropping of data or hypotheses or the collection of additional data. Although sometimes this results in papers that are substantially better than the ones originally submitted, other times it may compromise the integrity of the research, especially when key parts of the research are removed at the request of the editorial team. An example is a manuscript that was submitted to a top tier journal as a leadership paper and was even funded by an agency for its contribution to leadership knowledge. During the review process all reference to leadership was removed at the request of the review team. Although not previously tested in the literature, the review team did not think that the leadership effects on team process were novel or particularly interesting, even though leadership played a dominant role in the initial submission. The first stage of the model that was tested in the initial submission comprised of leadership variables. The associate editor (AE), who is not a leadership researcher, was only interested in the second stage of the model, which involved team process variables. The focus of the original submission was the influence that leaders have in determining the team process of the members of their work groups. The research ended up being published as a “teams” paper. The authors certainly could have removed the paper from the review process when asked to take leadership out of the paper, but perhaps found it too emotionally taxing to withdraw a revise and resubmit paper from a top tier journal's review process given the high premium placed on publishing in top journals (Starbuck, 2016).

More important than simply being frustrating for the authors and the funding agency, taking leadership out of the original model, along with the accompanying hypotheses, essentially compromised the integrity of the research. Suppressing results of hypothesis tests amounts to a specific type of harking (Kerr, 1998), which Rubin (2017) labeled sharking. Extending beyond Rubin's definition of sharking as the removal of hypotheses that are not supported, ² a specific form of sharking identified by Lishner (2021, p. 252) involves the “suppression of a priori hypotheses that are supported by the results of a study but are not favored by the researcher.” I contend that the example of the review team requiring that the leadership portion of the tested model be removed represents yet another form of sharking: when results not favored by the review team are suppressed. So, in addition to the practice of journal editors encouraging sharking by requesting that authors remove unsupported hypotheses (Rupp, 2011; Starbuck, 2016), they also are guilty of sharking when they require that authors remove hypotheses that the review team does not see as being particularly interesting, despite being supported empirically. ³

Suppressing the leadership results in the example in which the review team omitted leadership from the paper also leads to an “unmeasured” variables problem that has been identified as an especially serious issue when testing structural equation models, the form of analysis used by the researchers in the example. The omission of key causes in a structural equation model “results in biased estimates of structural parameters,” (James et al., 1982, p. 46). Related to the cautions raised by James et al. (1982), taking out leadership from the research appears to have also created an endogeneity problem (Busenbark et al., 2022), given that results of the original submission showed that leaders played a key role in affecting the motivation, level of cooperation, and ultimate effectiveness of the teams that were studied.

The competition between journals may be at least part of the reason that top tier journals have developed a strong preference for manuscripts that present novel and counterintuitive results. Papers that test published theory or are replication studies are summarily desk rejected. As noted by Hollenbeck and Wright (2017), this quest for provocative research has incentivized researchers to engage in various forms of sharking and has encouraged review teams to promote it for the purpose of increasing the novelty of the research. This, however, is not to suggest that reviewers and AE's are not well-intentioned. Quite the contrary, in my experience as an author and as a reviewer, AEs and reviewers devote substantial time and effort during the review process. They read papers carefully and provide well-articulated detailed comments. My perception is that on average, AEs and reviewers spend considerably more time reviewing papers today than they did when I was getting started in my career. Reviewers also tend to go beyond criticism by providing constructive suggestions for improvement. The question I am raising is whether, in some cases, they are getting too involved helping authors make a “valued-added” contribution. By acting almost as though they are coauthors, they may in effect reduce the academic freedom of the authors and inadvertently participate in the proliferation of sharking.

The pressure on editors to increase visibility relative to competing journals as well as the need for researchers to publish in top journals for the purpose of reaching the career objectives of getting a good job, becoming tenured, and finally promoted to full professor appears to have caused our field to drift away from following the scientific method that we advocate in our methods courses. Platt's (1964) classic article on the role of alternative hypothesis testing as a key part of programmatic research remains required reading in many doctoral programs. However, despite teaching the value of disproving hypotheses, journals, and thus researchers, tend to focus on providing support for a favored hypothesis(ses). In our methods classes we stress that theories can be falsified but not proven to be true. Yet, in practice, journals focus on support for provocative hypotheses instead of encouraging tests of alternative hypotheses in the attempt to falsify. Rather than directly testing competing hypotheses, at best, we add control variables. In one of our papers in which we presented competing hypotheses, review teams at several journals where the paper was rejected, complained that we should have presented only the hypothesis with the strongest theoretical rationale, counter to the recommendations of Platt (1964) and others, such as Bacharach (1989) and Mackenzie and House (1978).

So, what can we do to address the issues raised here? I am not suggesting a return to the past. Indeed, problems with our research and its evaluation existed then as well (Dunnette, 1966) and the field has made great strides forward, especially with the sophistication of our analyses. Reviews tend to be more developmental than in the past. It seems that what we need is a balance between the old and new approaches to reviewing research. Review teams need to maintain high standards and provide guidance to help authors craft papers that best express their ideas/findings, but at the same time should respect the integrity of the submitted work. We should require strong theoretical rationale for hypotheses without insisting that researchers develop new theory or present counterintuitive results. Indeed, the classic hypothesis that happy workers are productive workers sounds obvious and not very interesting, yet thousands of studies producing inconsistent results demonstrated the importance of post-hoc theorizing to determine why the results were inconsistent and to identify boundary conditions for the relation between job satisfaction and performance. Review teams should demand solid research designs that focus on validity but should honor the academic freedom of authors by not attempting to completely overhaul what was originally submitted. Doing so should reduce the number of revisions requested, ⁴ speeding the dissemination of research findings and assisting authors in their quest to meet career objectives. Replications should once again be recognized as a vital part of the scientific process by being published, even in top tier journals, as was previously the case (McKinley, 2010). I applaud the creation of the Journal of Management Scientific Reports, but given the importance of replication to scientific method, top tier journals should return to their practice of publishing replications. Scientific honesty should be rewarded so that authors never feel compelled to adjust models or hypotheses based on results. ⁵ Openly presenting post-hoc findings in discussion sections, or tharking (Hollenbeck & Wright, 2017) should be encouraged. To reinforce such an approach to our research and the way it is reviewed, perhaps training could be provided for reviewers and editors so that expectations are clarified. In sum, review teams can serve multiple stakeholders at once: helping authors best present their ideas for the benefit of their careers while at the same time serving the academic community through the creation of knowledge.