The Characteristics of Quality Scholarly Submissions

Abstract

This editorial examines author-reported information on author team characteristics and decision points during the paper development process. In addition to presenting the typical practices of our authors, we present data linking author team practices to the actual recommendations and ratings made by reviewers on these papers. The ultimate goal of this editorial is to attempt to identify some potential drivers of publishing success at Journal of Management. While there does not seem to be a magic feather that predicts manuscript success, two of our findings suggest that seeking outside expertise when developing a team of authors, and frequent electronic communication with those authors, might have a positive effect on the assessed quality of the manuscript.

Keywords

publishing research author teams collaboration

As a community of scholars, we all desire for our research to have the greatest impact possible. Publishing quality research in quality outlets allows us to best disseminate our results, advance knowledge, and contribute to our own professional development as well as that of our students and collaborators. There exist myriad resources for authors that provide guidance on building theory (e.g., Colquitt & Zapata-Phelan, 2007; Corley & Gioia, 2011; Fulmer, 2012; Hillman, 2011; Mayer & Sparrowe, 2013; Okhuysen & Bonardi, 2011), reviewing the literature (e.g., Bauer, 2009; Cropanzano, 2009; LePine & Wilcox-King, 2010; Short, 2009), and developing quality research (e.g., Aguinis, Boyd, Pierce, & Short, 2011; Bono & McNamara, 2011; Feldman, 2004; George, 2012; Kepes, McDaniel, Brannick, & Banks, 2013; MacKinnon, Coxe, & Baraldi, 2012; Ployhart & Ward, 2011; Rogelberg, Adelman, & Askay, 2009). In addition, studies have been conducted showing the ways in which collaborations influence the quantity of papers published by an individual author (Katz & Martin, 1997; Lee & Bozeman, 2005) as well as how characteristics of published papers influence their citation rates (e.g., Judge, Cable, Colbert, & Rynes, 2007; Starbuck, 2005; see also Ilgen, 2007). And of course, there is the voluminous teams literature, which provides insights on the predictors of team productivity in general (e.g., Mathieu, Tannenbaum, Donscach, & Alliger, 2014; Park, Spitzmuller, & DeShon, 2013).

What we do not yet know much about is how characteristics of author teams, and the decisions they make together as they move through an editorial process, predict the quality of papers, as the papers are perceived by reviewers. The scientific record is grounded in the peer review process, and the success or failure of a paper at any given journal starts and stops with reviewer reports, which are then synthesized by action editors, who issue final decisions. Such reviewer reports contain not only qualitative comments that are provided to authors but also quantitative ratings on criteria of relevance to the journal, which assist action editors in making editorial decisions.¹ We are aware of no research that has considered reviewer reports as data in the study of research collaboration. This is likely due to the fact that such data are rarely available. Whereas researchers have access to the published record (i.e., accepted papers) and citation rates, reviewer ratings of accepted and rejected papers are not publicly available, nor are they shared with authors as part of the review process.

Journal of Management (JOM), in collaboration with Journal of Business and Psychology (JBP), has been conducting a systematic study of author teams and reviewer perceptions in an attempt to “pull back the curtain” on this particular issue. We have compiled the reviewer ratings of papers submitted to JOM, which were sent out for review (i.e., not desk rejected) between the years 2008 and 2012. In addition, we sent a survey to the first author of each of these papers, asking him or her about many aspects of the paper’s history and the decisions made leading up to submission.² Combining these data sets allowed us to explore how authorship team composition and the decisions of the team related to assessed quality at JOM.³

In this editorial, we provide a high-level description of what we have learned thus far from these data.

Sample Characteristics

Author Team Characteristics

Table 1 provides summary information about the JOM submissions composing our sample. Among these papers, author team size ranged from 1 to 10 authors, with an average of 2.89 authors per paper (median = 3.0). Authors reported affiliations from all over the world, with nearly half reporting North American affiliations, about a quarter reporting European, and a 10th reporting Asian affiliations (with smaller percentages of Australian, South American, Middle Eastern, Indian, and African affiliations reported). Interestingly, whereas about half of the first, second, third, and fourth author seats were held by North Americans, when a fifth author was included, this position was 78% North American affiliated. Among second and third author positions, approximately one third were held by authors in Europe, and those positions were held by authors in Asia about 13% of the time.

Table 1

Descriptive Information Regarding Journal of Management (JOM) Reviewed Submissions (Accepted 2008-2012 and Rejected 2011-2012)

Variable	%
First author employment status at project origin
Graduate student	35.4
Assistant professor or equivalent	33.1
Associate professor or equivalent	14.4
Full professor or equivalent	16.3
Position outside of academia	0.8
Location of first author
Africa	0.4
Asia Pacific	14.0
Australia	4.3
Europe	28.4
India	0.4
Middle East	1.6
North America	50.6
South America	0.4
Primary discipline of first author
Business policy and strategy	24.5
Work/organizational/industrial/occupational psychology; micro OB or HR	41.6
Strategic HR	4.3
Organizational theory	5.4
Management–other	13.2
Psychology–other	4.7
Other	6.2
Article level of analysis
Micro (i.e., individuals as level of analysis)	43.6
Meso (i.e., teams/groups as level of analysis)	20.6
Macro (i.e., firms or broader as level of analysis)	28.0
Research methods focus	1.9
Other	5.8
Nature of article
Empirical	84.0
Non-empirical	13.2
Other	2.7
Papers submitted to a conference prior to JOM submission	74.3
Funding
Full internal funding	5.5
Full external funding	10.9
Partial internal funding	7.8
Partial external funding	10.9
Both internal and external funding	4.3
None	60.5
Articles based on broader data set (used in other papers)	22.4
Articles receiving friendly review prior to submission	76.2
Author teams adding a coauthor later in the paper’s development	16.8
Articles where final submitted research question differed from original question that prompted the research	17.2
Articles submitted previously to other journal	82.2

Note: OB = organizational behavior; HR = human resources.

We asked authors to indicate the percentage of contribution made by each member of the author team, both overall and specific to different stages of the research process (i.e., developing the research question, designing the study, collecting the data, analyzing the data, and writing up the paper). We were pleased to see that, compliant with the ethical standards of our field, authorship order was reflective of reported contribution, with earlier listed authors reported as contributing more than later listed authors both overall and across the process categories.

The first author position tended to be occupied mostly by assistant professors and graduate students. However, as the author positions progressed (i.e., from first to second to third), full professors tended to occupy the roles more frequently. We found an equal spread of authors collaborating within the same versus across universities. Most first authors had worked with their coauthors previously, with the large majority having worked with all of their coauthors three or more times previously. For the papers composing our sample, 60.5% of the research was not funded, 16.4% was partially funded, and 23.1% was fully funded.

As is shown in Table 1, 41.6% of the authors listed their discipline as micro organizational behavior/human resources, with 43.6% of the papers reported as focusing on the individual level of analysis. A quarter of the authors listed their discipline as business policy and strategy, with 28% of the papers reported as having focused on the meso (teams) and 28% reported as having focused on the macro (organizations and higher) level of analysis. The majority (84%) of submissions were classified as empirical.⁴

Author Team Decisions

Before their papers were submitted, respondents reported that they had received an average number of 1.70 friendly reviews. Three quarters of respondents submitted their papers to conferences before they submitted the paper to JOM. Seventeen percent of author teams reported adding a coauthor in later stages of the paper’s history (e.g., following a revise-and-resubmit decision). The large majority of communication between authors was conducted face-to-face or via e-mail. Following those two, phone was the next most frequent mode of communication, with Skype being the least frequent. This pattern held across the various stages of the research process, although e-mail was used more frequently (62% of the time) during the writing stage.

Nearly a quarter of the author respondents reported that the data reported on in their paper were from a larger data collection effort. Nearly a fifth reported that the final research question reported in the paper differed from that originally developed during the project’s early inception.

To What Extent Do These Characteristics Influence Reviewers’ Reactions?

We analyzed the extent to which author team characteristics and decisions affected reviewer/editor reactions to papers. We considered three outcome variables: (a) whether the paper was accepted or rejected, (b) reviewer recommendations (i.e., the aggregate across reviewers as to whether the paper should be rejected, revised and resubmitted, etc.),⁵ and (c) reviewer composite scores (i.e., the average across the evaluation criteria, averaged across reviewers). With regard to author team characteristics, we examined the role of team size, geographic diversity, institution diversity, and grant funding. When assessing author team decisions, we looked at friendly reviews, prior presentations of the paper at conferences, the addition of a coauthor, methods of communication, major modifications to the paper, and previous submissions to other journals.

In general, we detected no “magic feathers.” That is, no set of ideal steps for composing one’s author team and working together emerged from the data, although some findings are worthy of note.⁶

Institutional Distribution of Authors

First, our analyses did imply a benefit to collaborating with scholars from other institutions. Specifically, we found that there was a marginally significant difference between accepted and rejected manuscripts in terms of institution distribution, t(203) = −1.68, p < .10, such that accepted manuscripts were written by teams with fewer team members in the same institution (M = .42) than rejected manuscripts (M = .52).⁷ This was even more strongly the case for micro (compared to macro/strategy) papers, t(85) = −2.82, p < .01. For both micro (r = −.26, p < .05) and macro/strategy papers (r = −.29, p < .05), reviewers’ recommendations were associated with institution distribution such that more positive recommendations were associated with having coauthors at universities other than one’s own.

We suspect that whereas collaborating with local colleagues does have relevant proximity advantages, and offers important opportunities for cross-pollination across topic areas, it is not always the case that individuals can be found in one’s own department who have the specific expertise germane to a developing research idea. Indeed, expanding one’s coauthor pool to the entire universe of scholars allows for the recruitment of specialized knowledge and expertise that may not be available locally. This would suggest that researchers not necessarily “walk the halls” in search of collaborators but, rather, as new research ideas begin to form, that they think very strategically about who, among the entire population of relevant scholars, might offer something unique to the project and would be rewarding to work with. This speaks to the well-documented value of networking (Sparrowe, Liden, Wayne, & Kraimer, 2001; Wolff & Moser, 2009).

“Friendly” Reviews

It has been a long-standing policy at JOM for authors to first receive a “friendly review” from a trusted colleague prior to submission, and many papers and conference sessions on increasing publication success point to this practice as highly developmental. That being said, we are aware of no empirical analysis of the effectiveness of such a practice. Our findings revealed that papers that obtained a friendly reviewer were more associated with reject as opposed to accept decisions, χ²(1, 239) = 3.30, p = .07. There was also a statistically significant negative relationship between whether a friendly review had been conducted and both reviewer recommendations (r = −.16, p < .05)⁸ and reviewer composite ratings (r = −.11, p = .10).⁹

Of course endogeneity must be considered here. It may be that “better” papers with more compelling results are less likely to need friendly review, so it is not that friendly review “hurts” a paper but, rather, that authors are making accurate judgment calls regarding the added utility of requesting a friendly review. Regardless, obtaining a friendly review is a costly endeavor, both to the authors (who are delayed from submitting) and to the friendly reviewer to whom the authors become indebted. Further, even when expert friendly and journal reviewers are employed, there is always a likelihood that their recommendations will not converge. As such, addressing a friendly reviewer’s concerns might spark additional concerns of a journal reviewer. Berka, Olien, Rogelberg, Rupp, and Thornton (in press) also bring up the fact that since friendly reviewers are not anonymous, friendly reviewers may “hold back” on their criticism in order to maintain positive relationships, which would limit the developmental value of this exercise. Combined, these issues would call into question the utility of the friendly review process, at least among more expert author teams and for papers that are relatively more compelling theoretically and methodologically. JOM is taking into consideration modifications of its friendly-review policy following further investigation of this issue.

Caveat on Conference Presentations

Whereas we did not find that previously presenting a paper at a conference was associated with how the paper fared at JOM, JBP data did reveal a significant, positive association between conference presentations and paper success (Berka et al., in press). Despite our null finding, we continue to see value in this practice, especially relative to the practice of obtaining a friendly review. Whereas a friendly review may significantly delay a paper from being submitted and is limited to the feedback of one individual, conference reviews occur on a set time frame, often involve multiple reviews, and once the paper is presented, offer the opportunity for a wide variety of feedback from individuals with differing perspectives in addition to opportunities for networking with potential reviewers and/or future coauthors. Future research should continue to investigate this issue.

Communication

We also explored whether different communication methods could have an influence on manuscript outcomes. We first looked at whether the use of rich (Skype, face-to-face) and nonrich (e-mail, phone) forms of communication had an effect on reviewer recommendations, composite scores, and manuscript decisions. While there was no relationship between the use of rich forms of communication and manuscript outcomes, there was a correlation between nonrich forms of communication and reviewer composite ratings (r = .17, p < .05). As this was certainly an unexpected finding, we probed further and looked at the use of specific communication media on our outcomes. Of all our communication methods, the use of e-mail had a significant effect on outcomes. First, manuscripts that were accepted had greater use of e-mail communication (M = 233.88) across the paper development process than those that were rejected (M = 190.56), t(193) = −2.21, p < .05. In addition, there was a positive correlation between use of e-mail and both reviewer recommendations (r = .16, p < .05) and reviewer composite scores (r = .21, p < .01).¹⁰

Past research has suggested that the use of e-mail might be a detriment to team functioning (Shapiro, Furst, Spreitzer, & von Glinow, 2002). Nevertheless, research examining team processes has actually shown that while the timing of such processes and the patterns of emergence may differ, both electronic and face-to-face teams still converge on similar outcomes, such as team mental models (e.g., McComb, Kennedy, Perryman, Warner, & Letsky, 2010). Everything conveyed in a research article is in writing, and it is via this method of communication by which authors communicate with editors and reviewers (and if successful, the public). Whereas authors who have media-rich, verbalized conversations may generate great ideas together, if they are not able to translate these insights into logical, persuasive, and theory-backed prose, then any benefit to be gained via media-rich communication may be lost. With e-mail, researchers are forced to put their ideas into words, which can be carefully scrutinized by fellow coauthors. Such a process may be to the benefit of the paper (and authors’ responses to reviewers). Further, given the very busy schedules of most researchers, and the distribution of coauthors across time zones, asynchronous communication may be maximally efficient for moving papers forward in time. Such a communication form also affords coauthors time to deliberate on each other’s ideas, in much the same way reviewers would.

Recommendations

On the basis of these findings, we recommend certain practices when developing a team of authors and conducting research together. While these are by no means a guarantee of paper success, we believe that certain practices would be beneficial for any manuscript. First, researchers should think about reaching out to potential coauthors outside of their own institutions. Certainly drawing on the local talent pool brings advantages, such as more direct channels of communication and the possibility for frequent “sit-down” meetings. However, reaching out to the broader population of researchers might allow for more targeted recruitment of scholars. This might include experts in one’s own area, experts in complementary areas, or successful researchers with reputations of being conscientious coauthors. We recommend that researchers think about expanding their teams beyond their department’s halls and think globally when drawing on high potential talent outside of their own university.

Similarly, although conventional wisdom suggests that rich modes of communication, such as Skype and face-to-face contact, will improve the quality of their research, our findings suggest that scholars should consider using electronic forms of communication to improve their manuscript’s quality. E-mail communication allows for the frequent exchange of ideas, while also encouraging collaborators to make and defend their points via the same medium of their intended product. It also allows for the cutting and pasting of the text in question (or alternatively, the attachment of relevant documents with written comments and edits) and allows the e-mail writer more time to think and structure his or her arguments. We therefore recommend that authors consider using e-mail often, but strategically and thoughtfully, to improve the quality of their manuscripts.¹¹

Similarly, authors should be strategic in how they seek their feedback. Friendly reviews are required by many journals, including JOM, before submission. Rather than seeking feedback from colleagues who may give little criticism and offer only praise, researchers should strategically seek out individuals who are likely to give constructive criticism, have enough editorial experience to anticipate common reviewer concerns, and will provide feedback in a timely manner. In fact, explicit instructions to these friendly reviewers should encourage them to be as “critical as humanely possible” in order to help counter the potential bias associated with the review not being anonymous. This may require reaching beyond one’s network of close colleagues in order to tap expertise that will align with that of a potential reviewer.

Finally, researchers will benefit from both persistence and resilience (Feldman, 2004). There are a number of ways to improve manuscripts such that they find a home. Continuing to develop and refine a manuscript, rather than shopping around an identical version of the manuscript to different journals, not only improves its chances of finally being accepted but also encourages higher-quality scholarly work. Researchers would likely see an improvement in the quality of their manuscripts the more they solicit strategic reviews, receive feedback, and make constructive changes to their manuscripts. Although we acknowledged above the possibility for inconsistent feedback across reviewers (i.e., by our pointing out that a friendly reviewer and journal reviewers are unlikely to bring up the same points of criticism), research shows that at the career level of analysis, researchers are reliably evaluated(Hollenbeck & Mannor, 2007). The more authors work toward reconciling differences in reviewer opinions in subsequent versions of the paper, the higher the likelihood for interrater agreement among subsequent reviewers. Assuming such revisions increase the quality of the paper, the probability of success goes up with each subsequent submission.

Moving Forward

As we mentioned at the onset, a principal goal of this editorial is to help “pull back the curtain” on author practices and manuscript success/quality. Given the vast amount of time authors spend pursuing and navigating scholarly pursuits, we encourage additional and deeper investigations using a variety of different approaches in an attempt to develop a science around our science. In the case of this editorial, our results were largely descriptive (i.e., not hypodeductive), and our sample size and the nature of our variables did not lend themselves to complex inferential statistics. Future studies might consider the effect of author team characteristics and decisions across multiple journals. Findings at one journal might be reflective of field-specific characteristics that are not generalizable across different areas within organizational research (e.g., journals that appeal more to management vs. those that appeal more to industrial and organizational psychology).¹² Similarly, we also believe that future research should predict and test, across different journals, differences in the decisions and characteristics of micro versus macro/strategy (as well as interdisciplinary) research teams. While certain factors might similarly predict manuscript quality across these two domains, there may likely be field-specific differences that might improve manuscript quality in one but not the other. Finally, we argue that future research should draw upon the groups and teams literature to identify and examine different group characteristics and processes that might encourage higher-quality manuscripts. Such investigations might also consider a larger set of team characteristics than we did here. Perhaps most notably, such studies might leverage extant team theory to examine meaningful interactions among characteristics to understand manuscript quality in a much deeper and more nuanced manner (e.g., for larger author teams, it may be the case that certain development practices take on increased importance). Together, such investigations not only stand to build knowledge of research team functioning and effectiveness but also aid us all, as a community of scholars, in identifying and implementing strategies aimed at maximizing the impact of our research.

Footnotes

Notes

References

Aguinis

Boyd

B. K.

Pierce

C. A.

Short

J. C.

2011. Walking new avenues in management research methods and theories: Bridging micro and macro domains. Journal of Management, 37: 395-403.

Bauer

T. N.

2009. The Journal of Management review issue: Celebrating 35 years. Journal of Management, 35: 1297-1301.

Berka

Olien

Rogelberg

Rupp

D. E.

Thornton

M. A.

in press. The science of science: An inductive exploration of manuscript quality and publication success in small research teams. Journal of Business and Psychology.

Bono

J. E.

McNamara

2011. From the editors: Publishing in AMJ. Part 2: Research design. Academy of Management Journal, 54: 657-660.

Colquitt

J. A.

Zapata-Phalen

J. A.

2007. Trends in theory building and theory testing: A five decade study of the Academy of Management Journal. Academy of Management Journal, 50: 1281-1303.

Corley

K. G.

Gioia

D. A.

2011. Building theory about theory: What constitutes a theoretical contribution? Academy of Management Review, 36: 12-32.

Cropanzano

2009. Writing nonempirical articles for Journal of Management: General thoughts and suggestions. Journal of Management, 35: 1304-1311.

Feldman

2004. The devil is in the details: Converting good research into publishable articles. Journal of Management, 30: 1-6.

Fulmer

I. S.

2012. Editor’s comments: The craft of writing theory articles. Variety and similarity in AMR. Academy of Management Review, 37: 327-331.

10.

George

2012. From the editors. Publishing in AMJ for non-U.S. authors. Academy of Management Journal, 55: 1023-1026.

11.

Hillman

2011. Editor’s comments: What is the future of theory? Academy of Management Review, 36: 607-609.

12.

Hollenbeck

J. R.

Mannor

M. J.

2007. Career success and weak paradigms: The role of activity, resiliency, and true scores. Journal of Organizational Behavior, 28: 933-942.

13.

Ilgen

D. R.

2007. Citations to management articles: Cautions for the science about advice for the scientist. Academy of Management Journal, 50: 507-509.

14.

Judge

T. A.

Cable

D. M.

Colbert

A. E.

Rynes

S. L.

2007. What causes a management article to be cited—article, author, or journal? Academy of Management Journal, 50: 491-506.

15.

Katz

J. S.

Martin

B. R.

1997. What is research collaboration? Research Policy, 26: 1-18.

16.

Kepes

McDaniel

M. A.

Brannick

M. T.

Banks

G. C.

(2013). Meta-analytic reviews in the organizational sciences: Two meta-analytic schools on the way to MARS (the Meta-Analytic Reporting Standards). Journal of Business and Psychology, 28: 123-143.

17.

Lee

Bozeman

2005. The impact of research collaboration on scientific productivity. Social Studies of Science, 35: 673-702.

18.

LePine

J. A.

Wilcox-King

2010. Editors’ comments: Developing novel theoretical insight from reviews of existing theory and research. Academy of Management Review, 35: 506-509.

19.

MacKinnon

D. P.

Coxe

Baraldi

A. N.

2012. Guidelines for the investigation of mediating variables in business research. Journal of Business and Psychology, 27: 1-14.

20.

Mathieu

J. E.

Tannenbaum

S. I.

Donsbach

J. S.

Alliger

G. M.

2014. A review and integration of team composition models. Journal of Management, 40: 130-160.

21.

Mayer

K. J.

Sparrowe

R. T.

2013. Integrating theory in AMJ articles. Academy of Management Journal, 56: 917-922.

22.

McComb

Kennedy

Perryman

Warner

Letsky

2010. Temporal patterns of mental model convergence: Implications for distributed teams interacting in electronic collaboration spaces. Human Factors, 52: 264-281.

23.

Okhuysen

Bonardi

J.-P.

2011. Editors’ comments: The challenges of building theory by combining lenses. Academy of Management Review, 36: 6-11.

24.

Park

Spitzmuller

DeShon

D. P.

2013. Advancing our understanding of team motivation: Integrating conceptual approaches and content areas. Journal of Management, 39: 1339-1379.

25.

Ployhart

R. E.

Ward

A. K.

2011. The “quick start guide” for conducting and publishing longitudinal research. Journal of Business and Psychology, 26: 413-422.

26.

Rogelberg

S. G.

Adelman

Askay

2009. Crafting a successful manuscript: Lessons from 131 reviews. Journal of Business and Psychology, 24: 117-121.

27.

Shapiro

D. L.

Furst

S. A.

Spreitzer

G. M.

von Glinow

M. A.

2002. Transnational teams in the electronic age: Are team identity and high performance at risk? Journal of Organizational Behavior, 23: 455-467.

28.

Short

2009. The art of writing a review article. Journal of Management, 35: 1312-1317.

29.

Sparrowe

R. T.

Liden

R. C.

Wayne

S. J.

Kraimer

M. L.

2001. Social networks and the performance of individuals and groups. Academy of Management Journal, 44: 316-325.

30.

Starbuck

W. H.

2005. How much better are the most prestigious journals? The statistics of academic publication. Organization Science, 16: 180-200.

31.

Wolff

Moser

2009. Effects of networking on career success: A longitudinal study. Journal of Applied Psychology, 94: 196-206.