Abstract
Background:
Reports indicate that there are gender-based differences in aspects of the peer-review process.
Methods:
This is an analysis of editorial board members' reviews of original research submissions based on gender using the web-based management program, Editorial Manager, from January 1, 2002, through December 31, 2008. We evaluated recommendations of editorial board members for acceptance/rejection using a four-tier system, agreement with editor's final decision, turnaround time from review request to submission, and editors' grades of reviews on a 5-point scale. We evaluated the performance of editorial board members with advancing tenure, seeking trends in recommendations over time.
Results:
We included 6062 manuscript reviews representing 5958 manuscripts; 67% were assigned to male editorial board members and 33% to females. There were 38 editorial board members (25 men, 13 women) with tenure duration from 2 to 4.9 years, and 3 editors (2 men, 1 woman) serving 7, 7, and 6 years, respectively. Women were less likely to accept or accept with minor revisions than men (p < 0.003). Median turnaround times were 14 (0–55) days for women and 10 (0–33) days for men (p < 0.001). The editors' grades assigned to women were more often than men's grades in the very good to exceptional category (p < 0.0001). There was no difference based on gender, with approximately 73% decision congruence overall with the editors' final decisions. Men rejected more manuscripts than did women with advancing tenure on the editorial board (p < 0.0001).
Conclusions:
There are differences based on gender of editorial board members' recommendations about manuscript triage, turnaround time, and editors' grades assigned. Overall, however, these differences do not affect editors' ultimate decisions about manuscript publication.
Introduction
Women in the United States are entering the medical field in greater numbers than ever before. Currently, women represent >75% of residents in obstetrics and gynecology training. 1 Women constituted 50% of academic appointees in 2004 2 and, over time, have been found in increasing numbers in the highly visible and prestigious appointments to editorial boards of major medical journals. 3 Recently, the National Academy of Sciences and Institute of Medicine called for “reasonable representation of women on editorial boards and in other significant leadership positions,” 4 and in response to this, Jagsi et al. 5 evaluated the representation of women on the editorial boards of 16 major biomedical journals over a 35-year period. Included in their evaluation was the editorial board of Obstetrics and Gynecology. In that analysis, 371 of 3287 (11.7%) editorial board members were women, with an increasing trend toward greater female representation over time from 1.4% in 1970 to 16.0% in 2005. Nonetheless, women remain the minority on all the editorial boards studied.
Obstetrics and Gynecology is a peer-review journal that is the official publication of the American College of Obstetricians and Gynecologists (ACOG). It was founded in 1953, and its original editorial board compromised 8 members, selected to represent each of the College Districts in existence at the time. As ACOG has undergone metamorphoses over time by adding or dividing Districts, so has the editorial board changed. Not only has it expanded to maintain representation from each of the ACOG districts, currently 10 in number but it has also adopted a policy of involving newly elected or junior faculty members and sought to involve members of the international academic community. Expansion has occurred in order keep pace with the demands for reviews of an ever-increasing number of manuscript submissions. The editorial board of Obstetrics and Gynecology now includes so people, 6, or 30%, of whom are women. Editorial board members are selected by the editors of the journal and usually serve for a 4-year period. Each month, the editorial board members are assigned eight manuscripts to review.
Differences in reviewer and editor behavior based on gender have been studied infrequently. 6,7 Although there is some indication that female reviewers reject manuscripts at proportionally higher rates than do male reviewers, 6 the overall influence of gender on the eventual decisions about publication are not affected. With a growing number of women on the editorial board of Obstetrics and Gynecology, we sought to evaluate differences in their review behavior compared with the male editorial board members.
Materials and Methods
This study was approved by the Institutional Review Board of ACOG.
Data source
Using the Obstetrics and Gynecology manuscript tracking database system, Editorial Manager, we conducted a retrospective cohort study of all reviews by editorial board members of original research submitted for publication in Obstetrics and Gynecology during the period January 1, 2002, to December 31, 2008. Our starting point was based on implementation of the web-based manuscript tracking system. By the time of the study, the manuscripts had been processed and final decisions had been made.
Subjects
We collected the review data from the editorial board members of Obstetrics and Gynecology for the period of the study. At the inception of the investigation, there were 16 editorial board members. This expanded in 2006 to 19 editorial board members and again in 2008 to 20 editorial board members. There were 3 senior editors appointed to Obstetrics and Gynecology during the study period; the editor-in-chief and deputy editors are men and were appointed in 2001; the associate editor is a woman and was appointed in 2002.
The following data were extracted from the database and tabulated by gender and sorted by year: research articles received from 2002 to 2008, the editorial board member's recommendations based on a four-tier system, turnaround times from the date the manuscript was sent for review until the requested review was received, editor grades of editorial board member's reviews, and congruence with the editors' final decisions (acceptance or rejection). Board member recommendations were recorded in one of the following categories: accept as is, accept with major revisions, accept with minor revisions, or reject. Editors graded reviews on a scale of 100–20 (with 100 being exceptional, 80 being very good, 60 being good, 40 being below average, and 20 being unacceptable).
Data management
Our unit of measure was a review of an original research manuscript. This was done because the data included a wide range of member tenure times as well as incomplete tenures. Additionally, some manuscripts were subjected to duplicate reviews by the same editorial board member or were assigned to two editorial members simultaneously.
To evaluate decision congruence between reviewer recommendation and the final editorial decision of acceptance or rejection, reviewer categories of outright acceptance, acceptance with minor revision, and acceptance with major revisions were collapsed into a single accept category so that the comparison could be made.
To evaluate the performance of the editorial board members over time, we elected to divide their tenure into 6-month blocks. The choice of 6 months was deemed as a reasonable time between the minimum of weekly manuscript submissions and a year, which would limit a longitudinal evaluation. For the purposes of evaluating the correlation between the editorial board members' accept or reject recommendations for a given manuscript to the editors' final disposition, we limited the period of analysis to the first eight 6-month blocks of time, as inclusion of the 6-month blocks of time beyond the eighth caused diffusion of available data.
We hypothesized that there would be no difference in the behavior of participants in Obstetrics and Gynecology's peer-review process in terms of frequency of recommendations for acceptance or rejection and timeliness of response or in the editors' average grades assigned to each editorial board member. We hypothesized that there would be no difference based on gender between the editorial board members' recommendations and the editors' final decision. We also hypothesized that with increasing duration of service to Obstetrics and Gynecology, the frequency of rejections of manuscripts would also increase and that this phenomenon would be gender neutral.
Statistical analysis
Editorial board member recommendations and manuscript acceptance rates were analyzed using chi-square tests for independence, as was evaluation of gender differences in average reviewer ratings. Median turnaround times were assessed with the use of the two-sided Wilcoxon rank sum test. Overall agreement between reviewer recommendation and editor's final decision was analyzed by the kappa coefficient. The Cochran-Armitage test for trend was applied to evaluate any longitudinal pattern over time for rejection rates as well as for the evaluation of correlation of the editorial board members' reviews and the editors' final decisions about acceptance or rejection of a manuscript. Conventional levels of significance (p < 0.05) were used in all hypothesis testing. 8
Results
Over the 7-year period, 6062 reviews were completed, including 5859 manuscripts that were reviewed once by a board member. There were 109 manuscripts that had two or more reviews by editorial board members, including 58 that were reviewed more than once by the same editorial board member; these represented <1% of the total reviews. There were 38 editorial board members who started their tenure within the 2002–2008 period; of these, 25 were men and 13 were women. The tenure lengths of these individuals varied from 2 to 4.9 years. Twenty of the 38 members (53%) had not yet completed their terms, and of these 20, 12 had completed less than one half of the duration of their term. The completed portion of tenures varied from 6 months to 4.9 years, with 15 members having completed at least 3.5 years. Given the extent of varied and uncompleted tenures, evaluation of the data was based on reviews. Individual member variations, particularly in behavior and characteristics over time, are not accounted for in the statistical results.
We found a difference in editorial board member recommendations for manuscripts based on gender (p < 0.003) (Table 1). Manuscripts reviewed by women were less likely to receive either accept or accept with minor revision recommendations than were manuscripts reviewed by men. Partitioning of the overall resulting chi-square statistic of 14.24 by means of the cell chi-square statistics indicate that the difference between men and women in the outright acceptance category alone contributed 43%. The outright acceptance and acceptance with minor revisions categories accounted for 79% of the overall statistic.
Pearson chi-square 14.243, p < 0.003.
The median turnaround times between assignment and return of review were statistically different. The male editorial board members took 10 (0–36) days (4072 manuscripts), whereas the female editorial board members took 14 (0–55) (1990 manuscripts) (p < 0.0001). The editors' grades of the editorial board members' reviews demonstrated that reviews by women were more likely to be assigned grades of very good to exceptional than were the reviews by men. (p < 0.0001) (Table 2). Overall, <1.5% of the review grades fell in the lower two categories. Approximately 50% of the grades were in the good (grade of 60) category, and almost 50% were in the very good (grade of 80) to exceptional (grade of 100) categories.
Pearson chi-square 60.874, p < 0.0001.
Our evaluation of decision congruence between the editorial board members and the editors demonstrated no differences based on gender, with 72% congruence overall. The kappa statistic for all reviews regardless of gender was 0.40 (95% confidence interval [CI] 0.37–0.42). Stratification based on gender revealed the following: for female editorial board members, the kappa statistic was 0.40 (95% CI 0.36–0.44), and for male editorial board members, 0.40 (95% CI 0.37–0.43).
The longitudinal evaluation using 6-month time blocks of editorial board members' performance using the Cochran Armitage test for trend suggested that reviews by male members were more often recommended for rejection with increasing tenure. 8 The same was not true for reviews by women (p < 0.0001 for men, p = 0.11 for women) (Table 3). A similar pattern was observed for the correspondence of the review recommendation and the editor's final decision based on the Cochran-Armitage trend test, with the percent of agreement increasing for reviews by men over time (p < 0.0002) but not for reviews by women (p < 0.86). 8
Discussion
We have noted the increase in female representation on the editorial board of Obstetrics and Gynecology over the last decade. Editorial board memberships are important positions that afford the individual the opportunity to guide the underlying direction of the journal and thereby potentially impact the tenor of research being conducted within our scientific communities. Editorial board members make critical decisions and dictate policies that govern the dissemination of scientific information in such ways as creating rules for public protection from release of information that has not been subjected to scientific scrutiny, generation of policies that address potential conflict of interests of researchers, and writing policies about advertisements 9 and Internet dissemination of information. Jagsi et al. 5 commented on the importance of awareness of these considerations for the evolving composition of these editorial board appointments to involve more women and forces affecting that distribution as time passes.
Kennedy et al. 10 evaluated the percentages of women on editorial boards of general and specialty medical journals in comparison with the numbers of women physicians in the journals' respective specialties. Compiling the number of women editors, deputy editors, assistant editors, and members of editorial boards of 12 major journals and comparing this information to the percentage of women physicians in the journals' specialties as published by the American Medical Association from data in 1999, they found parity in only 41% of specialties represented, and only one journal had a greater percentage of women on its editorial board than there were women in the specialty. Obstetrics-gynecology was among that 41%.
Gilbert et al. 6 determined that whereas gender bias may exist in the evaluation of scientific manuscripts, it does not appear to affect the ultimate publication rates. Their detailed study of the gender of editors, reviewers, and corresponding authors of articles published in the Journal of the American Medical Association in 1991 revealed no apparent effect of gender differences on the final outcomes of the review process or acceptance for publication. This has been demonstrated as well in peer review of a gender-related study. 7 In the latter study, men and women reviewers were found to be impartial in their judgments of a standardized gender-related research report and in their assessment of a sociopolitically sensitive conclusion where two controversial alternatives were described as the underlying reasons for the difference in the anticipated income of men and women.
We were concerned about the possibility of publication bias based on the gender of the editorial board members of Obstetrics and Gynecology. Publication bias exists in many forms. The biomedical peer review literature tends to represent a tendency to publish based on the direction or the strength of research findings. 11,12 This bias exists in three stages of the peer review process. 13 Gender bias, the differential handling of a manuscript based on the gender of the author, editor, or reviewer, may be a component of the second, or publication, phase. Some literature focuses on the association between reviewer gender and reviewer recommendations, 14,15 and we sought to define the behavioral differences of the editorial board members of this journal using data from its web-based manuscript review system. This system was implemented by Obstetrics and Gynecology in 2002 and provided a ready resource.
We found that although female editorial board members were less likely to recommend acceptance without revisions and acceptance with minor revisions than their male counterparts, their ultimate recommendations for acceptance or rejection were in accord with the editors' final decisions. We also found that with increasing tenure, male editorial board members appeared more likely to recommend rejection, unlike female editorial board members, whose patterns of recommendations remained essentially stable. To what this tendency in the male editorial board members can be attributed is unknown, although indirect feedback is provided at the annual editorial board meeting for this journal. Where target acceptance and rejection rates are discussed may influence review behavior. It is unlikely that direct education for improving the quality of manuscript reviews is responsible, as the need to provide this to our editorial board members did not arise. Further, there is evidence in at least one study that direct education for the purpose of improving the quality of manuscript reviews is ineffective. 16
We were not surprised to find that the vast majority of the editors' grades for our editorial board members occupied the upper three strata, from good to exceptional. 17 Selection for participation in the editorial board for this journal is based on past experience with reviews for it, and individuals with scores routinely in the lower two grading strata are rarely selected for participation.
We acknowledge the limitations of this evaluation, which includes the relatively recent adoption of the web-based editorial management system in 2002, the small numbers of editorial board members, and the short time line for study conduct. Because the constitution of the editorial board reflects a rotating assignment usually of 4-years duration, many of the reviewers were still in the midst of their tenure at the time this investigation was undertaken. This required that we manage the review data considering each review as independent; we acknowledge that, in reality, this may not have been the case. This was an unavoidable circumstance, however, that could have created some ambiguities in evaluating the longitudinal performance of the editorial board members.
There is general acceptance of the importance of the peer review process for promoting scientific integrity. Continual questioning and evaluation are important aspects to its success. Gender bias and other gender-related issues are deserving of attention and periodic evaluation, and a consciousness of the importance of placing competent women in these prestigious positions is imperative in ensuring adequate and fair representation and maintaining high standards of quality for such journals as Obstetrics and Gynecology. Providing mentorship to women to provide them access to these types of positions will be a key factor in the achievement of gender balance on editorial boards of peer-reviewed journals. 18
Conclusions
We found differences based on gender of editorial board members' recommendations about manuscript triage, turnaround time, and editors' grades assigned in this publication focused on women's health. We were reassured to find that despite these differences, there was no effect on the editors' ultimate decisions regarding manuscript publication.
Footnotes
Disclosure Statement
The authors have no conflicts of interest to report.