We Love Our Peers!

Two years ago, the American Journal of Sports Medicine and SAGE, our publisher, invited orthopaedic surgery residents and sports medicine fellows from the Chicago area to attend a focus group session exploring how young surgeons-in-training acquire new information. While the opportunity to share their experience with journal publishing professionals may not have been as enticing as the steak dinner that accompanied it, our guests were nevertheless happy to describe their learning process.

The comments of the trainees were consistent and enlightening. We learned that most of their reading was driven by an acute need to know. Rarely did they simply sit down and leaf through a textbook or journal. Instead, they usually conducted a focused search to prepare for an upcoming surgical case or discuss a specific topic in conference. Most of them used a general internet search engine such as Google, rather than a medical database, to gather references on the topic at hand. Within seconds, Google’s magical software would generate an endless list of information sources. Since their available time was not infinite, they selected articles from what they considered the top peer-reviewed journals to download and read.

As an editor, I was heartened to learn that this rising generation of cyber-savvy surgeons still valued the peer-review process as a means of evaluating and validating new medical knowledge. Since its introduction a few hundred years ago, peer review has been recognized as a valuable if imperfect screening tool in medical and scientific publishing. Despite its acknowledged limitations, there is widespread trust in the peer-review process. The Editor-in-Chief of the Journal of Pediatrics has likened peer review to democracy, which Sir Winston Churchill famously defined as “the worst form of government, except for all those other forms that have been tried from time to time.” ¹

AJSM is profoundly grateful for the inestimable contributions of its peer-review team to the quality of its content. Each December we publish their names and a brief thank-you note, which constitutes rather meager compensation for the incalculable hours they devote to their vocation. While some individuals indeed “burn out” over time or find their available hours consumed by other obligations, I am repeatedly impressed by the conscientiousness and loyalty of the many senior reviewers and editorial board members who have been meticulously vetting AJSM submissions for decades. Nevertheless, a seemingly inexorable expansion in the number and variety of our submissions means that we are always on the lookout for sports medicine professionals willing and able to produce thoughtful, detailed, and knowledgeable reviews. They come to us in several ways: Often a fellowship director, PhD advisor, mentor, or senior colleague will recommend a bright, energetic candidate. Additional individuals who impress the editors with their publications or meeting presentations are invited to review. Sometimes the names of pertinent experts are culled from the reference lists of submissions. Other potential reviewers simply volunteer.

The authors of submitted articles provide an additional source of reviewers for AJSM. Since introducing online review in 2002, our journal has invited authors to suggest possible reviewers for their work or, conversely, those they wish to exclude. While few authors list exclusions, we always accede to these requests. More commonly, authors do suggest possible reviewers, and we make it a point to invite at least one of these individuals, unless they are unavailable or affiliated with the authors themselves. Although many of the recommended people are already in our database, these suggestions have brought many excellent new reviewers to our attention.

Several studies, conducted primarily in general medicine and pediatrics journals, have compared the work of author-recommended reviewers with that of editor-selected ones.^6,8-11 Using a validated instrument to judge the excellence of the reviews, these studies have consistently found similar quality ratings of the work of both groups of reviewers.^8-11 They have also regularly noted that author-nominated reviewers are more likely to recommend acceptance. Although this may lead one to conclude that author-recommended reviewers are more lenient or sympathetic to the plight of the authors, another possible explanation is that they are more expert on the specific topic of the paper under scrutiny than the editor-selected reviewers. While we have not conducted a formal analysis comparing these 2 groups of reviewers for AJSM, I have noted that it is not unusual for an author-nominated reviewer to recommend rejection of the paper. In any case, the editors are aware of the provenance of each reviewer and remain alert for any review that seems uncritically effusive.

Editors are always searching for ways to improve the quality of their journals’ manuscript reviews. One means to that end is selectively assigning more manuscripts to reviewers who produce higher quality evaluations. The editors of the Annals of Emergency Medicine (AEM) have studied peer review extensively and have reported their experience with such a system. Since 1994, AEM has evaluated all their reviews on a 5-point scale, in which a level 1 review is considered “unacceptable effort and content” and a level 5 defined as “exceptional, hard to improve.” Over a 14-year period ending in 2008, the average quality score assigned to each review was 3.8. ² In 2003, AEM divided its reviewer pool into 3 tiers based upon their average quality score and their responsiveness to review invitations. ⁵ Every 6 months since, reviewer performance has been evaluated and tier assignments adjusted accordingly. Although reviewers from all 3 tiers are still utilized, editors have been instructed to assign submissions preferentially to the 25% of reviewers who constitute the top tier. When they reviewed their experience, the editors found that the principal result of this intervention was a reduction in the percentage of late reviews from 32% in 2003 to 13% in 2009; interestingly, review quality was unchanged. ⁵

Although AJSM does not have a formal tier system, the editors assign all reviews a quality rating on a 3-letter scale. Other reviewer metrics are tracked separately, including the average number of days required to complete a review, the number of review invitations accepted, and any instances of assignments accepted but never completed. Reviewers with the highest quality and efficiency ratings are preferentially invited to evaluate new submissions, whereas those with poor performance by either of these standards are avoided and periodically culled from the database This tends to concentrate the assignments within a subset of the reviewer team, as evidenced by the substantial number of individuals beyond the AJSM editorial board who complete 5 or more reviews during a calendar year and are honored as principal reviewers in the next volume of the journal.

Another approach for improving the quality of manuscript reviews is to provide additional training for the reviewer team. Strategies employed by different journals have included structured workshops,^3,10,12 self-guided training modules, ¹⁰ and individual mentoring by senior reviewers. ⁷ For its part, AJSM stages an annual educational seminar for its reviewers during the summer meeting of the American Orthopaedic Society for Sports Medicine. Each seminar usually consists of two 45-minute presentations by editors, editorial board members, or other experts on a focused topic in study design or analysis. Although other competing educational and recreational activities abound, a large and enthusiastic audience is always in attendance. In addition to this yearly event, all reviewers are informed of the initial editorial decision on manuscripts that they review and given access to the other reviews of the paper.

Available studies of the ability of reviewer education initiatives to improve review quality have demonstrated surprisingly unimpressive results. On the positive side, a workshop conducted by a statistician and 2 pulmonologists for the Chinese Journal of Tuberculosis and Respiratory Diseases resulted in measureable short-term improvements in the review quality of the 45 attendees. ¹² On the other hand, the editors of AEM have shown little benefit from reviewer education in controlled studies of structured workshops ³ and individual mentoring. ⁷ Even though AEM workshop attendees thought the experience would improve their performance, objective comparison of their reviews before and after the session showed minimal change. ³ Another 3-armed study conducted by the BMJ, which compared reviewers who attended a workshop or completed a self-taught training package with controls, found that both teaching methods effected a small but significant immediate improvement in review quality. ¹⁰ Unfortunately, this improvement was no longer apparent when the subjects’ work was re-evaluated 6 months later.

In light of this evidence, are we to believe that reviewer education is futile? Are good reviewers born rather than made? I think that both of these alternatives are true. One of my duties at the University of Chicago is to conduct the monthly orthopaedic journal club. In an experience that is undoubtedly replicated in most orthopaedic training programs, residents must analyze and discuss in detail articles of variable quality from a range of major orthopaedic journals. As the residents progress through the 5-year program, improvement in the comprehensiveness, sophistication, insightfulness, and conciseness of their analyses is easy to see. This leads me to believe that the efficacy of education is most obvious in the early stages of the development of future reviewers. By the time surgeons and other sports medicine professionals complete their training and are eligible to become journal reviewers, the effect of continuing education, while still valuable, is more incremental and may be difficult to measure with available instruments, whose ability to document further improvements may be limited by a ceiling effect.

However, while a strong educational foundation is vital for performing an outstanding review, it is not sufficient. The best reviewers seem to share behavioral traits such as punctuality, fairness, diplomacy and, yes, compulsive attention to detail, that are difficult to impart to a fully trained professional via educational initiatives. We can probably thank their parents and early role models for instilling these qualities. For these remarkable individuals, reviewing is an obligation or even a calling to which they always apply their best efforts. Such reviewers understand the importance of their undertaking to colleagues and patients worldwide, and produce the type of review they would want to receive as authors themselves. Periodically, we observe individuals who have the acknowledged expertise to fashion an outstanding review, yet fail to fulfill their potential. Such examples only serve to illuminate the critical combination of nature and nurture required to produce a top-notch journal reviewer. The editors of AJSM are grateful that so many of these exceptional individuals choose to review manuscripts for our journal. Because of their hard work and dedication, young members of the orthopaedic sports medicine community, such as the participants in our focus group, continue to believe in the value of the peer-review process.