Abstract
Background:
The peer review system faces increasing strain from rising manuscript volumes, reviewer fatigue, and well-documented interreviewer disagreement. Large language models (LLMs) have shown potential to support the peer review process, but their ability to replicate editorial decisions at high-impact medical journals and their utility as manuscript screening tools remain unknown.
Purpose:
To compare the agreement between an LLM and the final editorial decision on manuscripts submitted to the American Journal of Sports Medicine and to evaluate the potential of LLMs as a manuscript screening tool.
Study Design:
Cross-sectional agreement study.
Methods:
Fifty-four manuscripts randomly selected from submissions to the American Journal of Sports Medicine (September 2024–October 2024) were reviewed by a locally deployed LLM (Ministral 3 14B; Mistral AI) using a standardized prompt. The artificial intelligence (AI) produced a categorical recommendation (reject, cascade, revision, or accept) and a numerical score (0-100) for each manuscript. Agreement with the final editorial decision was assessed by Cohen kappa (4-category model) for pooled human reviewers (n = 139 reviews) and the AI (n = 54). Screening performance was evaluated by positive predictive value (PPV), sensitivity, and specificity.
Results:
Pooled human reviewers demonstrated fair agreement with the final decision (κ = 0.181 [P < .001]; 42.4% agreement), while the AI demonstrated slight, nonsignificant agreement (κ = 0.126 [P = .099]; 37.0% agreement). The AI recommended revision for 61.1% of manuscripts, of which 72.7% were ultimately rejected or cascaded, demonstrating systematic “revision bias.” When the AI recommended rejection, 54.5% of those manuscripts were ultimately rejected and 27.3% were cascaded; when the AI recommended cascade, 50% were rejected and 50% were cascaded. However, when the AI recommended rejection or cascade (n = 21), 90.5% received a final decision of rejection or cascade (PPV, 90.5%; specificity, 81.8%). Manuscripts with an AI score <70 were rejected or cascaded 88.0% of the time (PPV, 88.0%).
Conclusion:
AI cannot replicate the nuanced judgment of human peer reviewers at a high-impact sports medicine journal. When AI recommended rejection or cascade, 90.5% of manuscripts received that final decision (descriptive PPV, 90.5%; 95% CI, 71.1%-97.3%), suggesting potential utility as an exploratory first-pass screening tool warranting further validation in larger cohorts. However, AI could not reliably distinguish manuscripts destined for outright rejection from those that would be cascaded to a sister journal—an important limitation for editorial triage applications.
Keywords
Peer review remains the cornerstone of scientific publishing, yet the system is under increasing strain. The volume of scholarly manuscripts continues to rise, while the pool of willing and qualified reviewers has not kept pace. A global survey of >11,000 researchers found that approximately 42% felt overwhelmed with existing commitments, while 70% declined review invitations because manuscripts fell outside their area of expertise. 12 Editors on average contact 2.4 to 3.6 potential reviewers, with some journals having to solicit upward of 10 potential reviewers before securing an agreement to review a single manuscript.4,12 This growing imbalance has led to delays in the editorial process, increased reviewer fatigue, and concerns about the sustainability of the traditional peer review model.4,11,12
Compounding the challenge of reviewer availability is the well-documented variability in peer review itself. Studies across medical and scientific disciplines have consistently shown poor interreviewer agreement, with kappa values typically ranging from 0.08 to 0.28.3,14 Reviewers frequently disagree on the merits of the same manuscript, a finding that has raised fundamental questions about the consistency and reliability of the process. 14 Editors are often faced with rendering decisions on manuscripts with divergent reviews. In the context of high-impact journals that reject the majority of submissions, even small inefficiencies in the review process can meaningfully affect authors and editorial resources.
The emergence of large language models (LLMs) has generated considerable interest in their potential to support the peer review process.2,5,6 A landmark study evaluating GPT-4’s feedback on >3000 Nature family journal papers and >1700 submissions to the International Conference on Learning Representations (ICLR) found that the overlap in feedback between artificial intelligence (AI) and human reviewers (30.9% for Nature, 39.2% for ICLR) was comparable to that between 2 human reviewers (28.6% and 35.3%). 10 Additionally, more than half of the researchers surveyed found the AI feedback helpful, and >80% rated it as more useful than feedback from at least some human reviewers. 10 However, LLMs consistently award higher grades than human reviewers and rarely recommend rejection.15,17
Despite this growing body of evidence, most existing studies have focused on AI as an independent reviewer or have been conducted in fields outside of clinical medicine. Few studies have directly compared the agreement of AI-generated recommendations with final editorial decisions using real-world submissions to high-impact medical journals. Furthermore, none have specifically evaluated AI's potential as a screening or triage tool—that is, its ability to identify manuscripts unlikely to survive the peer review process—rather than as a replacement for human reviewers. Bauchner and Rivara 2 proposed that LLMs could be used to assist editors in triaging manuscripts, but empirical evidence for this application remains absent.
The purpose of this study was twofold. The primary aim was to evaluate the agreement between an LLM and the final editorial decision on manuscripts submitted to a high-impact sports medicine journal, as compared with the agreement between human peer reviewers and the same final decision. The secondary aim was to assess the potential utility of AI as a manuscript screening tool by evaluating whether AI recommendations and numerical scores could predict manuscripts that would ultimately be rejected or cascaded to a sister journal. We hypothesized that while AI would demonstrate lower categorical agreement with the final editorial decision as compared with human reviewers, it would show utility as a screening tool by reliably identifying manuscripts ultimately rejected or cascaded.
Methods
Study Design
This cross-sectional agreement study compared AI-generated peer review recommendations with the final editorial decision on manuscripts submitted to the American Journal of Sports Medicine (AJSM). The study was designed to evaluate categorical agreement (primary aim) and screening utility (secondary aim) of an AI reviewer relative to human peer reviewers.
Manuscript Selection
Fifty-four manuscripts were randomly selected from a total of 107 submissions to AJSM during September and October 2024 using a random number generator applied to the sequential submission log maintained by the editorial office. Manuscripts were eligible if they had completed peer review and received a final decision by the time of data collection. Manuscripts were excluded if the final decision was not yet rendered or if the submission was subsequently withdrawn. We contacted corresponding authors, who provided consent for their manuscripts to be reviewed by AI after being informed that the AI review would be conducted in a closed (nonpublic) system and not be used in the decision-making process. Manuscripts were included regardless of study design or topic. Reviewers were selected by the assigned editor, taking into account the topic of the manuscript and the reviewer's stated areas of expertise. For each manuscript, the final decision letter sent to the corresponding author by the editor in chief was recorded as the gold standard outcome. The final editorial decision by the editor in chief took into account the human reviews, the recommendation of the associate editor assigned to the manuscript, as well as the editor in chief's opinion. Final decisions were categorized as reject (including rapid reject), cascade to the Orthopaedic Journal of Sports Medicine (OJSM; including rapid cascade), or major revision. No submitted manuscript in the sample received an initial accept decision. Each manuscript had up to 4 volunteer human peer reviewers. The fourth review of a manuscript was excluded from analysis (5 manuscripts with a fourth reviewer). This exclusion applies to reviewer 4 as an analytical unit, not to individual reviews.
AI Review Generation
AI reviews were generated by Ministral 3 14B (Mistral AI), a 14 billion–parameter LLM. The model was run locally on a stand-alone university workstation (GMKtec EVO-X2, AMD Ryzen AI Max+ 395 processor, 128 GB RAM) via the Ollama framework with no internet connection. This local deployment ensured that no manuscript data were transmitted to external servers or commercial platforms, preserving the confidentiality of unpublished submissions.
Each manuscript was provided to the model as a complete PDF document, including all text, figures, tables, and supplementary materials. The AI was prompted once per manuscript using a standardized prompt (Appendix A, available in the online version of this article). The prompt instructed the model to act as an experienced peer reviewer for a high-impact sports medicine journal, produce a continuous narrative review without headings or bullet points, and provide a final categorical recommendation, as summarized in Table 1. Development of this prompt was based on established peer review frameworks, including the approach described by the editor in chief of AJSM for reviewing manuscripts submitted to sports medicine journals. 13 Default model parameters were used. The AI produced 3 outputs for each manuscript: a qualitative written review, a categorical recommendation (accept as is, request revision, cascade to sister journal, or reject), and a numerical score from 0 to 100. Two manuscripts had missing or indeterminate AI scores and were excluded from score-based analyses. It should be noted that the AI prompt presented cascade to OJSM as a direct categorical option alongside rejection, revision, and acceptance. This differs from the typical editorial workflow, in which cascade is considered secondarily after a rejection decision. The AI therefore had no means to replicate the 2-step human decision process; its cascade recommendations reflect a direct assessment of manuscript fit for AJSM rather than a true redirection decision, and this represents a structural limitation of the current implementation.
Artificial Intelligence Reviewer Output Parameters a
Outputs were generated using a standardized prompt (Appendix A, available in the online version of this article) applied once per manuscript.
Category Mapping
A 4-category model was used for all agreement analyses: reject (0), cascade to sister journal (1), revision (2), and accept (3). The mapping of raw reviewer and AI output categories to this model was performed after data collection. Final editorial decisions of “rapid reject” were mapped to reject, “rapid cascade” to cascade, and “major revision” to revision. Human reviewer categories of “reject” were mapped to reject; “cascade to OJSM” to cascade; “reconsider after major revision,”“accept with revisions,” and all revision variants to revision; and “accept in present form” to accept. AI categories of “reject” were mapped to reject; “cascade to OJSM” to cascade; “major revision,”“minor revision,”“request revision,” and “intermediate revision” to revision; and “accept as is” to accept. The complete mapping table is provided in Appendix B (available online). The 4-category model was selected as the primary analytical framework because it mirrors the actual decision categories observed in this data set. Among the 54 manuscripts included, the final editorial decision was limited to 3 categories: “reject” (including rapid reject), “cascade to OJSM” (including rapid cascade), and “major revision.” No manuscript received an initial “accept” decision during the study period. Collapsing all revision subtypes from human reviewers and the AI into a single revision category was therefore necessary to maintain alignment with the gold standard, which itself contained only 1 revision category.
Screening analyses were conducted using 2 complementary approaches. In the primary analysis, rejection and cascade were examined as distinct outcomes to evaluate whether the AI could discriminate between manuscripts that would be outright rejected and those that would be cascaded to OJSM. In a secondary analysis, rejection and cascade were combined into a single binary nonadvancement outcome, reflecting the editorial triage perspective that both represent manuscripts that will not be published in AJSM. Whether a cascaded manuscript is subsequently accepted at OJSM does not alter the AJSM editor's decision regarding suitability for AJSM.
Statistical Analysis
All analyses were performed in R Version 4.5.2 (R Foundation for Statistical Computing) with the irr package for agreement statistics. Statistical significance was set at P < .05. All tests were 2-tailed. Three human reviewers were compared independently against the final editorial decision for their assigned manuscripts, yielding separate kappa estimates per reviewer; all available human reviews were then combined into a single long-format data set for pooled comparisons (139 reviewer-manuscript pairs). The AI produced 1 review per manuscript and was therefore analyzed at the manuscript level throughout (n = 54). The final editorial decision, defined as the decision letter sent to the corresponding author by the editor, served as the gold standard and was not derived from or influenced by reviewer recommendations.
Unweighted and quadratic-weighted kappa values were computed for pooled comparisons. Kappa values were interpreted according to Landis and Koch 8 : <0.00 as poor agreement, 0.00-0.20 as slight, 0.21-0.40 as fair, 0.41-0.60 as moderate, 0.61-0.80 as substantial, and 0.81-1.00 as almost perfect. Confusion matrices were generated for AI and pooled human reviewers versus the final decision. An ordinal delta score was calculated as the absolute difference between each reviewer's mapped category and the final decision category (0 = exact match, 3 = maximum disagreement). Delta distributions were compared between human reviewers and AI using the Wilcoxon rank sum test.
The screening utility of AI was assessed by 2 complementary approaches. In the primary analysis, the proportion of manuscripts receiving each distinct final decision (reject, cascade, or revision) was calculated separately for each AI recommendation category to evaluate whether the AI could discriminate among these outcomes. In the secondary binary analysis, AI recommendations of reject or cascade were classified as a positive screening result and compared against a binary outcome of final decision reject or cascade versus revision or accept. In this context, sensitivity reflects the proportion of manuscripts ultimately rejected or cascaded that were correctly identified as such by the AI, while specificity reflects the proportion of manuscripts sent for revision that the AI correctly identified as unlikely to be rejected or cascaded. Sensitivity, specificity, positive predictive value (PPV), and negative predictive value were calculated with 95% confidence intervals via the Wilson method, with Fisher exact test for the 2 × 2 table. Kappa 95% CIs were calculated via bootstrap resampling (B = 2000). The AI's numerical score was analyzed by the final decision category via the Kruskal-Wallis test. A threshold analysis was performed across multiple AI score cutoffs (<60, 65, 70, 75, and 80) to evaluate screening performance at different levels of stringency.
Results
Fifty-four manuscripts submitted to AJSM were included in the analysis. The final decision was reject in 22 cases (40.7%, including 3 rapid rejects), cascade to OJSM in 21 cases (38.9%, including 1 rapid cascade), and major revision in 11 cases (20.4%). No manuscripts received an initial accept decision during the study period. A total of 139 individual human reviewer reports were available. The AI reviewed all 54 manuscripts and provided a categorical recommendation and numerical score (0-100) for 52. Two manuscripts had missing or indeterminate scores.
Agreement Between Reviewers and Final Decision
Using a 4-category model (reject, cascade, revision, accept), individual human reviewers demonstrated variable agreement with the final decision (Table 2). When human reviewers were pooled (139 reviews), agreement was fair and statistically significant (κ = 0.181; P < .001), with the final decision matched in 59 of 139 reviews (42.4%). The AI demonstrated slight, nonsignificant agreement with the final decision (κ = 0.126; P = .099), matching it in 20 of 54 cases (37.0%). The pooled AI agreement (37.0%) was numerically lower, but the Wilcoxon rank sum test comparing ordinal distance (delta) from the final decision did not reach significance between human reviewers and the AI (P = .183) (Table 3).
Cohen’s Kappa Agreement With the Final Decision: 4-Category Model a
Categories: reject (0), cascade to sister journal (1), revision (2), accept (3). Bold indicates statistical significance (P < .05). Reviewer 4 excluded (n = 5). Kappa 95% CIs calculated via bootstrap resampling (B = 2000).
Delta Score Distribution: Ordinal Distance From Final Decision a
Delta score represents the absolute ordinal distance between the reviewer's mapped category and the final editorial decision, where 0 indicates exact agreement and 3 indicates maximum disagreement across the 4-category scale (reject = 0, cascade = 1, revision = 2, accept = 3). Lower mean delta indicates closer overall alignment with editorial outcomes. Wilcoxon rank sum test used for group comparison.
The confusion matrices revealed the AI's predominant failure mode (Tables 4 and 5). Of 54 manuscripts, the AI recommended revision for 33 (61.1%), reject for 11 (20.4%), and cascade for 10 (18.5%). It never recommended acceptance. Among the 33 manuscripts that the AI recommended for revision, 24 (72.7%) received a final editorial decision of reject or cascade. By contrast, when human reviewers recommended revision (n = 66), the final editorial decision was reject or cascade in 65.2% of cases. The AI thus demonstrated a systematic tendency to recommend revision for manuscripts that would ultimately be rejected or cascaded per the final editorial decision—a pattern that we term “revision bias.”
Confusion Matrix: Artificial Intelligence Recommendation vs Final Decision a
Overall accuracy: 37.0% (20/54). Rows represent artificial intelligence recommendations; columns represent final editorial decisions. Diagonal cells (bold) represent exact agreement. Overall accuracy calculated as the proportion of manuscripts in which the artificial intelligence recommendation matched the final editorial decision.
Confusion Matrix: Pooled Human Reviewer Recommendation vs Final Decision a
Overall accuracy: 42.4% (59/139). Pooled across reviewers 1 to 3. Rows represent pooled human reviewer recommendations (reviewers 1-3; n = 139 reviewer-manuscript pairs); columns represent final editorial decisions. Diagonal cells (bold) represent exact agreement. Overall accuracy calculated as the proportion of reviewer-manuscript pairs in which the reviewer recommendation matched the final editorial decision.
AI as a Screening Tool
Rejection and cascade were analyzed as distinct outcomes. Among the 11 manuscripts that the AI recommended for rejection, the final decision was reject in 6 (54.5%), cascade in 3 (27.3%), and revision in 2 (18.2%). Among the 10 manuscripts that the AI recommended for cascade, the final decision was reject in 5 (50.0%) and cascade in 5 (50.0%), with none sent for revision (Table 6).
Screening Performance: Artificial Intelligence Predicting Final Decision of Reject or Cascade a
Data are presented as No. (%). Artificial intelligence categorical recommendations are compared directly against the final editorial decision by the editor in chief for all 54 manuscripts. For the artificial intelligence, final decisions of reject and cascade are shown separately and combined. Binary screening metrics (artificial intelligence reject or cascade predicting final decision by the editor in chief's decision of reject or cascade): sensitivity, 44.2% (95% CI, 30.4%-58.9%); specificity, 81.8% (95% CI, 52.3%-94.9%); positive predictive value, 90.5% (95% CI, 71.1%-97.3%); negative predictive value, 27.3% (95% CI, 15.1%-44.2%). Fisher exact test, P = .170. Proportion confidence intervals calculated using the Wilson method.
Despite limited overall agreement, the AI showed strong performance in identifying manuscripts unlikely to survive the review process. When the AI recommended reject, the final decision was reject or cascade in 9 of 11 cases (81.8%). When the AI recommended cascade, the final decision was reject or cascade in 10 of 10 cases (100%). When the AI recommended reject or cascade (n = 21), the PPV for a final decision of reject or cascade was 90.5%, with a specificity of 81.8%.
The AI's numerical score further discriminated between outcomes. Manuscripts ultimately rejected received a mean AI score of 63.9 (SD, 14.4) as compared with 70.2 (SD, 11.3) for those cascaded and 72.2 (SD, 17.3) for those sent for revision, although this difference did not reach statistical significance (Kruskal-Wallis, P = .148). Based on a score threshold <70, 25 manuscripts were flagged, of which 22 (88.0%) received a final editorial decision of reject or cascade for a PPV of 88.0% (95% CI, 70.0%-95.8%), sensitivity of 53.7% (95% CI, 38.7%-67.9%), and specificity of 72.7% (95% CI, 43.4%-90.3%).
A threshold sweep analysis demonstrated that the <70 threshold offered the optimal balance of PPV and sensitivity, with PPV remaining >82% at all tested thresholds while sensitivity increased from 22% at <60 to 76% at <80 at the cost of declining specificity (Table 7).
Artificial Intelligence Score Threshold Screening Analysis a
Binary outcome: final decision of reject or cascade vs revision or accept. Bold indicates optimal threshold balancing PPV and sensitivity. PPV, positive predictive value.
Discussion
This study evaluated the ability of an LLM to replicate human peer review decisions on manuscripts submitted to a high-impact sports medicine journal. The principal findings were that AI demonstrated only slight, nonsignificant agreement with the final decision (κ = 0.126; P = .099), as compared with fair agreement for human reviewers (κ = 0.181; P = .001). However, despite this limited categorical agreement, the AI showed potential as a screening tool: when it recommended rejection or cascade, the final decision agreed in >90% of cases, and manuscripts receiving an AI score <70 were rejected or cascaded 88% of the time. These findings are descriptive and exploratory given the sample size.
The AI's overall agreement with the final decision (37.0%) was numerically lower than that of the pooled human reviewers (42.4%; κ = 0.181; P < .001). These values are consistent with the broader peer review literature, which has repeatedly demonstrated that interreviewer reliability is only modest.3,7,8 In a large comparative analysis of 198 manuscripts, human reviewer agreement was no better than chance, 15 and a study of >3000 Nature family journal submissions showed that AI-human feedback overlap was comparable to human-human overlap. 10 Our findings add to this literature by demonstrating that the limited agreement between AI and editorial decisions at a high-impact medical journal is not dramatically different from the agreement observed between individual human reviewers and the same decisions.
The most notable finding of this study was the AI's systematic “revision bias.” Of 54 manuscripts reviewed, the AI recommended revision for 33 (61.1%), and 24 of those (72.7%) ultimately received a final decision of rejection or cascade. The AI rarely recommended outright rejection (20.4% of manuscripts), whereas the final decision was rejection or cascade for 79.6% of the sample. Stated simply, AI functioned as an “easy grader” of manuscripts. This pattern is consistent with findings from other studies reporting that LLMs tend to be more lenient than human reviewers. 18 The ChatGPT-4 analysis by Shopovski et al 15 found that AI never recommended rejection across 198 manuscripts, and Suleiman et al 16 noted that ChatGPT's comments on methodology had the lowest proportion of agreement with human reviewers (3.6%). The likely explanation is twofold: LLMs are trained on text that favors constructive and diplomatic feedback, and the prompt used in this study instructed the AI to provide “critical but constructive” commentary, which may have inadvertently calibrated the model toward recommending revision rather than rejection. By contrast, when human reviewers in our study recommended rejection (n = 35), the final decision was rejection or cascade in 100% of cases, suggesting that human reviewers apply a more definitive threshold when recommending against publication. Critically, the distinction between AI pattern recognition and human peer review judgment warrants emphasis. Human reviewers bring domain expertise, awareness of current literature, recognition of methodological subtleties, and contextual reasoning that cannot currently be replicated by a language model trained on generalized text.
Despite this revision bias, the AI demonstrated considerable value as a potential screening tool. When the AI did recommend rejection or cascade (n = 21), the PPV for a final decision of rejection or cascade was 90.5% (95% CI, 71.1%-97.3%), with a specificity of 81.8% (95% CI, 52.3%-94.9%). Similarly, the AI's numerical score provided useful prognostic information: manuscripts scored <70 were rejected or cascaded 88% of the time. A threshold analysis showed that a score of 70 offered a practical balance between screening yield and predictive accuracy, flagging nearly half of all manuscripts while maintaining a PPV of 88%. These findings align with the proposal by Bauchner and Rivara 2 that LLMs could serve a triage function in the editorial process; they also align with the observation by Liang et al 10 that GPT-4’s alignment with human reviewers was higher on lower-quality submissions, reaching 43.8% overlap on rejected ICLR papers as compared with 39.2% overall. Kousha and Thelwall 6 similarly concluded in their comprehensive review that there is good evidence AI can help with initial quality control of submitted manuscripts, even though its value for full reviewing has not been clearly demonstrated. This screening performance, however, reflects the combined outcome of rejection or cascade. When these outcomes were examined separately, the AI did not reliably discriminate between manuscripts that would be outright rejected and those that would be cascaded to OJSM—an editorially meaningful distinction, as the 2 represent different outcomes for authors and for the flow of manuscripts to the sister journal. The current findings therefore identify the potential of AI to identify manuscripts unlikely to be published in AJSM but not to determine their specific disposition.
If validated in larger studies, these findings carry meaningful practical implications for high-impact journals. High-impact sports medicine journals receive thousands of submissions annually, the majority of which are ultimately rejected. Currently, the editor in chief spends considerable time determining which manuscripts should proceed to peer review and which should be rapidly rejected or cascaded. An AI prescreening system that identifies manuscripts unlikely to survive the review process could reduce this burden and potentially further reduce reviewer burden by triaging submissions before they enter the full peer review process. This would not replace human expertise but rather reallocate it toward manuscripts with a realistic chance of acceptance. 1 Importantly, the descriptive PPV observed in this study suggests that few manuscripts flagged by AI would be erroneously excluded—the greater concern is false negatives (manuscripts that AI fails to flag), which would simply proceed through standard editorial review. A survey of the top 100 medical journals found that 78% had already established guidance on AI use in peer review, with 59% explicitly prohibiting use and 41% permitting limited use, suggesting that the field is actively grappling with how to integrate these tools responsibly. 9
A major obstacle to the implementation of AI as a reviewer is copyright law. As the authors retain the right to their intellectual property before publication, uploading manuscripts to a public AI model would infringe on those rights. It is important to note that currently we are not aware of any orthopaedic journals that have routine access to closed LLMs. It may be worthwhile for journal editors to lobby large medical publishers to purchase in-house LLMs to help screen submitted manuscripts. Furthermore, this could allow individual journals to train these models by submitting strong human reviews that align with the final editorial decisions, potentially improving the agreement of AI with future editorial decisions as a result.
Future studies should evaluate AI screening performance in larger samples across multiple journals, compare different LLMs and prompt strategies, and incorporate blinded expert assessment of AI review quality. The development of journal-specific calibration—adjusting AI recommendations to match a journal's historical acceptance rate—could improve categorical agreement. As the technology evolves, periodic reevaluation will be necessary to track improvements in LLM performance.
This study has several limitations. The sample size of 54 manuscripts, while sufficient for a proof-of-concept investigation, has limited statistical power and precludes more sophisticated analyses, such as a reliable receiver operating characteristic curve estimation. As with all agreement studies using editorial outcomes as a reference standard, the final decision reflects not only manuscript quality but also journal fit, topic priority, and reviewer expertise; however, for the purpose of evaluating AI-assisted triage, the actual editorial outcome represents the only operationally meaningful benchmark available at the time of submission. The wide confidence intervals around our screening metrics reflect this constraint, and the Fisher exact tests for screening performance did not reach statistical significance (P = .17), although the descriptive findings remain compelling. The study used a single LLM with a single prompt, and results may vary with different models, prompt designs, or fine-tuning approaches. The category mapping required subjective decisions about how to align the AI's output categories with the journal's decision categories, which may have affected agreement metrics. In addition, although the AI reliably identified manuscripts that would not advance at AJSM, it could not discriminate between manuscripts that would be outright rejected and those that would be cascaded to OJSM. Because cascade and rejection represent distinct editorial outcomes with different implications for authors and for the sister journal, this inability to distinguish between them is an important limitation of the current AI implementation as a triage tool. The AI reviewed manuscripts as PDF text and could not fully evaluate figures, images, or supplementary materials—a limitation shared with several prior studies.12,14 Additionally, the study was conducted at a single high-impact sports medicine journal with a high rejection rate; generalizability to journals with different acceptance rates or in other specialties requires further investigation. A formal a priori power analysis was not performed, as this study was designed as a proof-of-concept investigation using a convenience sample of available manuscripts rather than a prospectively powered agreement study. The sample size of 54 manuscripts was determined by feasibility and availability during the study period; future studies should prospectively calculate sample size requirements based on expected kappa values and desired precision. Finally, we did not formally assess the qualitative accuracy of the AI's written reviews, which represents an important avenue for future research.
AI cannot replicate the nuanced judgment of human peer reviewers at a high-impact sports medicine journal—a distinction that reflects the irreplaceable role of domain expertise, contextual reasoning, and clinical experience in the evaluation of scientific manuscripts. However, when AI recommended rejection or cascade, 90.5% of manuscripts received that final decision (descriptive PPV, 90.5%; 95% CI, 71.1%-97.3%), suggesting potential utility as an exploratory first-pass screening tool warranting further validation in larger cohorts.
Supplemental Material
sj-docx-1-ajs-10.1177_03635465261463006 – Supplemental material for Artificial Intelligence Cannot Replace Peer Reviewers but May Help Editors Triage: A Comparative Analysis of a Large Language Model and Human Reviewer Recommendations at the American Journal of Sports Medicine
Supplemental material, sj-docx-1-ajs-10.1177_03635465261463006 for Artificial Intelligence Cannot Replace Peer Reviewers but May Help Editors Triage: A Comparative Analysis of a Large Language Model and Human Reviewer Recommendations at the American Journal of Sports Medicine by Romir Patel, Christopher Shultz, Matthieu Ollivier and Daniel C. Wascher in The American Journal of Sports Medicine
Supplemental Material
sj-docx-2-ajs-10.1177_03635465261463006 – Supplemental material for Artificial Intelligence Cannot Replace Peer Reviewers but May Help Editors Triage: A Comparative Analysis of a Large Language Model and Human Reviewer Recommendations at the American Journal of Sports Medicine
Supplemental material, sj-docx-2-ajs-10.1177_03635465261463006 for Artificial Intelligence Cannot Replace Peer Reviewers but May Help Editors Triage: A Comparative Analysis of a Large Language Model and Human Reviewer Recommendations at the American Journal of Sports Medicine by Romir Patel, Christopher Shultz, Matthieu Ollivier and Daniel C. Wascher in The American Journal of Sports Medicine
Footnotes
Submitted March 24, 2026; accepted June 5, 2026.
One or more of the authors has declared the following potential conflict of interest or source of funding: M.O. is on the editorial board for the American Journal of Sports Medicine and Knee Surgery, Sports Traumatology, Arthroscopy. M.O. receives consulting fees and royalties from Stryker Corporation and Newclip Technics. D.C.W. serves as the deputy editor for the American Journal of Sports Medicine and associate editor for the Orthopaedic Journal of Sports Medicine. C.S. receives educational support from Arthrex, Inc and Desert Mountain Medical.
References
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
