Custom GPT models for complex rheumatology systematic reviews: A two-part evaluation of data extraction and prognosis appraisal

Introduction

Systematic reviews are central to evidence-based decision-making in rheumatology, offering transparent methods for identifying, appraising, and synthesizing primary studies for clinical practice and policy.^1,2 However, they are resource-intensive, ³ with two particularly demanding steps: full-text data extraction, ⁴ and structured risk-of-bias appraisal in prognostic research. ⁵ These challenges are accentuated in systemic lupus erythematosus (SLE) and other multisystem autoimmune diseases, where heterogeneous phenotypes, fluctuating activity, multi-organ outcomes, and high-dimensional biomarkers increase cognitive burden and the risk of inconsistency.^6,7

Metabolomics exemplifies the complexity of evidence synthesis in SLE. ⁸ Studies often use diverse analytical platforms and sample types, with key findings reported across tables, figures, and supplementary materials, making data extraction time-consuming and prone to variability.^4,9,10 Similarly, prognostic reviews require structured risk-of-bias assessment using tools such as the Quality in Prognosis Studies (QUIPS) tool, which evaluates six domains of potential bias in prognostic factor research and requires reviewer training and consensus across domains. ⁹ Against this backdrop, large language models (LLMs) have emerged as promising tools for streamlining systematic review workflows. ¹⁰ Recent GPT-based systems can process full-text articles and generate structured outputs, including extraction tables and risk-of-bias appraisals.^11–13 However, evidence from real-world biomedical applications remains limited, and current findings suggest that LLMs may complement rather than replace human reviewers, highlighting the need for rigorous benchmarking against expert standards. ¹⁴

Recent benchmarking studies have evaluated LLM performance in rheumatology and related specialties, showing potential for information retrieval, clinical reasoning, guideline interpretation, and evidence summarization, but with variable accuracy depending on task complexity and domain specificity.^15–17 Similar studies in other medical fields have assessed LLM-assisted data extraction, risk-of-bias assessment, and evidence synthesis, generally reporting efficiency gains but inconsistent agreement with expert reviewers. ¹⁸ These findings support human-supervised use of LLMs and highlight the need for domain-specific benchmarking in complex rheumatology systematic reviews, particularly for detailed extraction and QUIPS-based prognostic appraisal. Dedicated systematic review platforms such as Rayyan, Covidence, and ASReview are widely used to support screening, deduplication, reviewer collaboration, and record prioritization.^19–23 However, these tools are less focused on flexible full-text extraction and domain-specific risk-of-bias appraisal. Customized GPT-based models may complement these platforms by using task-specific instructions and templates to generate structured outputs from full-text articles. ²⁴ Our study therefore evaluates the added value of customized LLMs for later-stage review tasks, specifically data extraction and QUIPS-based appraisal in complex rheumatology reviews.

Methods

This methodological study was conducted under a protocol approved by the Local Research and Ethics Committee of the Instituto Mexicano del Seguro Social (approval number: R-2025-2106-011). The study involved secondary analysis of published literature and comparison of reviewer-generated and AI-generated assessments. No patient-level data were collected, and the human reviewers served as investigators rather than research participants. Therefore, informed consent was not required.

Results

Data extraction component

A total of 15 studies were included to evaluate qualitative concordance between data extracted manually by human reviewers and data extracted using the customized language model Systematic Review Extractor Pro. The comparison focused on shared extraction domains defined in the prespecified template, including study characteristics, sample descriptors, analytical techniques, and participant demographics.

Concordance varied substantially across domains. The highest agreement was observed for structural and explicitly reported variables: country of study achieved full agreement (100%) across all 15 articles, study design showed high concordance (73.3%), and sample type was matched in 53.3% of cases. In contrast, concordance for sample size and analytical platform/technique was 26.7%, while age, sex distribution, and female-to-male ratio showed 0% agreement. Domain-level concordance results are summarized in Table 1.

OPEN IN VIEWER

Table 1.

Concordance between human reviewers and Systematic Review Extractor pro across data-extraction domains in 15 SLE metabolomics studies.

Extraction domain	Concordant fields (n)	Total fields assessed (n)	Concordance (%)
Country of study	15	15	100.0
Study design	11	15	73.3
Sample type	8	15	53.3
Sample size (SLE cases and controls)	4	15	26.7
Analytical platform/technique	4	15	26.7
Age	0	15	0.0
Sex distribution/female-to-male ratio	0	15	0.0
Overall concordance	—	—	25.0

SLE, systemic lupus erythematosus. Concordance was defined as semantically or numerically equivalent extraction between the human-adjudicated reference and Systematic Review Extractor Pro. Overall concordance reflects the pooled estimate across the complete extraction template, not only the domains displayed in this summary table.

When pooled across all fields, mean concordance was 25.0%, indicating low overall agreement between GPT-based and human extractions. Discordance was concentrated in variables requiring interpretation of tables, figures, or supplementary materials: human reviewers consistently captured detailed participant-level data (e.g., age with dispersion measures, sex distribution, and case/control counts), whereas the GPT model frequently omitted or incompletely reported these values when they were not stated in continuous prose. In contrast, Systematic Review Extractor Pro performed better on higher-level methodological attributes (e.g., country, study design, sample type), supporting its potential role in semi-automated extraction of general study descriptors.

Discussion

This study evaluated a customized GPT-based tool (Systematic Review Extractor Pro) for automated data extraction in systematic reviews of metabolomic profiling in SLE. The model produced a five-fold reduction in extraction time compared with human reviewers. Still, concordance was limited for key clinical variables (sample size, age, and sex), particularly when information was embedded in tables or supplementary materials. In contrast, agreement was high for structural study attributes (e.g., country, design, sample type), suggesting greater utility for streamlining general descriptors than for numerically precise fields. In a parallel QUIPS evaluation, the customized GPT reviewer achieved transparent domain-level outputs, yet agreement with adjudicated human ratings ranged from essentially no agreement to substantial agreement, although most domains showed slight to fair agreement. Overall, these findings reflect both the promise and current constraints of LLMs for supporting systematic review workflows in the complex, heterogeneous literature on autoimmune diseases.

The domain-specific variability in concordance emphasizes differences in how LLMs handle narrative versus structured content. High agreement on general descriptors (e.g., country, design, sample type) suggests strong performance when information is presented in continuous text. In contrast, lower concordance for demographic and numeric variables reflects persistent difficulty extracting data from tables, figures, and supplementary materials, where clinically essential details are often concentrated. This pattern is consistent with prior work showing that LLMs excel at natural language comprehension but are less reliable for structured numerical extraction and fragmented reporting.^28–30 Given that metabolomics studies commonly present dense tabular results and layered methods, our findings highlight the challenges of using general-purpose LLMs for extraction tasks requiring high numerical fidelity. ³¹

The marked reduction in extraction time highlights the potential of LLMs to streamline systematic review workflows. With an approximately five-fold decrease in time per study versus manual extraction, Systematic Review Extractor Pro could scale one of the most resource-intensive steps in evidence synthesis. Although concordance was lower for granular participant-level data, the model performed well for general study characteristics, supporting pragmatic use cases such as pre-screening, initial data structuring, and updating reviews, particularly for rapid or scoping reviews where speed is prioritized.^26,32 Embedding these tools in human-in-the-loop pipelines, where reviewers verify and adjudicate extracted fields, may improve both throughput and consistency.^33,34 Prior work similarly suggests AI-assisted review tools can reduce workload and screening time while maintaining acceptable reliability for higher-level study features.^35–37

In contrast to its stronger performance for high-level study descriptors, the QUIPS analysis indicates current LLM limitations for nuanced risk-of-bias appraisal in prognostic research. Agreement between GPT-Reviewer and adjudicated human ratings was essentially absent or slight for study participation, study attrition, prognostic factor measurement, and outcome measurement; fair for statistical analysis/reporting; and substantial for study confounding, despite structured prompts and required justifications. While the model produced clear, QUIPS-aligned outputs, it struggled to integrate dispersed methodological cues across text, tables, and figures and to apply subtle decision thresholds. Accordingly, QUIPS appraisal remains a high-stakes interpretive task best supported, not replaced, by LLMs, which may be most helpful in structuring evidence, pre-populating rationales, or flagging domains for focused human review within a human-in-the-loop workflow.^38–40

Systematic reviews of complex diseases such as SLE present unique challenges due to inherent clinical heterogeneity, fluctuating disease activity, and diverse phenotypic expression.^6,7,41 The difficulty is further compounded when reviewing studies that employ high-throughput technologies such as metabolomics, which generate high-dimensional, multivariate datasets and often employ diverse analytical platforms, sample types, and statistical approaches.^42,43 Synthesizing findings across such heterogeneous sources requires meticulous extraction of methodological and demographic details that are frequently embedded within complex tabular structures or supplementary materials. ⁴ In this regard, AI-driven tools like Systematic Review Extractor Pro may offer strategic advantages, ⁴⁴ particularly in supporting the initial mapping of study characteristics, identifying methodological patterns, and flagging gaps in reporting. ¹⁰ As the field of precision medicine continues to expand, the application of natural language processing and LLMs in evidence synthesis holds promise for navigating the complexity of omics-based research in autoimmune diseases, facilitating more timely and scalable knowledge integration.^45–47

This evaluation has several notable strengths. To our knowledge, it represents the first assessment of a customized GPT-based model for data extraction within a systematic review focused on metabolomics in SLE, a domain characterized by substantial clinical and methodological heterogeneity. The study also leveraged a predefined extraction template derived from a PROSPERO-registered review. ⁴⁸ It adhered to PRISMA standards, ensuring methodological integrity and facilitating direct comparison between human and AI-based extraction outputs. Furthermore, by incorporating detailed time measurements, the study provides valuable empirical evidence of the efficiency gains achievable with LLMs.

Limitations

This study has several limitations. First, manual extraction and adjudicated human QUIPS assessments were treated as the reference standard. Although these procedures represent current best practice in systematic reviews, human reviewers are themselves subject to variability, interpretation differences, and extraction errors, which may have influenced the measured concordance between human and GPT-generated outputs. Second, this study was designed to evaluate agreement rather than accuracy. Adjudicated human assessments were used as the operational reference standard because no independent gold standard exists for many systematic review tasks, including data extraction and QUIPS-based risk-of-bias appraisal. Consequently, measures such as sensitivity, specificity, or area under the receiver operating characteristic curve were not applicable. The reported concordance and weighted kappa statistics should therefore be interpreted as indicators of agreement with expert reviewer judgments rather than objective measures of correctness. ⁴⁹ Third, the evaluation was based on a relatively small sample of studies (15 metabolomics studies for data extraction and 19 prognostic studies for QUIPS appraisal). While sufficient for a methodological proof-of-concept, the sample size limits the generalizability of the findings to other disease areas, study designs, and systematic review contexts. Future studies should evaluate customized GPT-based tools across larger and more diverse datasets. Fourth, the customized GPTs were built on a general-purpose LLM rather than a model specifically trained for biomedical evidence synthesis. ⁵⁰ Consequently, performance was lower for variables requiring interpretation of structured content, including numerical tables, figures, and supplementary materials. This limitation likely contributed to the reduced concordance observed for participant demographics, sample sizes, and other quantitative variables. Fifth, the dynamic nature of commercial LLM platforms presents challenges for reproducibility. The customized GPTs were accessed through the OpenAI ChatGPT environment, and model behavior may vary over time due to platform updates, model revisions, subscription-level access, system instructions, and session context. To reduce contextual carryover, each article was processed in a separate chat session using the same predefined configuration and instructions. Nevertheless, exact replication of outputs may not be possible as underlying models evolve. Sixth, LLM outputs should not be considered intrinsically authoritative. Although the models were restricted to the uploaded study documents and web browsing was disabled, LLMs are trained on large-scale heterogeneous datasets and may generate unsupported, inaccurate, or incomplete responses. Furthermore, the underlying training data of commercial models are not fully transparent and may contain inaccuracies, biases, or outdated information. ⁵¹ As a result, complete control over the provenance and quality of all knowledge incorporated into the model cannot be guaranteed. For this reason, all AI-generated outputs required human verification and adjudication.

Another important limitation relates to the use of online, commercial LLM-based systems rather than closed-loop, locally deployed AI models. Because general-purpose LLMs are trained on large-scale online data and operate within proprietary cloud-based environments, their outputs may be affected by training-data bias, unsupported inferences, hallucinations, and limited transparency regarding the provenance of generated content. In addition, online AI platforms may be less suitable for workflows involving sensitive or patient-level information because uploaded data may raise concerns regarding compliance with data protection laws and institutional governance requirements. Recent closed-loop AI models developed in pediatric rheumatology, including Morgaf for childhood-onset SLE, ⁵² and Maverik for childhood-onset chronic nonbacterial osteomyelitis, ⁵³ illustrate an alternative approach in which data are processed offline or within restricted computer-based environments, without reliance on online databases or external conversational platforms. Such closed-loop models may offer advantages in data security, auditability, and protection from online data contamination or bias. Therefore, although customized GPT-based tools may support evidence-synthesis workflows using published literature, future applications involving sensitive clinical data should prioritize closed-loop, locally governed AI architectures with transparent validation, secure data handling, and human expert oversight. Seventh, internet connection speed and bandwidth were not formally measured during GPT evaluations. Although all analyses were conducted using a stable internet connection and no connectivity interruptions were observed, minor variations in network latency may have influenced recorded response times. Given the substantial difference between human and GPT-assisted extraction times, the impact of this factor is likely to have been minimal. Nonetheless, future benchmarking studies should consider recording network performance metrics to further enhance reproducibility. Additionally, generation parameters such as temperature were not directly accessible within the Custom GPT environment. Consequently, output variability attributable to platform-defined generation settings could not be formally controlled or evaluated. Finally, the extraction and appraisal workflows were not fully automated and required user involvement for sequential PDF uploads, template provision, prompt execution, and verification of outputs. These requirements highlight the continued importance of human–AI hybrid workflows and suggest that current customized GPT-based systems should be viewed as decision-support tools rather than replacements for trained systematic reviewers.

Conclusions

This two-part methodological study supports LLMs as complementary tools in systematic review workflows, particularly for accelerating the extraction of general study attributes and structuring risk-of-bias appraisals. Systematic Review Extractor Pro yielded substantial time savings and high agreement for structural descriptors, but performance was weaker for nuanced clinical variables drawn from complex tables, figures, and supplementary materials. Likewise, the GPT-based QUIPS reviewer produced transparent domain-level judgments. Yet, agreement with adjudicated human ratings ranged from essentially no agreement to substantial agreement, although most domains showed slight to fair agreement.

Future work should prioritize domain-specific optimization and improved interaction with structured content (e.g., table parsers and robust PDF-to-structured-data pipelines), within human-in-the-loop systems where outputs are routinely reviewed and adjudicated. Overall, while these tools cannot replace human expertise for high-stakes tasks, their standardization and time-saving potential support their integration into methodologist-supervised, semi-automated review pipelines.

Supplemental material

Supplemental material

Supplemental material

Supplemental material

Supplemental material