Accuracy and readability analysis of ChatGPT responses to frequently asked questions about olecranon fractures

Introduction

Olecranon fractures are among the most common injuries of the elbow, accounting for approximately 10–18% of all forearm fractures. Their occurrence is primarily associated with direct trauma, such as a fall resulting in a direct impact to the posterior aspect of the elbow, or with indirect mechanisms, including a fall onto an outstretched hand leading to forced hyperextension of the elbow joint. Because olecranon fractures significantly compromise elbow stability and function, the principal objectives of treatment are to achieve anatomical reduction, restore articular congruity and joint stability, facilitate optimal fracture healing, and minimize postoperative complications. ¹

In recent years, patients have increasingly turned to the internet as a primary source of medical information, particularly after sustaining traumatic injuries such as fractures. Many individuals search online to learn about surgical indications, recovery timelines, and potential complications before or after consulting a physician.^2,3 Previous studies have shown that approximately 90% of Americans had internet access in 2018, and nearly 75% searched online for health-related information, highlighting the growing importance of digital health resources. ⁴ Throughout the pandemic, a considerable number of healthcare professionals and members of the public have turned to the Internet, particularly YouTube, to seek information about their medical conditions. Studies indicate that 80% of individuals in the United States preferred utilizing online resources for health-related information during this period. ⁵ The importance of online health information becomes particularly evident during periods of uncertainty regarding disease diagnosis and treatment. Following the emergence of the first COVID-19 case in November 2019, the infection spread rapidly worldwide, and the World Health Organization declared a pandemic in March 2020. Considerable uncertainty existed regarding the diagnosis and treatment protocols of this infection, which affected multiple organ systems, particularly the respiratory system. As a result, many healthcare professionals and members of the general public turned to the Internet, particularly YouTube, to obtain information. ⁶ In addition to AI-based tools, online platforms and video-sharing websites such as YouTube continue to serve as important sources of health-related information. Previous studies evaluating online patient education materials and YouTube-based medical content have reported variable quality, reliability, and readability, highlighting the need for careful assessment of digital health information sources.^7,8 However, the accuracy, completeness, and readability of orthopaedic information available on the internet vary widely. ⁹ Numerous studies have demonstrated that online educational materials are often written above the average health literacy level of the general population, making them difficult for patients to comprehend.^10,11 The National Institutes of Health, the U.S. Department of Health and Human Services, and the American Medical Association recommend that patient education materials be written at or below the sixth-grade reading level to ensure accessibility for the general population. ¹² Consequently, misinformation or misinterpretation of online content may lead to anxiety, unrealistic expectations, and confusion during the treatment decision-making process. As patients increasingly use online resources to better understand the causes, treatment options, and expected outcomes of their conditions, the quality and readability of the information they receive may influence treatment-related expectations and decision-making. ¹³

Beyond traditional academic publications, digital platforms and social media have become increasingly influential in the dissemination of health-related information. Recent bibliometric studies have highlighted that online engagement and alternative metrics (Altmetrics) may complement traditional citation-based measures by reflecting the visibility and public reach of medical content. ¹⁴ As patients increasingly rely on digital sources for health information, evaluating the readability of information delivered through emerging technologies such as artificial intelligence (AI) has become increasingly important.

In response to the increasing demand for accessible and understandable medical information, AI–based conversational tools have recently emerged as alternative sources of health education. ¹⁵ Among these, ChatGPT (OpenAI, San Francisco, CA, USA) has attracted widespread attention for its ability to generate human-like, contextually relevant answers to user queries. As of November 2024, ChatGPT was estimated to have approximately 300 million weekly active users and 3.8 billion monthly visits. ¹⁶ The model has demonstrated potential in improving patient communication, facilitating education, and assisting clinicians in information retrieval. Recent investigations have evaluated the reliability of ChatGPT in providing orthopaedic information across multiple subspecialties. Fahy et al. assessed ChatGPT-3.5 and ChatGPT-4 for anterior cruciate ligament (ACL) injuries and found that both versions produced generally good-quality responses according to DISCERN criteria, although readability levels exceeded recommended health literacy standards. ¹⁷ Similarly, Megalla et al. analysed ChatGPT’s performance in rotator cuff tear–related patient education and reported high readability but moderate factual accuracy. ¹⁸ In addition, Prasad et al. examined ChatGPT’s answers to common patient questions about total hip arthroplasty, demonstrating coherent and comprehensive responses with occasional factual inconsistencies. ¹⁹ Collectively, these studies suggest that while ChatGPT can generate fluent and accessible orthopaedic content, its factual reliability remains inconsistent. However, trauma-related orthopaedic conditions—particularly upper-extremity fractures such as olecranon injuries—have not yet been systematically investigated in this context. Given the limited literature regarding ChatGPT-generated patient information on olecranon fractures, the present study was designed as a focused evaluation of ChatGPT-4o.

Therefore, this study aimed to evaluate the quality, reliability, and readability of ChatGPT’s responses to common patient questions regarding olecranon fractures. By systematically assessing the quality of AI-generated information using standardized evaluation tools, we sought to determine ChatGPT’s reliability as a patient education resource in trauma orthopaedics. Although AI–based conversational models have been investigated in various orthopaedic domains, literature addressing trauma-related conditions, particularly upper-extremity fractures, remains limited.

Materials and methods

This cross-sectional study evaluated the quality, reliability, and readability of ChatGPT-4o responses to frequently asked patient questions regarding olecranon fractures.

Results

The mean DISCERN and GQS scores of ChatGPT-generated answers were 51.25 ± 4.13 and 4.10 ± 0.58, respectively, indicating overall good quality and reliability.

The mean FRE score was 62.00 ± 5.12, suggesting a moderate readability level.

The mean FKGL, SMOG, and GFI values (9.23 ± 1.31, 8.42 ± 0.94, and 9.37 ± 0.96, respectively) indicate that the content corresponds approximately to a ninth-grade reading level, which is slightly higher than the recommended eighth-grade level for patient education materials (Table 1).

OPEN IN VIEWER

Table 1.

Descriptive statistics of quality and readability scores.

Variables	Mean ± SD	Minimum	Maximum
DISCERN Score	51.25 ± 4.13	44.50	57.50
GQS score	4.10 ± 0.58	3.00	5.00
Flesch Reading Ease (FRE)	62.00 ± 5.12	53.00	71.00
Flesch–Kincaid Grade Level (FKGL)	9.23 ± 1.31	6.78	11.69
Simple Measure of Gobbledygook (SMOG)	8.42 ± 0.94	6.65	10.66
Gunning Fog Index (GFI)	9.37 ± 0.96	7.80	11.30

A very strong positive correlation was observed between DISCERN and GQS scores (r = 0.913, p < 0.001), indicating consistent quality assessment between the two scoring systems.

Both DISCERN and GQS scores showed moderate negative correlations with the Flesch–Kincaid Grade Level (FKGL) (r = –0.445, p = 0.050; r = –0.467, p = 0.038, respectively), suggesting that higher-quality responses tended to be written at slightly lower grade levels, reflecting better readability. No significant correlations were found between DISCERN or GQS and the other readability indices (FRE, SMOG, and GFI) (Table 2).

OPEN IN VIEWER

Table 2.

Significant correlations between quality (DISCERN and GQS) and readability scores.

Comparison	r	p-value
DISCERN vs GQS	0.913	<0.001*
GQS vs FKGL	–0.467	0.038*
DISCERN vs FKGL	–0.445	0.050

Interobserver reliability between the two evaluators for both DISCERN and GQS scores was assessed using Pearson’s correlation coefficient (r) and the Bland–Altman method.

Although the Intraclass Correlation Coefficient (ICC) is commonly used for such analyses, it was not appropriate in this dataset due to the very small interrater variance, which led to unstable ICC estimates. Therefore, Pearson correlation and Bland–Altman analysis were used as robust alternatives.

Pearson’s r evaluates the linear association between the two sets of scores, with higher r values indicating stronger agreement. A correlation coefficient (r) above 0.9 was considered excellent, between 0.7–0.9 strong, and between 0.5–0.7 moderate agreement.

The Bland–Altman method was applied to assess systematic bias and the limits of agreement (LoA) between the two raters. Mean differences between the raters’ scores were calculated, and the limits of agreement were defined as the mean difference ± 1.96 times the standard deviation (SD) of the differences. This approach determines whether the two raters assessments differ by a clinically relevant amount.

In the present study, Pearson correlation showed excellent agreement for DISCERN (r = 0.95, p < 0.001) and moderate agreement for GQS (r = 0.64, p = 0.003). The Bland–Altman analysis demonstrated a mean bias close to zero, with limits of agreement ranging from –3.14 to +2.94 for DISCERN and –1.10 to +1.10 for GQS, indicating no systematic deviation between raters and clinically acceptable consistency (Table 3).

OPEN IN VIEWER

Table 3.

Interobserver agreement for DISCERN and GQS scores.

Measure	Pearson’s r	p-value	Mean difference (bias)	95% limits of agreement (±1.96 SD)
DISCERN	0.95	< 0.001	∼0.00	–3.14 to +2.94
GQS	0.64	0.003	∼0.00	–1.10 to +1.10

Discussion

In recent years, the use of AI–based applications such as ChatGPT has rapidly increased among patients seeking medical information. Many individuals now rely on conversational AI tools to obtain explanations about their diagnoses, treatment options, and postoperative care, often before consulting a physician. While this trend enhances accessibility to medical knowledge, it also raises concerns regarding the accuracy, reliability, and readability of the information provided. ChatGPT is available 24/7 and enables patients with access to this technology to obtain health information at any time, which can be particularly beneficial for those with urgent medical needs or those living in remote areas. ²¹ In this context, the present study aimed to evaluate the reliability, quality, and readability of ChatGPT generated responses about olecranon fractures, a common orthopaedic condition that frequently prompts online health queries.

Previous investigations across various orthopaedic subspecialties have consistently shown that ChatGPT provides medically accurate yet linguistically complex responses. Fenn et al. evaluated ChatGPT’s output for hamstring injury–related questions and reported generally satisfactory accuracy, although many responses lacked completeness. The average readability corresponded to a tenth-grade level, indicating that while the content was factually correct, it might not be accessible to all patients. ²² Similarly, Stevens et al. compared ChatGPT, Google Gemini, and Bing Copilot for carpal tunnel syndrome–related queries and found comparable DISCERN scores across all platforms, with ChatGPT achieving “fair” quality (mean = 45), suggesting reliable but unsourced information. ²³

In the hip preservation domain, Gaddis et al. demonstrated that ChatGPT-4.0 produced excellent or satisfactory answers to periacetabular osteotomy–related questions, achieving 98% accuracy. However, the responses were written at an eleventh-grade level, far exceeding patient education recommendations. When explicitly prompted, readability could be adjusted to the seventh- or eighth-grade level, underscoring the model’s adaptability. ²⁴ Comparable findings were observed by Lack et al. in reverse shoulder arthroplasty queries, where ChatGPT offered fair-quality responses (mean DISCERN = 44) but at a college-graduate reading level, thus limiting patient comprehension. ²⁵

A systematic review by Kodra et al. synthesized 17 studies on orthopaedic sports medicine and confirmed that ChatGPT responses generally ranged from moderate to high quality (mean DISCERN = 41–62) with strong inter-rater reliability, yet readability scores frequently aligned with high-school or college-level comprehension (FKGL = 10–16). The authors emphasized that high reading complexity remains a major barrier to accessibility. ²⁶

Similar concerns were echoed by Atahan et al., who found that although ChatGPT-4.0’s answers to orthopaedic trauma questions were readable (mean FKGL = 10.5), readability alone was not a reliable indicator of clinical accuracy or safety. ²⁷

Beyond orthopaedics, studies on musculoskeletal malignancies and general patient-education materials have reported even higher linguistic demands. Guirguis et al. observed that ChatGPT-3.5 responses for bone cancer information averaged a 16th-grade level—equivalent to a college senior—falling short of NIH readability recommendations that advocate a sixth-to eighth-grade level for patient education. ²⁸ Kirchner et al. further demonstrated that AI dialogue platforms can successfully lower complex orthopaedic texts from high-school to sixth-grade readability without compromising factual accuracy, suggesting a feasible solution for future clinical integration. ²⁹

The findings of the present study align closely with these prior observations. ChatGPT generated high-quality and reliable responses regarding olecranon fractures, reflected by mean DISCERN and GQS scores of 51.25 ± 4.13 and 4.10 ± 0.58, respectively, consistent with “good” quality in previous orthopaedic reports. However, the mean readability corresponding to a ninth-grade level indicates that its responses remain slightly more complex than recommended patient-education standards. Readability challenges are not limited to English-language health information. In a Turkish study evaluating internet-based patient education materials related to low back pain, the materials were found to have a moderate readability level, low reliability, and poor quality, with readability levels exceeding the recommended sixth-grade level. ³⁰ This pattern mirrors the broader literature, reinforcing that while ChatGPT can deliver coherent and generally high-quality medical explanations, linguistic refinement is essential to maximize accessibility for the general population.

Interestingly, higher DISCERN and GQS scores were associated with lower FKGL values, suggesting that higher-quality responses tended to be written at slightly more accessible reading levels. Although the observed correlations were moderate, these findings should be interpreted with caution, as quality and readability instruments assess distinct constructs. Therefore, the observed associations may not necessarily reflect a direct relationship between information quality and linguistic complexity. Nevertheless, the findings suggest that improving readability does not inevitably compromise information quality.

From a clinical perspective, responses written above the recommended reading level may be difficult for some patients to fully understand, potentially limiting the effectiveness of AI-generated educational content. Therefore, efforts to improve readability while preserving information quality may increase the potential utility of AI-based tools in patient education.

Strengths and limitations

This study has several strengths. To our knowledge, it is the first to assess ChatGPT’s responses to patient questions specifically about olecranon fractures, providing an original contribution to AI-based patient education in orthopaedic trauma. The use of validated instruments (DISCERN and GQS) ensured objective evaluation, while four complementary readability indices offered a comprehensive linguistic assessment. Independent dual-rater scoring and correlation analyses (Pearson and Bland–Altman) enhanced methodological rigor.

However, some limitations exist. The analysis included only English-language responses, limiting generalizability. ChatGPT’s data sources and internal algorithms are not fully transparent, and model updates may alter future performance. External validation against clinical guidelines was not conducted. Therefore, the findings should be interpreted as an assessment of information quality and reliability rather than direct factual accuracy. In addition, readability scores were calculated using a single online readability calculator. Previous studies have demonstrated that readability estimates may vary across different assessment platforms; therefore, variations in the reported readability scores cannot be excluded. ³¹ Finally, the study focused on patient-oriented information rather than clinical decision-making. Despite these limitations, it provides valuable insight into AI-generated educational content in orthopaedic trauma.

Conclusions

ChatGPT-4o showed good quality and reliability in answering questions about olecranon fractures, consistent with previous orthopaedic findings. However, its ninth-grade readability exceeds the recommended level for patient materials, potentially limiting comprehension. ChatGPT may serve as a supportive tool for patient education, but physician oversight remains essential. Future studies should focus on AI models that adapt language complexity while maintaining information quality and reliability.

Supplemental material

Supplemental material