Performance and usability of retrieval-augmented large language models for stroke patient and caregiver support

1. Introduction

Stroke continues to be a significant global health concern as the leading cause of death and permanent disability worldwide. ¹ The burden is particularly high in China due to rising incidence and recurrence rates, especially among older populations and those with uncontrolled vascular risk factors. ² Stroke is also a leading cause of death in China, with a mortality rate of 197.36 per 100,000 people. ¹ Although acute-phase treatment is critical, effective stroke management extends well beyond the acute stage. ³ From emergency response to long-term rehabilitation and secondary prevention, patients and caregivers frequently face persistent gaps in receiving timely and accurate stroke-related health education information.

Large language models (LLMs), especially ChatGPT, ⁴ are examples of artificial intelligence technology that have advanced significantly in recent years and shown exceptional text generation and comprehension skills. These models show immense potential in healthcare, especially as intelligent question-answering systems,^5,6 which are expected to provide stroke patients, their families and non-specialist medical staff with a convenient way to acquire knowledge, assist in education and decision support. ⁷ However, pure LLMs also reveal significant shortcomings when applied to medical scenarios that require extremely high accuracy. The biggest problem is that they might create hallucinatory information, ⁸ which is unacceptable in the medical field because it appears plausible but is false or fabricated. A significant challenge lies in their inability to continuously assimilate updated medical guidelines and care practices, alongside their struggle to offer authoritative and traceable advice for complex, specific care dilemmas. These factors critically impede the reliability and practical efficacy of deploying LLMs directly for caregiver assistance.

Retrieval-augmented generation (RAG) integrates an information retrieval component with a large language model, enabling the system to proactively retrieve pertinent, accurate, and up-to-date information from one or more external knowledge bases before to response generation.^9,10 This architecture notably curtails the generation of factually incorrect information, leading to improved accuracy and robustness in the generated replies. It also strengthens information provenance, allowing caregivers to validate the sources for enhanced confidence. RAG technology is demonstrating transformative potential in healthcare, particularly in supporting patients and caregivers. Prior studies have demonstrated the utility of RAG frameworks in various clinical contexts, ¹¹ including providing clinical decision support, ¹² enhancing patient education, ¹³ and creating dynamic plans for both chronic disease management^14–16 and personalized treatment.^17,18 However, most of these studies have primarily focused on system-side evaluation, with limited empirical validation involving end users such as patients or caregivers. For stroke, most existing studies have concentrated on the application of LLMs in zero-shot scenarios, primarily supporting clinical decision-making processes or aiding in stroke diagnosis and prognostication.^19–21 Although recent studies have begun to explore the integration of RAG with LLMs for stroke-related classification tasks, such as using GPT-4o to identify stroke subtypes within unstructured electronic health records, ²² the application of RAG-enhanced LLMs for interactive, patient- or caregiver-facing stroke education and support is an understudied topic. As a result, they have limited insights into usability and acceptability in the real world.

This study aims to develop and systematically evaluate a RAG question-answering system tailored for stroke-related queries. By constructing a localized knowledge base and comparing the performance of three leading large language models (GPT-4, Claude3.7, and Qwen3) under both zero-shot and RAG configurations, we assess technical performance, clinical validity, and end-user usability. The research seeks to address critical gaps in AI-assisted stroke care, providing a reliable, guideline-based information tool for patients and caregivers while exploring the potential of RAG to enhance LLM reliability in healthcare contexts.

This study makes multifaceted contributions by uniquely integrating AI technology, clinical needs, and end-user demands. Unlike prior research, our localized knowledge base combines diverse authoritative sources to ensure regional applicability, while our evaluation framework comprehensively assesses LLM performance through technical metrics (184 exam questions), clinician-led validation (30 open-ended questions), and end-user usability (SUS/NPS scales). This design provides novel insights into RAG’s potential to mitigate hallucination and improve traceability, while revealing trade-offs between performance enhancement and output understandability. For clinical practice, our work delivers a feasible, evidence-based solution for stroke management; for health policy, it lays the groundwork for AI integration in chronic disease care, emphasizing the balance between reliability and accessibility.

2. Methods

3. Results

3.2. Expert evaluation of response quality

Three clinical experts independently evaluated the model responses across four dimensions: accuracy, completeness, relevance, and understandability. Each response was independently rated, and the mean scores for the four evaluation dimensions and the overall averaged scores under zero-shot and RAG configurations are summarized in Table 1 and Figure 3. For the dimensions of accuracy and completeness, all three models demonstrated clear and statistically significant performance gains (all P< 0.01). In terms of relevance, the RAG enhancement also led to improved scores across all models. This improvement was most pronounced for GPT-4o, with its mean score increasing from 4.50 (SD 0.62) to 4.79 (SD 0.53; P < 0.01).

OPEN IN VIEWER

Table 1.

Expert evaluation of response quality for LLMs under zero-shot and RAG configurations.

Model	Dimension	Zero-shot	RAG	p
GPT-4o	Accuracy	4.28±0.58	4.72±0.54	<0.01
	Relevance	4.50±0.62	4.79±0.53	<0.01
	Completeness	4.16±0.69	4.60±0.56	<0.01
	Understandability	4.53±0.62	4.54±0.54	1
Claude3.7	Accuracy	4.41±0.58	4.81±0.45	<0.01
	Relevance	4.50±0.64	4.62±0.61	0.1999
	Completeness	4.27±0.68	4.53±0.66	<0.01
	Understandability	4.64±0.57	4.36±0.53	<0.01
Qwen3	Accuracy	4.29±0.69	4.58±0.65	<0.01
	Relevance	4.63±0.63	4.79±0.46	0.0677
	Completeness	4.23±0.75	4.54±0.58	<0.01
	Understandability	4.77±0.45	4.44±0.62	<0.01

OPEN IN VIEWER

Figure 3.

Comparison of zero-shot and RAG performance across three LLMs: (a) Comparison of zero-shot and RAG configurations across four dimensions (accuracy, completeness, relevance, and understandability) for each model; (b) Comparison of the overall average scores for each model under the two configurations.

While the RAG enhancement improved factual accuracy, this did not consistently lead to better understandability. For GPT-4o, the understandability score remained stable, showing no significant change after the RAG enhancement (mean 4.53 vs. 4.54; P=1). The mean score for Claude3.7 dropped from 4.64 (SD 0.57) to 4.36 (SD 0.53) (P< 0.01). Similarly, the score for Qwen3, which had the highest baseline understandability, decreased from 4.77 (SD 0.45) to 4.44 (SD 0.62) (P< 0.01). This made the answers less accessible to a lay audience compared to the more conversational and simplified language used by the Zero-shot models. This result underscores a critical challenge: while RAG excels at improving informational fidelity, it necessitates a sophisticated generation strategy to maintain high understandability for non-expert end-users.

Among the models, Claude3.7 achieved the highest individual scores in accuracy (4.81 ± 0.45) under RAG. However, GPT-4o demonstrated the best overall performance, achieving the highest average score across the four dimensions (4.66) under RAG. Qwen3 achieved the highest overall average score under the Zero-shot configuration (4.48).

3.3. Consistency among expert evaluation

We assessed inter-rater reliability among the expert evaluations. The ICC(3,k) model was applied to assess the degree of agreement among multiple observers evaluating the same set of items. ²⁶ The ICC values and corresponding 95% confidence intervals are presented in Table 2. The three clinical experts had moderate to good inter-rater reliability for all models. In general, RAG-based responses had higher inter-rater agreement than zero-shot outputs for all the models that were tested.

OPEN IN VIEWER

Table 2.

Intra-class correlation coefficients (ICCs) for expert evaluations.

Model	Zero-shot (95%CI)	RAG (95% CI)
GPT-4o	0.61 (0.28-0.80)	0.76 (0.55-0.87)
Claude3.7	0.52 (0.17-0.75)	0.73 (0.51-0.86)
Qwen3	0.59 (0.21-0.80)	0.62 (0.31-0.80)

4. Discussion

The RAG-based question-answering system evaluated in this study has important clinical value for stroke patients and their caregivers. Throughout the stroke care continuum, from acute hospitalization to rehabilitation and long-term community living, patients and caregivers frequently encounter challenges in accessing accurate, timely health information. Developed as a supplementary patient-centered educational resource rather than a diagnostic or therapeutic tool, the system aims to provide reliable answers to common stroke caregiving questions, improve health literacy, strengthen caregiver confidence, and support informed self-management. To establish the reliability and suitability of the system for its intended users, a multi-stage validation approach was adopted. Model performance was objectively evaluated using standardized stroke-related questions derived from professional examination banks, followed by blinded expert assessment across four dimensions: accuracy, relevance, completeness, and understandability. This was complemented by real-world usability testing involving stroke patients and caregivers using the SUS and NPS. Overall, these evaluation procedures provide preliminary evidence supporting the reliability, usability, and acceptability of the RAG-enhanced approach in stroke patient education.

We systematically compared GPT-4o, Claude 3.7, and Qwen3 under zero-shot and RAG configurations for stroke caregiving. The results demonstrated that RAG-based models generally outperformed their zero-shot counterparts in terms of accuracy, completeness, and relevance. Among the models tested, GPT-4o demonstrated the best overall performance under the RAG configuration, a finding that aligns with previous comparative studies of LLMs in medical RAG applications.^27,28 This indicates its compatibility with structured knowledge input and its effectiveness in addressing clinical questions. However, the integration of RAG was not universally beneficial across all evaluation dimensions. Notably, while RAG consistently improved accuracy, relevance, and completeness, it led to a decrease in understandability scores for Claude3.7 and Qwen3. This limitation has also been observed in other medical RAG applications. For example, a recent evaluation of RAG-enabled models for postoperative rhinoplasty patient queries reported that although responses were generally accurate and comprehensive, readability and understandability fell below recommended standards for patient education materials, ²⁹ highlighting persistent challenges in producing patient-friendly outputs even when grounded in authoritative sources. One possible reason is that retrieval augmentation may bring up more specialized or guideline-oriented content from source materials. This content may be clinically accurate, but it can also include technical terms and formal language that are harder for non-experts to understand. This suggests that while retrieval augmentation enhances factual reliability and information richness, it may inadvertently reduce clarity for end users and compromise contextual integration and coherence, especially when responses become denser or more formal. ¹⁷

Interestingly, Qwen3 achieved the highest ratings in both relevance and understandability under the zero-shot configuration. One possible explanation is that Qwen3, as a Chinese-developed LLM, may be better aligned with the linguistic and cultural nuances of the Chinese context in which this evaluation was conducted. This alignment may have supported the generation of more contextualized and reader-friendly responses, even without external knowledge augmentation. Moreover, Qwen3 also outperformed GPT-4o and Claude3.7 in the structured multiple- and single-choice question tests under the zero-shot configuration, which may reflect its potential adaptation to the linguistic patterns and item formats of Chinese professional or licensure examination items, from which the test questions were derived. However, this interpretation remains speculative and should be examined in future comparative studies.

Departing from many studies that focus solely on algorithmic performance, we incorporated direct user testing. ³⁰ The usability evaluation of the RAG-based assistant powered by GPT-4o showed a high level of user acceptance. These findings indicate that the assistant has good usability and was generally well-accepted by its target users, including stroke patients and family caregivers, in real-world contexts. Furthermore, during user testing, we observed that when participants phrased questions in a more colloquial or emotionally expressive manner, such as “My mother had a stroke, I’m so worried, how should I help her with limb function exercises?”, the system appeared to generate responses with a more empathetic tone. They were more understandable compared to the answers to standardized questions, such as “How should stroke patients perform limb function exercises”. This observation suggests that an advanced LLM like GPT-4o, within a RAG framework, may support factual accuracy while also appearing to capture and respond to users’ emotional context, adapting its communication style to provide more compassionate and human-centered support in real-world interactions. ³¹ However, this observation was exploratory and was not assessed using a formal empathy evaluation metric.

The real-world implementation of a RAG-based stroke care assistant depends not only on its technical accuracy but also on a robust governance framework to ensure its long-term safety and trustworthiness. ³² Firstly, given the dynamic nature of stroke guidelines, the knowledge base requires a continuous update process to remain synchronized with the latest clinical evidence. ³³ Secondly, guided by the principles of human-centered design, the system must demonstrate contextual adaptability, tailoring its interface and language to diverse use cases, from hospital discharge planning to community rehabilitation. ³⁴ Furthermore, although the underlying RAG framework is transferable across healthcare settings, implementation in other countries or regions would require localization of the knowledge base to reflect local clinical guidelines, healthcare practices, and patient needs. Such localization is essential to ensure the relevance, accuracy, and contextual appropriateness of the information provided. Thirdly, safety assurance needs to go beyond initial testing to include the option of localized, hospital-based deployment to better protect patient privacy and data security. ³⁵ It also needs continuous post-deployment monitoring, which involves tracking retrieval accuracy, detecting outdated content, and flagging high-risk queries for expert review. ³⁶ Ultimately, sustained user trust hinges on algorithmic transparency,^34,37 such as the consistent citation of information sources, and a demonstrable responsiveness to user feedback.

There are some limitations. First, inter-rater reliability analysis involved only three experts, resulting in wide ICC confidence intervals; future studies should include larger evaluator panels. Second, we did not formally assess fairness or accessibility across sociodemographic groups with varying health literacy levels, which remains an important direction for future research. Third, usability testing was conducted in a single controlled setting with 20 participants from one center, limiting generalizability and precluding assessment of long-term user behavior changes. Fourth, while RAG improved accuracy and completeness, it occasionally reduced understandability due to longer or more technical responses. Future research should evaluate long-term outcomes in real-world settings and explore strategies to balance readability with factual accuracy.

5. Conclusion

This study shows that combining RAG with LLMs can improve the quality of responses to stroke-related questions. Usability testing further indicated strong user acceptance among stroke patients and family caregivers, supporting the early feasibility of the system’s performance and usability. The trade-off between informativeness and understandability seen in some RAG outputs, on the other hand, shows an important area for future improvement. Continued research is needed to enhance linguistic accessibility and assess long-term clinical benefits in real-world caregiving contexts.

Supplemental material