Assessing Clinical Reasoning in Hospice and Palliative Medicine Fellows

Introduction

Clinical reasoning is an essential skill in medical education ¹ and is defined as the cognitive process of evaluating and managing a patient’s medical problems. ² Within graduate medical education, there is increasing recognition of the need to explicitly teach and assess clinical reasoning.^1,3,4 Clinical reasoning includes diagnostic reasoning, the process of identifying what disease a patient has ⁵ ; and management reasoning, the process of making decisions about patient management, including choices about tests, treatment, allocation of resources, and follow-up. ⁶ Prior work suggests that diagnostic reasoning is focused earlier in medical training while management reasoning becomes increasingly prominent as trainees progress. ⁷

In hospice and palliative medicine (HPM), clinical reasoning is especially important due to the potential uncertainties that arise in diagnosis, prognosis, and treatment pathways for patients with serious illness and the goal to align our support and treatments with patients’ values and priorities. However, there is a dearth of structured approaches to assess clinical reasoning skills for HPM trainees.^4,8

Workplace-based assessments (WBAs) are assessment tools for advanced trainees that facilitate feedback at the “does” level based on the extended Miller’s pyramid.^9,10 Current WBAs specific to HPM focus more on whether specific content is collected,^8,11,12 whereas assessing clinical reasoning requires understanding the learner’s cognitive process. While recent literature emphasizes the need for more robust assessment strategies in HPM fellowship training, ⁴ clinical reasoning assessment remains underrepresented.

The Assessment of Reasoning Tool (ART), developed in 2018, is a WBA focusing on diagnostic reasoning. It assesses five domains: hypothesis-driven data gathering, constructing a problem representation, forming a differential diagnosis, developing a plan with prioritization, and reflecting on cognitive biases. ¹³ The ART has demonstrated efficacy and feasibility in assessing clinical reasoning and facilitating structured faculty feedback for pediatric interns. ⁵ To the best of our knowledge, the ART has not been applied to more advanced medical learners such as HPM fellows. Of note, assessment tools for management reasoning are in development,^6,14 but none have been validated to date.

This study evaluates the feasibility of implementing the ART in HPM fellowship training and explores its impact on clinical reasoning assessment using a mixed-methods approach.

Methods

Survey design

We created anonymous online surveys with quantitative and open-ended questions distributed via Qualtrics. The survey underwent iterative review by the study authors and a content expert. We also conducted cognitive interviewing and pilot testing with an HPM fellow–faculty pair to strengthen evidence for validity. ¹⁵ The survey evaluated six aspects of clinical reasoning via 4-point Likert scale items (1 = least favorable, 4 = most favorable); domains are detailed in Table 1. The post-intervention survey also inquired about the number of uses needed for an HPM fellow or faculty member to develop comfort with the ART, how much time was added to clinical teaching rounds by using the ART, and barriers to use.

OPEN IN VIEWER

Table 1.

Summary of Pre- Versus Post-ART Survey Responses from HPM Fellows and Faculty

Fellow	Pre-ART (n = 6)	Post-ART (n = 16)	p Value b
Frequency of verbalizing CR a	3.33	3.44	0.65
Frequency of CR feedback a	2.33	3.25	0.05
Satisfaction with CR feedback a	2.67	3.44	0.15
Frequency of self-reflecting for bias in CR a	2.67	2.81	0.02
Comfort level with CR terminology a	2.83	3.19	0.76
Time spent on CR feedback (mean total minutes)	2.92	5.28	0.35
Faculty	Pre-ART (n = 7)	Post-ART (n = 7)	p Value b
Frequency of verbalizing CR a	2.43	3.14	0.12
Frequency of CR feedback a	3.00	2.75	0.61
Satisfaction with CR feedback a	3.00	2.71	0.61
Frequency of self-reflecting for bias in CR a	2.14	1.88	0.63
Comfort level with CR terminology a	1.43	1.88	0.70
Time spent on CR feedback (mean total minutes)	3.75	5.25	0.12

a

Mean score on 4-point Likert scale, 1 = least favorable and 4 = most favorable; descriptors varied by question.

b

Wilcoxon signed-rank test.

ART, Assessment of Reasoning Tool; CR, clinical reasoning.

Results

Pre-intervention versus post-intervention

Fellows reported significant increases in the frequency of feedback on clinical reasoning (2.33 vs. 3.25; p = 0.05) and self-reflection for cognitive bias (2.67 vs. 2.81; p = 0.02). They reported no difference in frequency of verbalizing clinical reasoning, satisfaction with clinical reasoning feedback, comfort level with clinical reasoning terminology, and time spent on clinical reasoning feedback. In contrast to fellows, faculty did not report any significant changes between pre- and post-intervention survey responses (Table 1). Overall, fellows somewhat agreed (12 responses [75%]) or strongly agreed (four responses [25%]) that the ART was perceived as effective in assessing clinical reasoning, whereas faculty responses ranged from somewhat disagree (four responses [50%]) to somewhat agree (three responses [38%]).

Post-intervention surveys also showed that all fellows (100%) felt comfortable with the ART within the first week of use, while 50% of faculty responses showed they never became comfortable with the ART. In terms of perceived time added by the ART to the case presentation/teaching during rounds, 50% of fellow responses reported no added time, while faculty responses showed up to 10 minutes were added (Fig. 1). The most frequently cited barriers to ART use were lack of time (eight fellows [50%] and six faculty [75%]) and clearer guidance on how to use the ART (five fellows [31%] and two faculty [25%]).

OPEN IN VIEWER

FIG. 1.

Perceived time added to clinical teaching rounds by using the ART. ART, Assessment of Reasoning Tool.

Qualitative results

Qualitative post-intervention surveys (four fellow and six faculty responses) revealed three themes: focus on diagnostic reasoning when most HPM case scenarios emphasize management reasoning; limited applicability beyond symptom-focused case scenarios such as clarifying goals of care; and a need for clearer guidance on how to use the ART effectively. Table 2 shows representative quotes for each theme.

OPEN IN VIEWER

Table 2.

Themes and Representative Quotes from Qualitative Analysis

Theme	Representative quote
ART focuses more on diagnostic reasoning than management reasoning	Fellow: “It is geared toward diagnostics which is not always the focus of our consults” Faculty: “This does help the fellow broaden his/her differential and think of other reasons that might contribute to the symptom, which helps broaden the management.” Faculty: “ART might be effective in assessing internal medicine residents’ ability to formulate a differential diagnosis, to prioritize which items are most likely and which items the learner cannot miss…Often times, a differential diagnosis is already formulated by the primary team. I think it is less useful to go through the differential dx of abdominal pain in a patient with stage IV cancer with mets to the abdomen...HPM fellows should focus their clinical reasoning not on “what” but more on “why” they are choosing a management strategy? Why did you choose a fentanyl patch? What patient factors, labs, national shortages, demographics, etc contributed to that decision?”
ART has limited applicability to HPM case scenarios	Fellow: “I think the tool needs to be tweaked for palliative specific cases.” Faculty: “I think some of the language could be altered to better fit within the HPM framework. For example, it can be hard to capture total pain evaluation and management with this tool as is.” Faculty: “We only had 2 days when we used it... 3 days when [new consults] were only Goals of care.”
There needs to be more guidance on how to use the ART effectively	Fellow: “It would be helpful to understand what ART is specifically so that I can see when/how it is used.” Faculty: “Change is hard, especially on a busy service where my priority is seeing patients…Unfortunately, I was not able to gain that comfort this week.”

ART, Assessment of Reasoning Tool; HPM, Hospice and Palliative Medicine.

Discussion

We found that the implementation of the ART, a published tool for assessing diagnostic reasoning, in HPM fellowship training was feasible. Fellows perceived improvements in certain aspects of clinical reasoning feedback and reflection. However, faculty did not report similar benefits, and several limitations constrain the tool’s applicability for HPM faculty and fellows.

The improvement in the frequency of feedback on clinical reasoning and self-reflection for cognitive bias among fellows suggests the ART provides an opportunity for fellows to receive feedback and reflect in the diagnostic reasoning domain. The HPM fellows may have been more engaged knowing that they were being assessed at the “does” level of the extended Miller’s pyramid, where competence transitions to entrustment for independent practice.^10,16,17 Moreover, structured WBAs such as the ART help establish shared mental models between learners and faculty, which in turn promotes meaningful feedback. ¹⁸

In contrast, faculty did not perceive a meaningful improvement in assessing HPM fellows’ clinical reasoning with the ART. We noted that the type of clinical reasoning being assessed may not have aligned optimally with the learner stage of a fellow. The ART focuses on diagnostic reasoning. ⁵ However, much of the clinical reasoning for HPM clinicians and thus HPM fellows often centers more on management reasoning. ⁶ Several qualitative comments echoed this distinction, pointing out that HPM training often occurs in the context of developing a plan rather than establishing a diagnosis. Nonetheless, one faculty commented on the benefit of developing a broader differential diagnosis, which remains a valuable skill, particularly for early fellows.

A second explanation for the faculty perception is the limited application of the ART across all HPM case scenarios. Based on feedback from the pilot study, faculty were guided to use the ART for cases of uncontrolled symptoms. Faculty provided qualitative comments regarding this limitation, reflecting their broader clinical experience and consideration of how the ART might not be applicable to non-symptom-based HPM case scenarios.

A third explanation is that social desirability bias may have inflated the fellow ratings, considering the small sample size. This is less likely, as our findings reflect prior work showing that trainees report greater confidence in their program’s ability to assess clinical reasoning than the faculty express. ⁴ Actively engaging fellows during faculty development could further improve the quality of the shared mental model and potentially increase the perceived impact of the ART by faculty.

A fourth explanation is that faculty may have experienced ceiling effects in self-assessed outcomes given their established baseline assessment practices. This is supported by the pilot survey data, which showed that most faculty rated existing clinical reasoning assessment tools (e.g., chart stimulated recall, postencounter note, script concordance tests) as somewhat or very effective. Considering the additional resources associated with learning a new WBA, the benefit of the ART may not have been perceived as significant compared to prior assessment tools.

While both fellows and faculty perceived no to minimal added time to clinical teaching rounds, “lack of time” remained the most frequently cited barrier to using the ART. This paradox may reflect the cognitive load associated with using assessment tools. ¹⁹ Ad hoc analysis of paired observation data showed repeated ART use did not lessen the perceived time added. Moreover, despite a dedicated faculty development session, qualitative data from faculty revealed a lack of comfort and uncertainty about the ART—consistent with literature showing that one-time faculty development sessions have limited impact. ²⁰ The fact that 85% of faculty shared a lack of comfort with WBAs during the pre-intervention survey highlights the ongoing challenge of effective faculty development in the era of competency-based medical education. ²¹ The request for clearer guidance from both fellows and faculty reinforces this idea and aligns with prior research that faculty often experience discomfort and low confidence when implementing new WBAs without calibration. ²² It is important to recognize that faculty are the key to any assessment tool’s effectiveness. ¹⁶ Achieving consistent assessment of clinical reasoning—both diagnostic and management—will require ongoing faculty development and broader cultural shifts within HPM.

Limitations of this study include the single-site design and small sample size. Repeated weekly responses from the same participants were included in inferential analyses, which may have introduced non-independence that could have inflated the statistical significance. Completed ART forms were not collected or analyzed. This may have clarified the scope of HPM case scenarios appropriate for the ART.

In conclusion, implementation of the ART in HPM fellowship training is feasible, and fellows perceived improvements in feedback and reflection on their clinical reasoning. However, the absence of faculty-perceived benefit, persistent discomfort despite dedicated faculty development, and the tool’s focus on diagnostic reasoning raise important questions regarding its application to HPM with a cost–benefit tradeoff. In its current form, the ART may be better suited as an early-fellowship adjunct rather than a standard assessment tool in HPM fellowship training. Further research, particularly on faculty development and assessment tools for management reasoning, would help define the optimal role for the ART.