Abstract
Background:
As more medical schools implement longitudinal palliative care (PC) curricula, the need for robust evaluation that demonstrates curricular outcomes has become increasingly important.
Objectives:
The authors prospectively evaluated a PC curriculum at the University of Pennsylvania’s Perelman School of Medicine, using the Context, Input, Process, Product model.
Design:
Between 2021 and 2025, data that provided evidence of value were generated from a variety of sources and stakeholders to address evaluation questions about acceptability, student knowledge and attitudes about PC topics and engagement in PC opportunities. Both qualitative and quantitative data were collected with an assortment of tools including evaluation scores, focus groups, interviews, written reflections, quiz scores, and medical record notes.
Results:
Results of this evaluation indicated that the curriculum achieved the intended outcomes of acceptance to diverse stakeholders, improved student knowledge and attitudes about PC, and increased student engagement in PC. Unanticipated impacts of the curriculum were also captured.
Conclusions:
The evaluation provided multisource evidence of the value of a longitudinal curriculum in PC and may serve as an example for others charged with evaluating longitudinal PC curricula in medical school.
Keywords
Key Message
The authors prospectively evaluated a longitudinal palliative care (PC) curriculum for medical students using the Context, Input, Process, Product model. Results indicated that the curriculum achieved the intended outcomes of acceptance to diverse stakeholders, improved student knowledge and attitudes about PC, and increased student engagement in PC.
Introduction
Medical schools increasingly incorporate longitudinal curricular themes as part of a larger movement toward “integrated curricula” in medical education. Various definitions of an integrated curriculum have been proposed, but all such curricula include “vertical” integration, or longitudinal content, as well as “horizontal” integration, meaning integration of content across courses at a single timepoint.1–4 Longitudinal curricula allow medical schools to adhere to accreditation standards, which call for “coordinated and integrated content within and across academic periods of study” and provide opportunities to introduce new and emerging topics. 5 They also facilitate learning by asking students to retrieve previously learned information from memory (retrieval practice), introducing material when it is contextually relevant to their training (situated learning), and switching between multiple related topics rather than mastering a topic before moving on to the next topic (interleaving).6–8
Despite these advantages, longitudinal curricula can be difficult to administer. The scope of such curricula is wide, and the variety of learning settings and number of stakeholders pose challenges to those seeking to design and implement longitudinal programs. 9 Without robust evaluation data providing evidence of the value of longitudinal curricula, medical school leadership may be reluctant to allocate sufficient resources to support them.10,11 Evaluation experts recommend the use of comprehensive evaluation models that encompass not only data about outcomes but also information about context and process to guide curricular evaluation.12,13 However, longitudinal curricular evaluations described in the literature often lack comprehensive evaluation models.14–20 Researchers in fields with longitudinal curricula have advocated for evaluation that is prospective, uses robust evaluation models, and includes the perspectives of multiple stakeholders in order to guide durable curricular change.21–24
Palliative care (PC) educational content is increasingly recognized as a crucial component of medical student education, and many medical schools deliver PC content through elective or block formats.25–27 While several schools have implemented mandatory, longitudinal PC curricula, none have prospectively evaluated these programs using evaluation models and multiple sources of data from diverse stakeholders.17–20 Such robust evaluation is needed in order to sustain and grow longitudinal undergraduate PC programming.
The University of Pennsylvania’s Perelman School of Medicine (PSOM) implemented a longitudinal PC curriculum called CARE-7 in 2021, with concurrent evaluation using the Context, Input, Process, Product (CIPP) model. 28 The CARE acronym stands for “Caring, Attention, Responsiveness, and Empathy” and 7 references the educational goals of the curriculum, which correlate to proposed medical student PC competencies (Table 1). 29 The curriculum and parts of the evaluation that focused on Context, Input, and Process have been previously described. 30 In this article, we focus on Product, or outcomes of our curriculum as a whole, with data collected between 2021 and 2025. To that end, we address three outcome evaluation questions that we determined to be of high importance: (1) Is the curriculum acceptable to students, faculty, and other stakeholders?, (2) Are students gaining the knowledge and attitudes outlined in curricular objectives?, and (3) Is there increased student engagement in PC topics?
CARE-7 Goals and Objectives
Curriculum Description and Evaluation Model
PSOM’s medical education program is a four-year program with an average class size of 150 students; CARE-7 spans all four years and consists of discrete sessions integrated into existing courses. The curriculum consists of 10 mandatory sessions (22 hours of content) taught by over 85 interdisciplinary educators. Two of these sessions, “Discussing Serious News” and “Pain and Symptom Management,” make up the PC Skills Practicum Day, a full day of PC content during the clerkship year. Two elective offerings are available for students who wish to gain additional experience (Table 2). CARE-7 emphasizes a lifelong learning approach to the material with an intentional focus on formative rather than summative assessment.
CARE-7 Curricular Sessions
In its first few years, CARE-7 was donor-funded and supported by three directors who oversaw curricular design and implementation, three coaches who served as educators and mentors, and an administrator. A separate curriculum advisor acted as a liaison to the Dean’s Office. A curriculum committee consisted of the directors, advisor, and PSOM faculty with experience in curriculum evaluation; the authors of this article consist of the members of the committee.
We chose to evaluate CARE-7 using the CIPP model, an evaluation model that considers the curricular environment (Context), designs content based on the environmental context (Input), monitors early rollout (Process), and evaluates the outcomes (Product) while emphasizing stakeholder involvement and a continual improvement approach.28,31 We were also guided by the program evaluation standards of utility, feasibility, accuracy, and fairness. 32 Thus, we strove to capture data that would be feasible to collect, provide accurate information, involve diverse stakeholders, and provide actionable information to improve the curriculum.
Curricular Outcomes
A variety of methods were used to evaluate outcomes for our curriculum. For each of our three evaluation questions below, we describe the methods used and summarize the findings. We also describe several unexpected impacts of CARE-7. Our evaluation protocol was submitted to and deemed exempt by the Penn Institutional Review Board.
Is the CARE-7 curriculum acceptable to students, faculty, and other stakeholders?
We addressed this question through student focus groups, evaluations, and stakeholder interviews. For clarity, we use the term “session” to refer to curricular sessions, as described in Table 2.
Student focus groups
Methods
We held 5 session-specific focus groups covering 7 of the 10 mandatory sessions from 2021 to 2023. Because third and fourth year students had received most of the curriculum by 2025, we added two more focus groups to capture their experience of the curriculum as a whole (Table 3). We provided question guides (Supplementary Appendix) to focus group facilitators, who were a combination of student volunteers, CARE-7 directors, and members of the curriculum committee. We collected deidentified notes for all focus groups. While no formal qualitative analysis was done, major takeaways and representative or noteworthy quotes (italicized in the findings section) were added to the evaluation database.
Student Focus Groups
Findings
In terms of curricular content, students reported that CARE-7 topics were high-yield, particularly noting pain and symptom management and end-of-life care as highly practical subjects. They shared that CARE-7 helped them “level up” in their communication skills. While some students were skeptical about the use of communication frameworks early on, they agreed that “once you are faced with these conversations in the clinical setting, it becomes very real, and a lot of folks have realized how helpful frameworks and practice can be.” Students said that they “couldn’t imagine having medical school without the curriculum,” primarily because of the knowledge and skills they gained about how to respond to and process patients’ suffering. They also felt that PC content included “requisite skills for being a physician” and that there was nowhere else in medical school where they would have learned these skills.
In terms of curricular structure, students valued having PC as one of their first clinical experiences in medical school because it modeled humanistic, interdisciplinary care in a welcoming setting. Nearing the end of medical school, students shared that CARE-7 had a clear through line that made sense to them.
Student evaluations
Methods
PSOM collects standard evaluations for many of its sessions. We requested evaluation data for all CARE-7 sessions where such data were available. Sessions are rated on a 5-point rating scale of 0—“Poor,” 1—“Fair,” 2—“Good,” 3—“Very Good,” and 4—“Excellent.”
Findings
For the eight mandatory CARE-7 sessions where evaluation data were collected, including the two sessions that make up the PC Skills Practicum Day, CARE-7 sessions were highly rated, ranging from 3.28 to 4.00 (Table 4). The PC Skills Practicum Day as a whole also received an evaluation score and was consistently one of the highest rated of the seven total practicum days. The average PC Skills Practicum Day score was 3.30 (N = 352) for the years 2023–2025, which was higher than the average practicum score of 3.12 (N = 1555) for the same period. In general, CARE-7 sessions were rated close to or above the average for comparable sessions at PSOM.
Student Evaluations of CARE-7 Sessions
Scores are based on a 5-point scale: 0—“Poor,” 1—“Fair,” 2—“Good,” 3—“Very Good,” and 4—“Excellent.”
PC, palliative care.
Stakeholder interviews
Methods
CARE-7 directors conducted informal interviews with CARE-7 educators to obtain feedback on specific sessions, both after initial implementation and following significant changes. Interviews were generally unstructured, though an interview guide was available for reference (Supplementary Appendix). We also solicited feedback from directors of medical school courses in which CARE-7 sessions were embedded. We provided annual updates to the CARE-7 donor and received feedback via email. In January 2025, we met with the co-chair of the Patient and Family Advisory Council and the program manager for Patient and Family Centered Care at our hospital to obtain feedback from stakeholders representing patients and caregivers.
Findings
Based on the feedback from interviews, educators appreciated the opportunity to teach CARE-7 sessions and believed that sessions were of high quality. While the lack of protected time for teaching was a concern, educators expressed gratitude for being involved in meaningful work. Directors of other courses in which CARE-7 sessions were embedded expressed positive regard for CARE-7, and some attributed changes in the learning environment to CARE-7. For example, medicine clerkship directors heard less distress from students around difficult clinical scenarios once the session on “Coping with Serious Illness and End-of-life” was implemented. The co-chair of the Patient and Family Advisory Council, the program manager for Patient and Family Centered Care, and the CARE-7 donor all valued the curriculum’s emphasis on lifelong learning, communication skills, and humanistic care of patients.
Are students gaining knowledge and attitudes outlined in curriculum objectives?
We addressed this question through quiz analysis, review of electronic medical record (EMR) notes, and analysis of student reflections.
Quizzes
Methods
We administered quizzes on the topic of pain management in serious illness before and after a recorded didactic on the topic, as preparation for the session on “Pain and Symptom Management.” Quizzes consisted of six multiple-choice questions. A post-test consisted of six separate questions that tested similar teaching points and were estimated by curriculum directors to be comparable in difficulty. We collected and compared mean quiz scores using t-tests for three classes of students from 2022 to 2024.
Findings
In total, 759 quizzes were collected, including 386 pre-quizzes and 373 post-quizzes. There was a statistically significant increase in the mean quiz scores from 3.16 (53%) to 3.90 (65%) (p < 0.001), demonstrating knowledge acquisition.
Medical record notes
Methods
The “PC Assessment” in the Surgical Clerkship required that students complete a comprehensive PC assessment with a patient with a serious illness and write a note in the EMR summarizing findings and making pertinent recommendations. The six domains that students were asked to assess included pain, nonpain symptoms, function, psychosocial assessment, spiritual assessment, and medical knowledge/decision making. Students received a note template that included a drop-down menu with possible recommendations in the plan section. We reviewed all 161 medical student notes from 2023 after de-identifying student and patient information, noting how many PC domains each student evaluated and tabulating their recommendations.
Findings
Of the 167 students enrolled in the surgical clerkship, 96% (N = 161) submitted completed EMR notes. Notes showed that students generally assessed the recommended PC domains and suggested medically appropriate interventions to improve the care of their patients. Of the 161 notes reviewed, 63% of students assessed all six PC domains, 30% assessed five domains, and 7% assessed four or fewer domains. Many students made multiple recommendations in their plans. In total, there were 89 recommendations to continue conversations about goals, values, and preferences as the clinical course evolves; 52 recommendations to improve pain or symptom management, 29 to consult PC; 30 to consult pastoral care; 15 to involve social work; and 19 for no follow-up needed. While those recommendations were from the templated options, there were also smaller numbers of student-written recommendations; for example, eight notes included recommendations for ways to improve communication.
Analysis of student reflections
Methods
Two CARE-7 sessions, the “Palliative Medicine Experience” in the pre-clerkship phase and the “PC Assessment” in the clerkship phase, required written reflections. We took a general inductive approach to analyze a sample of de-identified reflections. 33 Sixty-eight “PC Assessment” reflections were collected over 10 months in 2022 and 2023; 138 “Palliative Medicine Experience” student reflections were collected over 3 months in 2023. All reflections were coded independently by two to three investigators (composed of learners and co-directors), and topic areas were derived from clusters of coded data.
Findings
Details of the various topic areas are described elsewhere; here, we discuss take-home points from our inductive analysis. 34 Students reported a greater understanding of the content and substance of PC as a field, including its interdisciplinary nature, after the “Palliative Medicine Experience.” As this was one of their first clinical experiences in medical school, students wrote that the experience prompted them to reflect on systems-level considerations in medical care as well as consider their own professional identity and find meaning in medicine.
Students used “PC Assessment” reflections to further reflect on the patient and family experience of serious illness and appreciated that the assignment allowed them to get to know their patients in more depth. They shared that it was emotionally challenging to delve into their patients’ experiences of serious illness; nevertheless, students saw it as a privilege and shared that it highlighted the importance of core PC skills. Taken together, these student reflections demonstrate not only knowledge about PC topics but also an appreciation for the benefits of PC, both as a specialty and a set of skills that can add value to one’s medical practice.
Is there increased student engagement in PC topics?
We addressed this question through monitoring of students participating in CARE-7 electives.
Elective monitoring
Methods
CARE-7 electives consisted of a PC area of concentration (AOC) initiated in 2022 and a two-week clinical elective on an inpatient PC consult service for third and fourth year students. AOC and elective requests and completion rates were collected by CARE-7 directors. At the time of enrollment, AOC students completed a brief survey of demographic information, interests in medicine, and future training plans; they completed an exit survey when they finished the AOC.
Findings
Nine students enrolled in the AOC in its first three years. Of these, three felt it was likely they would pursue a PC fellowship after residency. The PC clinical elective was the only session that predated the implementation of CARE-7. In the seven years prior to CARE-7 implementation, the elective averaged 12 students per year. From 2021, when CARE-7 began its rollout to 2024, the elective averaged 18 students per year and declined an average of 6 students per year due to the lack of space. In part, the increase in students may be due to the addition of two elective sites in 2021.
Unexpected impacts
In addition to the intended outcomes reported above, CARE-7 had several unexpected impacts both within and beyond our health system. First was the founding of a new health-system-wide communication skills training program; because we trained so many faculty to teach communication skills through CARE-7, we were able to extend training sessions to practicing clinicians across our health system. Second, CARE-7 was used by the medical school for recruitment and promotional purposes. Prospective students mentioned CARE-7 as a unique asset during interviews, and some students said it was one of the major draws to PSOM. Finally, early dissemination of our curricular evaluation enabled us to support PC educators at other institutions; one institution shared that they implemented a new PC AOC modeled after the CARE-7 AOC.
Discussion
Comprehensive evaluation data for longitudinal PC curricula are scarce; our curriculum outcomes dataset helps address this gap by describing our prospective, multisource evaluation of the outcomes of a longitudinal PC curriculum and providing evidence of its value to diverse stakeholders.35–39 Interviews, focus groups, and evaluations speak to a curriculum that is valued highly by students, educators, and other stakeholders. Review of student work in the form of quizzes, medical record notes, and reflections demonstrates that students gain intended knowledge and attitudes. PC electives show robust student engagement. Finally, a review of unexpected impacts of CARE-7 shows that its influence expanded beyond PSOM to affect educational programming for practicing clinicians in our health system and for other medical schools.
Our outcomes data builds on previous studies by highlighting the importance of the longitudinal nature of the curriculum. Notably, students shared that their relationship with the curricular content changed over time; while they felt that early integration of CARE-7 content was essential, many students acknowledged that it was not until the later years of training that they fully recognized its value. Students also commented on the utility of different teaching modalities at different times in their training; for example, they appreciated the use of videos rather than simulated patients in earlier communication sessions, as they felt this to be less stressful, but valued hands-on skills practice with a simulated patient in their fourth year. Thus, the longitudinal nature of the curriculum allowed for reinforcement of knowledge and skills in a developmentally appropriate manner and allowed students to more fully process the content as they matured as clinicians.
We acknowledge several limitations in our data. First, quantitative evaluation scores were not available for all CARE-7 sessions, and response rates were not available to explain variation in evaluation numbers. Next, we acknowledge the lack of objective skills measurements such as an Objective, Structured, Clinical Examination, while also keeping in mind that we made deliberate decisions to prioritize formative assessment. We also acknowledge that some interviews and focus groups were run by CARE-7 directors, who admittedly had a stake in the success of the curriculum. At the same time, those individuals were knowledgeable about CARE-7 sessions and thus were agile facilitators. Finally, we did not talk to patients or families themselves but were in contact with patient/family advocates in our health system.
Lessons learned in our experience of evaluating an integrated PC curriculum across four years of medical school include the utility of being guided by an evaluation model that focused on the process of curricular development as well as outcomes, attention to evaluation standards, and the involvement of multiple stakeholders in the curricular design and evaluation process. Taken together, we provide multisource evidence of the value of a longitudinal curriculum in PC and an example for others hoping to evaluate such curricula.
Footnotes
Acknowledgments
The authors express their special thanks to the many people involved in CARE-7 development, implementation, and evaluation, including the following people who aided in outcomes evaluation: Bridget Durkin, MD, MBE; Ann Soliman, MD; Caitlin Merley, MD; Danielle Feffer, MD; Lindsay Kumble, MD; Aidan Crowley; Lillian Wieland; and Suzanne Rose, MD, MSEd.
Author Disclosure Statement
The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding Information
The CARE-7 curriculum is generously funded by a Penn Medicine Board member and spouse of the late Henry A. Jordan, MD, a Penn alumnus (M’62 from the School of Medicine).
Supplemental Material
References
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
