From Theory to Practice: Development and Evaluation of a Quality Improvement Curriculum for Psychiatry Residents

Introduction

Quality improvement (QI) is a methodology commonly used to assess and improve the performance of healthcare systems. ¹ To equip physicians with the skills to support healthcare improvement efforts, regulatory bodies for residency training in the United States, Canada, and the United Kingdom now require trainees to receive training in QI.^2–4 Despite the wealth of studies describing how QI can be taught, the theoretical foundations of what factors lead to effective QI curricula are yet to be routinely integrated into curricular development. ⁵

This theory-practice gap was explored by Brown et al in their realist review of QI in medical education which developed a programme theory: a model to explain how QI educational programs lead to positive outcomes. ⁵ A programme theory can be used by healthcare educators to rapidly identify which strategies in QI education will provide the greatest value. For example, a subanalysis of Brown et al’s realist review which focused exclusively on psychiatry trainees identified that situating QI projects halfway through residency was optimal because it ensured residents had (a) adequate knowledge of their health system to identify quality gaps and (b) enough time remaining in their programs to implement QI projects during longitudinal ambulatory care blocks. ⁶

The present study describes the implementation and evaluation of a new psychiatry QI curriculum in a medium-sized institution. We describe how theoretical principles of effective QI education identified by Brown et al were used to develop the curriculum, and thus, help to close the theory-practice gap of education. ⁵ This approach can be replicated by educators and researchers in any medical specialty to rapidly translate theoretical principles in QI education into on-the-ground advances.

Methods

Curriculum Design

Two authors (AN and DF) reviewed Brown et al's programme theory of QI education to identify curricular mechanisms used in effective QI curricula. ⁵ This refers to the strategies used to implement the curriculum, such as giving learners protected time for QI training or providing mentorship in QI. As depicted in Table 1, we deliberately applied each mechanism to the local institutional context. Some of the mechanisms studied by Brown et al could not be implemented due to a lack of local resources or infrastructure. For example, some QI curricula include structured experiential QI projects for learners or access to system-level quality measures that can be analyzed. These mechanisms were excluded from Table 1 and are explored in greater detail in Brown et al's review. ⁵

OPEN IN VIEWER

Table 1.

Mechanisms to support curriculum implementation.

Mechanism	Current implementation and rationale
Clear expectations of learners	Learners are expected to produce a “project proposal” by the end of the workshop. The proposal includes clear expectations such as an aims statement, quality measures, and a description of at least one plan-study-do-act cycle.
Protected time	Quality improvement (QI) teaching time is protected for all residents during the weekly academic day.
Mentorship and feedback	Residents can receive just-in-time feedback on all components of their project charter during the workshop. A supervising staff psychiatrist with an interest in education and QI is available to field questions about quality gaps and change ideas.
Curriculum leadership	The curriculum is championed by the program director of the psychiatry residency program and chair of the residency curriculum committee.
Incentives for participating	Residents interested in pursuing further work in QI can implement the project proposal, which will fulfill their residency research requirements.
Content delivery strategies	The workshop is taught using just-in-time learning through a combination of didactics, group discussion, and five breakout spaces for small groups to work on project proposals.
Education resources	Residents have access to workshop slides and their project proposals following the workshop.
Screening projects for appropriateness	Residents are taught how to select a “good” quality gap using decision matrices during the workshop. Workshop leaders elicit multiple quality gaps from trainees and help trainees choose a gap that has the potential for high clinical impact with a low workload burden. There is no formal preworkshop screening for projects.
Working in teams	Residents can work in pairs. This allows residents to receive feedback from a peer while being involved in every stage of developing a QI project proposal.

The content of the curriculum was drawn from the author expertise and national QI training objectives. ³ Previously published QI curricula were referenced to ensure that no significant conceptual areas were missed.^7–9 Overall, we designed a single-day 2.5-h curriculum combining didactic seminars and QI simulation (Supplement 1). Trainees had five separate 10- to 15-min didactic presentations with breakout sessions in between. During these presentations, they learned the definition of QI, the dimensions of quality, the components of a QI project based on the Model for Improvement, ¹⁰ and the differences between QI and clinical research or quality assurance. In each didactic session, the specific relevance of QI to psychiatry was highlighted. For example, residents learned about the dimensions of quality by discussing specific quality problems they encounter during routine psychiatric practice. In breakout sessions, trainees worked in groups of 2 to 3 to apply newly gained knowledge to develop their own psychiatry QI project charter (Supplement 2). This proposal included the identification of a clinical quality gap, an analysis of its root causes, and a proposal for how it may be addressed using the Model for Improvement. Just-in-time feedback during each breakout session from the authors (AN and MB) helped attendees improve their proposals or consider new perspectives. At the end of the session, trainees reflected as a group, shared what they learned from the experience, and determined future directions for the implementation of the draft proposals.

Results

The curriculum was delivered on September 1, 2021, to 5 PGY 4 residents and on December 1, 2021, to 7 PGY 2 residents. Of these individuals, 11 out of 12 (92%) participated in the curriculum evaluation. Most participants had previously encountered QI, though only 3 (27%) had prior involvement in its development or implementation (Table 2).

OPEN IN VIEWER

Table 2.

Prior participant experiences with quality improvement (QI).

Prior QI experience	Number of attendees (percent)
I have been involved in the development and/or implementation of a QI project.	2 (18%)
I have participated in a QI project without being directly involved in its development or implementation.	1 (9%)
I have heard/seen QI projects being developed and/or implemented in my workplace without having participated in the initiative in any capacity.	4 (36%)
None of the above apply to me.	4 (36%)

Residents proposed six unique project charters. Aims of these projects included efforts to:

Increase the uptake of psychiatry consult service order recommendations by inpatient teams;

OPEN IN VIEWER

Table 3.

Summary of outcomes.

QIKAT-R questions 1 and 2
Preworkshop average score (standard deviation)	Postworkshop average score (standard deviation)	Gain in score	p-value
Average QIKAT-R scores for questions 1 and 2
4.45 (2.15)	7.00 (1.07)	2.55	<.01
BASiC-QI Beliefs and Attitudes subscale summary
47.27 (9.87)	52.82 (5.86)	5.55	<.01

QIKAT-R, Quality Improvement Knowledge Application Tool Revised; BASiC-QI, Beliefs, Attitudes, Skills, and Confidence in Quality Improvement.

Finally, participants completed the BASiC-QI subscale for belief and attitudes. Aggregate scores for the pre/postgroups demonstrated a significant difference, with an average improvement of 5.55 points after the intervention.

Discussion

By participating in this curriculum, residents acquired greater QI knowledge and developed more positive beliefs toward QI. Through curriculum creation and evaluation, this study demonstrates an example of how theoretical principles in QI education can be applied to develop an effective curriculum, reducing the theory-practice gap. In the following discussion, findings will be contextualized by the available literature.

Knowledge outcomes of this curriculum compare favorably with other published psychiatry QI curricula. For example, a 10-month long, mixed didactic and experiential QI educational intervention for third-year psychiatry residents led to average QIKAT-R scores increases of 5.6 to 6.8 per scenario. ¹³ Of note, this study also used clinical scenarios that were created specifically for psychiatry trainees. ⁷ Specialty-specific scenarios may allow residents to more deeply analyze quality gaps that are relevant to their own clinical experiences. The developers of the QIKAT-R identified that “excellent” scores fell in the range of 6.7 to 8.5 on standardized scenarios. ¹¹ Outside of the psychiatry literature, an otolaryngology program delivered a 1-year long didactic QI curriculum based on online Institute for Healthcare Improvement modules, leading to a mean QIKAT-R score improvement of 4.64 to 7.38 per scenario. ¹⁴ Similarly, an Emergency Medicine program implementing a 3-year, modular QI curriculum found that mean QIKAT-R scores increased from 4.3 to 6.1. ¹⁵ In the present analysis, participant scores improved from 4.45 to 7.00 points, suggesting that they obtained a similar improvement in knowledge to previously published curricula.

Beliefs and attitudes toward QI are key outcomes of QI education to predict future interest in completing QI projects, though they are inconsistently measured in the QI literature. ¹¹ Participants in this curriculum gained 5.55 points on the BASiC-QI subscale. There is no standard interpretation of this magnitude of change in the literature. However, one study of a 4-month, mixed didactic and experiential QI curriculum for medical students led to an increase of 4.8 points on this subscale, ¹² comparable to the outcomes of the present curricula despite the intervention being a single half-day. Some curricula have utilized other scales for beliefs and attitudes such as the QI Confidence Instrument or Safety Attitudes Questionnaire, though no single scale is prominently used in the QI literature. ¹⁶ Further standardization of tools to measure beliefs and attitudes may support educators in comparing the effectiveness of QI curricula.

To our knowledge, this is the first article reporting the development process of a QI educational program based on a QI programme theory. By reviewing effective curricular mechanisms identified in the literature, we were able to rapidly choose and implement QI tools realistic to our institutional context. This process may be especially relevant for educators looking to improve existing QI curricula by identifying mechanisms they may be able to use to strengthen in their existing curriculum. In this study's curriculum, all mechanisms that could realistically be implemented into a single-day curricula identified in Brown et al's review were utilized. Next steps may include examining opportunities to implement more resource-intensive mechanisms, such as training faculty mentors in QI, designating a “QI champion” to lead the curriculum, and implementing experiential QI projects. ⁵

This program evaluation has several limitations to consider, the most significant of which is its smaller sample size, which is limited by the size of the psychiatry program itself. Due to the one-group quasi-experimental design of the evaluation and the small sample size, it is not possible to determine whether trainees attending this curriculum obtained strong improvements in their QIKAT and BASiC-QI scores through chance or other factors separate from the curriculum and its use of theoretical learning principles. Implementing the curriculum in multiple institutions across a greater time period would foster confidence in how this curriculum impacts participant knowledge, beliefs, and attitudes. Moreover, it is unclear whether improvements in attitude or knowledge about QI will persist over time. Evaluation of this would require follow-up assessment of participants—ideally 1 year after the intervention to be more comparable to other longitudinal curricula in the literature. Finally, the evaluation used a nonrandomized design, and therefore it is not possible to determine definitively which factors in the curriculum were most important.

Conclusions

This study demonstrates a worked example of how theoretical principles in QI education can be used to rapidly develop QI curricula, achieving comparable outcomes to other curricula published in the literature. This approach can be used by educators in any medical specialty to develop and improve their own QI curricula. However, further work is necessary to longitudinally evaluate whether resident knowledge and interest in QI translate into increased QI involvement following residency.

Supplemental Material

Supplemental Material

Supplemental Material

Supplemental Material