Assessing the Implementation of a Novel Digital Measurement-Based Care Training in Behavioral Health Using the PRISM Framework

Abstract

Background: Measurement-Based Care (MBC) is an evidence-based practice that has demonstrated challenges integrating into care settings even under ideal circumstances. This study uses the PRISM framework to critically examine a systematic integration of MBC as standard behavioral healthcare in an adult ambulatory psychiatric clinic. Since the initial implementation in 2018, 33 distinct implementation strategies were used to enhance the uptake of this practice. To prepare for sustainment, our team developed a digital Measurement-based care Training for Resilient Implementation in a Clinical setting (METRIC) to improve implementation. The present study evaluates the reach, implementation, and contextual domains impacting METRIC. Methods: METRIC was IRB-approved and distributed to 56 multidisciplinary clinicians through the institutional learning hub. Providers completed surveys about implementation and clinician attitudes of METRIC before, 1 month, and 3 months after training. Patient attitudes were assessed before (n = 98) and after (n = 80) implementation. Clinician and patient attitudes were analyzed using the Friedman and Wilcoxon tests, respectively. Results: Forty-seven out of 56 (83.9%) clinicians completed METRIC. Implementation measures suggest there were promising acceptability, appropriateness, and feasibility. Clinicians positively rated METRIC. Clinicians answered 84% of module questions correctly, though their attitudes about MBC did not significantly change after METRIC. Patient attitudes did not significantly change. Conclusions: This study evaluated the implementation of METRIC in an ambulatory mental health clinic for its feasibility, acceptability, and appropriateness. After implementing strategies to enhance provider uptake, METRIC was piloted to standardize MBC training and prepare for sustainment. METRIC was well-received and provides an opportunity for less provider reliance on system champions and academic partnerships to sustain MBC utilization in mental health settings. Our findings define positive aspects of digital MBC training and areas for additional research, which include exploring the benefits of MBC training, examining need and frequency, and whether specialized training is beneficial.

Plain Language Summary

Use of digital training to improve use of symptom monitoring in outpatient psychiatry

Tracking patient symptoms in psychiatry with validated scales is important for improving clinician–patient communication and guiding care decisions. These scales are referred to as patient-reported outcome measures (PROMs), and using them repeatedly over time as part of decision-making for patient care is known as measurement-based care (MBC). Research supports the use of such PROMs and MBC; however, implementing them in outpatient psychiatry clinics has proven to be challenging. This article describes one clinic's strategies to standardize training in the use of PROMs and integrate MBC as a standard of care. A novel part of this effort toward sustaining PROM use was the creation of a digital training tool called Measurement-based care Training for Resilient Implementation in a Clinical setting (METRIC), which provided clinicians with knowledge about PROMs, patient simulations, interpretation of PROMs, and what to do in common situations encountered in the use of PROMs. Clinicians were asked questions about whether they felt METRIC were acceptable, appropriate, and feasible. Patients were asked about satisfaction with their care, and results were compared before and after clinicians were trained with METRIC. There was no change in attitudes for patients or clinicians after clinicians completed METRIC, where clinician attitudes remained positive before and after training. This study shows that METRIC was successfully implemented in an outpatient psychiatry clinic, provided standardized training to clinicians via video, and possibly helps promote the sustainment of the use of measurement scales. Additional questions generated by this study include determining what specifically is beneficial about this training, whether it needs to be administered more than once, and whether more specialized training is needed.

Keywords

Measurement-based care MBC patient-reported outcome measures training behavioral health

Background

Measurement-Based Care (MBC) is an evidence-based practice (EBP) designed to enhance communication between clinician and patient by measuring specific symptoms and tracking scores over time with the goal of improving treatment outcomes (Lewis et al., 2019; Scott & Lewis, 2015). Previous studies show evidence of improved patient outcomes from practicing MBC, which is based on a systematic evaluation of patient functioning, leading to informed clinical decision-making (Barber & Resnick, 2023; Øvretveit et al., 2017). The MBC leverages patient-reported outcome measures (PROMs) to collect data on symptoms, so patients and providers can collaboratively make more informed treatment decisions. The Collect, Share, Act (CSA) model provides a three-step process for clinicians to engage in MBC by selecting relevant measures, sharing results with patients via review of PROM data and outcomes, and acting by making treatment modifications. The CSA model ensures proper incorporation of PROMs and results into patient care (Barber & Resnick, 2023). Furthermore, in the context of limited mental health providers for the demand, successful MBC implementation promotes high-quality, patient-centered care. When used correctly, MBC serves to enhance the standard of care in ambulatory psychiatry (Zhu et al., 2021).

Implementation of MBC

Integrating MBC into care settings has proven challenging even under ideal circumstances. Previous studies have provided recommendations for systematic MBC implementation with guidance on how to engage systems, providers, patients, and partners to improve adoption and fidelity to MBC (Boyd et al., 2018; Dollar et al., 2020; Lewis et al., 2019). Robust bodies of literature have summarized the barriers and facilitators of this practice (Amini et al., 2021; Cheung et al., 2023; Wray et al., 2018; Yule et al., 2024). Most common barriers include lack of provider buy-in to the benefits of MBC, technological difficulties, lack of patient adherence, time burden, and limited organizational support. Conversely, systematic and ongoing organizational-level, provider-level, and patient-level supports are indicated as facilitators from healthcare professionals and staff (Resnick & Hoff, 2020; Ridout et al., 2023; Van Tiem et al., 2022; Yule et al., 2024).

Among indicated barriers, lack of formal training and professional guidelines are often identified as critical to successful implementation (Aboraya et al., 2018; Arbuckle et al., 2013; Boswell et al., 2023; Hatfield & Ogles, 2007). Despite its demonstrated need, formal training on the implementation of MBC is scarce. The first of its kind was developed by Aboraya in 2022 as a curriculum for psychiatry professionals. This comprehensive educational program provides guidelines for standardized treatment. This education is intensive and exhaustive and may not be scalable in low-resource and overburdened clinical settings.

There is evidence that digital and brief educational programs are feasible to implement and effective in reaching providers who are learning evidence-based practices (Camargo et al., 2020; Chick et al., 2020; Kim Thi Vu et al., 2023; Nkulu-Ily, 2023; Tudor Car et al., 2019; Zhang & Thompson, 2023). Concurrently, Lyon and colleagues evaluated the impact of a brief online MBC training plus consultation in varied frequencies (BOLT + PTC). Results demonstrated that BOLT + PTC significantly improved assessment use, though more consultation did not always improve the impact of training on MBC use. Unfortunately, these training programs were not publicly available at the time of this study.

Practical, Robust, Implementation and Sustainability Model

Through our prior collaboration efforts (Gatto et al., 2025 Kablinger et al., 2022; O’Brien et al., 2024; Ko et al., 2023), we systematically integrated MBC as standard behavioral healthcare in an adult ambulatory psychiatric clinic. This process was guided by the Practical, Robust, Implementation and Sustainability Model (PRISM; Feldstein & Glasgow, 2008; McCreight et al., 2019). The PRISM is a well-established implementation framework that integrates the RE-AIM evaluation framework (Glasgow et al., 1999) within contextual domains, allowing for dynamic examination of the implementation and sustainability of an EBP. The PRISM works to address context by focusing on multilevel perspectives (i.e., organizational, clinician) and partner characteristics (i.e., patients, leadership), the external environment (i.e., policies, resources), and sustainability infrastructure (i.e., training, structural supports), while considering how to fit and cocreate interventions and implementation strategies that can best serve the context. Simultaneously, RE-AIM is integrated into PRISM as an evaluation framework to better understand the intervention, strategies, and context by focusing on reach, effectiveness, adoption, implementation, and maintenance (also referred to as sustainment). The PRISM informed the implementation of MBC in an adult ambulatory psychiatry clinic and provided a guiding framework for understanding the context while concretizing the need for considering sustainability strategies. The coming sections discuss the domains of PRISM most applicable to our implementation efforts, how digital training was determined to be a critical implementation strategy, with our evaluation of this strategy guided by RE-AIM.

Organizational Perspective. The MBC was introduced in this adult ambulatory psychiatry clinic to enhance evidence-based care. The ambulatory psychiatry clinic is part of a free-standing, multipsychiatric subspecialty community clinic in a private, nonprofit medical system and medical school in the southeast. The adult section includes psychiatrists, nurse–practitioners, medical students and residents, psychologists, and therapists. Benefits of using MBC are well documented, as MBC increases patient awareness of their symptoms, tracks their progress, and improves patient ability to reach their treatment goals (Lewis et al., 2019). Department administration provided operational and research funds for an external vendor measurement feedback system (MFS) and championed its integration into behavioral healthcare in 2019. Owl Health, a cloud-based system allowing for the capture and analysis of clinical data, was employed as the MFS. The measure library includes tools to assess mood, anxiety, substance use, eating disorders, and other psychiatric concerns. All new adult ambulatory patients were invited to complete a bundle of questionnaires at intake and on a monthly basis during their course of treatment, which included the PHQ9, GAD7, and BASE6 (Kablinger et al., 2022 ).

Clinician Perspective. The MBC research demonstrates several barriers and facilitators for clinicians, including time, technology, knowledge, and self-efficacy, among others (i.e., Murphy et al., 2021). To better understand how these barriers are present in adult ambulatory psychiatry, focus groups were conducted with 18 clinicians and 11 staff, examining how professionals perceive facilitators and barriers of MBC implementation, and to further inform the training program development (Ko et al., 2023). The findings underscored previously identified implementation challenges, such as negative views of MBC implementation (e.g., time burden), desire for further training, patient nonadherence, technological barriers, and lack of designated MBC staff. This feedback opened a window for novel training approaches.

Implementation and Sustainability Infrastructure. To present, implementation strategies were selected pragmatically and arose from the necessity to responsively address barriers as they emerged. As MBC integration improved in the organizational context, the need to plan for sustainability became apparent. Since champions were in place to promote EBP and lead training, a digital training program was proposed as a strategy to begin the transition to sustainment, standardize training, and decrease the need for clinician-led training.

Sustainment of MBC

Sustainability of EBPs is a moving target, and data about the optimal implementation strategies for MBC are mixed (Boyd et al., 2018; Dollar et al., 2020; Resnick & Hoff, 2020; van Sonsbeek et al., 2023; Williams et al., 2024). A systematic review of EBP sustainment efforts identified funding and maintenance of skills through training, supervision, or feedback as the most common strategies (Hailemariam et al., 2019). While training is often a necessary, but insufficient, part of the implementation process, it is possible to leverage training as a strategy to support sustainment. Training can be implemented to improve confidence and build self-efficacy, leading to more positive attitudes surrounding MBC, a factor shown to provide drive to implement and sustain MBC (Jensen-Doss et al., 2024). Moreover, MBC attitudes have been isolated as an important preimplementation determinant (Aarons et al., 2011; Mellor-Clark et al., 2016), while improved self-efficacy has been shown to impact how providers use MBC (Edbrooke-Childs et al., 2016; Gondek et al., 2016). When MBC is implemented with fidelity, providers experience more utility of the practice and have more positive attitudes. Providers who have been clinically trained to provide MBC are more likely to have positive attitudes of the practicality of MBC and use the tool more often (Williams et al., 2022). In another implementation study, clinics with a positive leadership climate were shown to complete MBC with greater fidelity, pointing to the importance of training and positive context that can affect implementation (Koerner et al., 2022; Williams et al., 2024).

Current Study

The focus of this study is to determine whether Measurement-based carE Training for Resilient Implementation in a Clinical setting (METRIC), a novel digital training module, is an appropriate implementation strategy to support MBC training efforts to promote the integration of MBC in behavioral health. This study uses PRISM (Feldstein & Glasgow, 2008) to guide the evaluation of the impact of METRIC in adult ambulatory mental health. Specifically, this study aims to evaluate the reach (e.g., provider engagement) and implementation (e.g., feasibility, acceptability) of METRIC as an MBC training strategy. We explore the impact of this strategy on overall attitudes (e.g., patient, provider) of MBC using the PRISM contextual domains. These results serve to inform the applicability of a digital and asynchronous MBC training program as an implementation strategy to support training in the transition to sustainment.

Methods

Pre-METRIC Implementation Activities

The MBC is a complex clinical practice that requires a multilevel approach to achieve successful implementation. In 2019, when Owl Health was launched, the MBC research team organized several concurrent strategies to facilitate the transition and integrate this EBP. To organize these efforts, the Expert Recommendations for Implementation Change (ERIC; Powell et al., 2015) taxonomy of implementation strategies was leveraged to retroactively code these strategies, which were further categorized by which part of the PRISM contextual domains or RE-AIM evaluation categories they best aligned with (Table 1). As implementation was pragmatic, real-world, and guided by barriers, rather than systematic strategy selection, the research team worked collaboratively to provide a historical account of implementation strategies that were used. Starting in 2019, 33 strategies were implemented as perceived need increased. These strategies are organized by initial implementation efforts focusing on increasing MBC completion rates (e.g., providing clinical supervision, mandating change), enhanced implementation efforts focused on increasing fidelity to the practice (e.g., needs assessment, altering incentives), preparation for digital training (e.g., focus groups, funding), and general sustainment efforts (e.g., compliance feedback, facilitation; Table 1). A standardized MBC protocol was developed to support measurement of depression, anxiety, psychological adjustment, and substance use at intake. Clinicians may choose to utilize additional or alternate measures in ongoing care, driven by patient concerns and clinical judgment. Despite these strategies, rates of clinicians providing MBC were variable (∼65%–75%), pointing to the need to tailor these strategies or consider new ones.

Table 1.

Implementation was Informed by the PRISM Framework, Which Guided Selection of Implementation Strategies Over Time.

Implementation Strategy	Application	PRISM Component	Implementation Strategy Categorization
Initial Implementation (Completion-Focused)
Provide clinical supervision*	Two clinical champions available to answer questions and provide feedback to clinicians	Sustainability Infrastructure	Provide Interactive Assistance
Tailor strategies	Refresher video; provided staff and clinicians access to an MFS demo account tailored to staff vs. clinician role	Implementation	Adapt and Tailor to Context
Develop academic partnerships*	Collaboration with research universities, academic medical centers	Maintenance	Develop stakeholder interrelationships
Identify and prepare champions	2 clinical champions present from implementation of MBC were part of the group who created and disseminated the virtual training	Organizational characteristics (leaders and managers)
Identify early adopters	At outset, relied on 2–3 clinicians to implement PROMs into ambulatory practice for new patient intakes	Organizational characteristics (leaders and managers)
Conduct educational meetings*	Didactics and hands-on use of MFS for trainees	Organizational perspective	Train and educate stakeholders
Conduct educational outreach visits*	Faculty meetings outreach snippets	Organizational perspective	Train and educate stakeholders
Mandate change*	MBC became a standard evidence-based practice in 2018	External Environment	Change Infrastructure
Enhanced Implementation (Process-Focused)
Conduct local needs assessment	Assessed via overall clinic performance on PROM completion	Maintenance	Use and evaluate iterative strategies
Purposely reexamine the implementation*	Shifting from train the trainer to standardized virtual training	Intervention
Stage implementation scale up*	Team testing and editing virtual training	Intervention
Assess for readiness and identify barriers and facilitators*	Focus groups	Recipients: Organizational characteristics of leaders, managers, and staff
Build a coalition	HR education, departmental quality analyst	Implementation, Sustainability Infrastructure	Develop stakeholder interrelationships
Capture and share local knowledge	Survey staff in a focus group	Organizational characteristics (staff)
Involve executive boards	Support from departmental chair and American Psychiatric Association advocacy	External Environment
Promote network weaving	Presented work at the School of Medicine TEACH (Teaching Excellence Academy for Collaborative Healthcare) annual program	Implementation, Sustainability Infrastructure
Distribute educational materials*	PROMs educational pamphlets, demo site to practice and review measures	Organizational perspective	Train and educate stakeholders
Use train-the-trainer strategies*	Ongoing training and consultation for the leader-level trainers who train new providers	Organizational perspective
Support clinicians*	Regular check in	Maintenance
Remind clinicians*	Faculty meetings, divisions meetings, email communications	Maintenance
Involve patients/consumers and family members*	Patient satisfaction survey / Community Engagement Studios via iTHRIV	Patient characteristics	Engage consumer
Use mass media	Weekly departmental newsletter sent to multidisciplinary providers, trainees and educational/clinical/research staff	Reach	Engage consumer
Alter incentive/allowance structures*	MBC completion targets set, contributed to annual evaluation	Maintenance	Utilize financial strategies
Change service sites*	Moved facility October 2023; addition of multiple service lines (geriatrics, grief and healing)	External Environment	Change Infrastructure
Preparation for Digital Training
Use advisory boards and workgroups	Interdisciplinary work team including faculty providers, graduate students, statisticians, innovations representatives, health analytics, HR education team	Implementation, Sustainability Infrastructure	Develop stakeholder interrelationships
Conduct ongoing training*	Ongoing training for new clinicians; made the training accessible online	Organizational perspective	Train and educate stakeholders
Make training dynamic*	Training included role-plays, quizzes, scenarios, etc.	Organizational perspective	Train and educate stakeholders
Fund and contract for the clinical innovation	Digital MBC system utilized for data collection and funded by department	Sustainability infrastructure	Utilize financial strategies
General Sustainment Efforts
Develop and organize quality monitoring systems*	Compliance feedback	Maintenance	Use and evaluate iterative strategies
Obtain and use patients’/consumers’ and family feedback*	Community Engagement Studios through iTHRIV	Patient characteristics, Maintenance	Use and evaluate iterative strategies
Facilitation*	Periodic feedback to clinicians on completion rates; department set yearly goals for completion rates	Implementation, Sustainability Infrastructure	Provide Interactive Assistance
Use other payment schemes*	Added a billing code in EHR (96127)	Sustainability Infrastructure	Utilize financial strategies

Note. *Ongoing implementation strategy in sustainment phase.

This table represents strategies used in the initial implementation period focused on improving MBC completion rates. Then strategies used to enhance implementation focused on adherence to MBC processes and theoretical alignment are listed. Strategies that served as precursors to digital training are listed. These are followed by strategies employed to facilitate a transition to sustainment of MBC.

Our Training Program

Our clinic has employed several implementation strategies to support the sustainment of MBC, including supervision, academic partnerships, change mandates, clinician reminders, altered incentive structures, quality monitoring systems, partner feedback, among others (see starred strategies in Table 1). Even with this bundle of strategies, MBC in adult ambulatory psychiatry relies heavily on champions and clinic leaders to ensure systematic training and successful knowledge transfer. Our team developed METRIC to improve implementation and reach while preparing for MBC sustainability. As an implementation strategy, METRIC aimed to help providers (1) develop a fundamental understanding of how MBC can enhance clinical practice, and (2) acquire a framework to provide PROMs feedback to patients. METRIC was developed to support the routine introduction of MBC by creating a digital tool that could facilely disseminate standardized knowledge of MBC. The creation of METRIC homogenized training on the importance and benefits of MBC usage for a multidisciplined clinical team.

METRIC was developed stepwise over the course of 1.5 years. Feedback on MBC was inconsistent, so we first assessed for readiness and identified barriers and facilitators via focus groups that included clinicians and staff (Ko et al., 2023). After years of collaborating on MBC-based projects, the research team ultimately decided to prioritize a digital training program for empirically based and practical reasons. Literature suggests that digital training practices should be implemented to help sustain the delivery of MBC (Koerner et al., 2022). METRIC was developed to train all providers in ambulatory psychiatry, as well as future trainees, residents, and faculty practicing MBC. Offering real-time didactic training for an entire clinic was not feasible related to scheduling challenges and clinic staffing concerns. Furthermore, creating digital training that is accessible on demand facilitates the onboarding of future providers, necessitating asynchronous learning.

Internal educational teams were contacted to embed METRIC into our institutional learning hub, where it could be individually assigned to anyone, with completion monitored. The first author developed a script for digital training modules that were iteratively adapted and edited by the research team. Content was built onto the digital platform, and team members filmed scenarios and role plays. METRIC contained interactive modules to maintain provider attention and engagement in the form of multiple-choice questions, true/false queries, and drag- and drop-definition matching (Table 2, Supplemental Table S1, Supplemental Figure S1). This pragmatic training is built into the institutional learning hub and designed to be self-paced and completed between 20- and 30-min to best fit the context and preferences of busy behavioral health providers. Training topics were intentionally selected to introduce key concepts and terminology related to MBC, providing a standardized foundation rather than an in-depth exploration of MBC. Content included explaining MBC to patients, addressing patient apprehensions or concerns about MBC, discussion of alignment between subjective PROMs and objective assessment during appointments, attaining supervision, and responding to PROM alerts such as active thoughts of suicide between sessions. Additionally, mock role plays addressing frequently asked questions provided real-time consideration of individual dilemmas that often arise during psychiatric care.

Table 2.

Outline of Digital MBC Training Modules.

Training Module	Format	Content
Assessment
Preassessment	Questionnaire	Assess clinician MBC experiences so far benefits and challenges of current use
Motivational Interviewing Process	Questionnaire	Establish base understanding of training and participant benefit/relevant interests
MBC Psychoeducation
What is MBC?	Psychoeducation	PROMS, collaborative review, treatment planning
Components & Technology	Matching	Defining MBC, ROM, PROM, MFS definitions
MBC Evidence Base	Psychoeducation	Reviewing effectiveness literature (i.e., symptom reduction, treatment length, age)
Benefits of MBC	Drag & Drop	Benefits to patient and practitioner interest
MBC Mechanisms	Psychoeducation	Highlight self-monitoring, feedback, and shared established goals
Practical Challenges	Psychoeducation	Common experienced and anticipated barriers
Critical Facilitators	Psychoeducation	Facilitators (i.e., brief measures, structured training, transparency, collaboration)
Long-Term Organizational Benefits	Psychoeducation	Possible organization-wide benefits of integrating MBC into routine care
Practical Demonstrations
Introducing technology	Psychoeducation	Step-by-step explanation of digital MBC system and how to use it/select relevant measures
Introducing MBC to your client	Video	Online symptom tracking over time, email reminders, benefits of completion before session
Explaining MBC process	Video	Measures’ use for patient well-being, benefit of longitudinal tracking, monitoring treatment goals
Addressing client ambivalence	Video	Importance and relevance of measure completion, asking client about difficulties with completion, assisting facilitation, offering assistance
Interpreting PROMs	Video	Graphing longitudinal data, highlighting trend changes, checking in with client to align scores with meaning for client experience, reviewing any critical items/alerts/responses
Suicide risk protocol	Video	Review chart, notify supervisor of alerts, reviewing risk assessment, clinician assessing patient risk

Participants

Clinicians. METRIC was IRB-approved (IRB-20-1065) and distributed to 56 psychiatry providers (MD/DO, NP, PhD, LCSW), where 47 clinicians completed the digital program. As part of an initiative to implement the practice of MBC in outpatient psychiatry throughout the hospital system, all faculty, staff, and trainees providing patient care were asked to participate in the training. Clinicians completed attitude surveys before they started the training (n = 44), 1-month posttraining (n = 22), and 3-month post-training (n = 21). The first group of clinicians enrolled in METRIC at the start of 2023, and the last group was enrolled in the fall of 2023.

Patients. All patients of the clinicians included in the study were invited to complete the patient satisfaction survey (PaSS). A waiver of informed consent was received as the study was classified as minimal risk research focused on quality improvement. Two independent cohorts of patients were evaluated for the PaSS; between-group differences were examined, rather than within individuals, to better evaluate change over time. This method controls for the inability to perform accurate repeated-measures analyses, as patient populations change significantly between administration periods due to drop-out, discharge, or breaks in treatment. Patients completed the PaSS at pretraining between October and December 2022 (n = 98) and posttraining between July and September 2023 (n = 80).

Measures

Module Data. Throughout the digital program, providers were asked comprehension questions to ensure learning was effective. A total of 16 questions were embedded in the training module, and the formats included open-ended (response required and reflection prompt), multiple-choice, matching, and true/false. The domains assessed in the module were: (1) what has gone well/challenges in MBC practice thus far, (2) interpreting PROM scores and using data to engage with the patient, (3) definitions of MBC, PROM, MFS, and routine outcome monitoring (ROM), and (4) MBC effectiveness based on published literature. In the initial METRIC implementation, technology barriers prevented real-time data collection, but these problems in extracting data were fixed in the second administration of METRIC. Data was recovered post hoc for a subset of participants (N = 19), but did not allow for any adaptations prior to the second implementation as originally intended.

Pragmatic Implementation Measures (Weiner et al., 2017) included the Acceptability of the Intervention (AIM), Intervention Appropriateness Measure (IAM), and Feasibility of Intervention Measure (FIM). These are each 4-item questionnaires designed to assess implementation success by targeting an intervention's acceptability, appropriateness, and feasibility. Items were modified to include METRIC as the target of interest. All scales demonstrated good reliability across the IAM (α = .96), FIM (α = .97), and AIM (α = .93).

Monitoring and Feedback Attitudes Scale (MFA): Jensen-Doss et al. (2018) are a 14-item assessment of clinician belief regarding the potential benefits (α = .88–.95) and risks (α = .86–.90) of MBC. Clinicians are asked to indicate on a 1 (Strongly Disagree) to 5 (Strongly Agree) scale how much they agree with definitions of routine progress monitoring and routine feedback without explicitly naming these MBC aspects, including 10 possible benefits and 4 possible harms. The MFA has demonstrated adequate reliability and validity, and consistent prediction of MBC use as reflected in client records (Patel et al., 2022).

Attitudes Towards Standardized Assessment Scales: Monitoring and Feedback (ASA-MF): Jensen-Doss et al. (2018) are an 18-item measurement of clinician attitudes toward using specific standardized measures and symptom monitoring to make clinical treatment decisions. The ASA-MF has been found to reflect attitudes toward clinical utility (α=.76–.89), benefits for treatment planning (α=.88–.94), and practicality (α=.79–.81; (Patel et al., 2022). Clinicians are again asked to indicate from 1 (Strongly Disagree) to 5 (Strongly Agree) how much they agree with definitions of routine progress monitoring and specific standardized measures in clinical decision-making.

Clinician Attitude and Satisfaction Survey (ClASS) is a 7-item survey created to assess clinician attitudes of MBC. Providers rank their perceptions of MBC, their engagement, and commitment to MBC on a 5-point Likert scale ranging from 1 (e.g., extremely negative) to 5 (e.g., extremely positive). This scale demonstrated good internal consistency (α=.80–.85).

Patient Satisfaction Survey (PaSS) is a 12-item measure examining patient attitudes about MBC. Clients are asked to indicate on a 1 (e.g., extremely negative) to 5 (e.g., extremely positive) about their experience and satisfaction with MBC utilization throughout their intervention. Total scores were the sum of all items. Two subscales were developed for frequency (i.e., completion, feedback; α=.39–.78) and patient experience (i.e., accurate assessment, reaching goals; α=.87–.92).

Procedures

The MBC was first piloted in the clinic in April 2019 and fully automated in Owl Health in February 2020 (Figure 1). In 2024, measure completion shifted to the electronic health record (Gatto et al., 2025). All clinicians were using MBC, with varying experience based on their tenure in the clinic. Since MBC was integrated into standard clinical care, the IRB categorized this study as quality assurance data; informed consent was waived. All clinicians in the ambulatory mental health clinic utilizing MBC were assigned to METRIC. METRIC was introduced via email and virtual department meetings, and clinician participation was strongly encouraged. Two clinic champions (AK, VO) were identified to highlight the importance of the MBC module and to help facilitate clinician completion. In addition, these champions, who serve in administrative and resident training roles, were accessible to clinicians for questions or consultation regarding MBC practice.

Figure 1.

Timeline of implementation of measurement-based care (MBC) in an adult ambulatory psychiatry Clinic.

In January 2023, clinicians were assigned a pretraining survey, which included the MFA, ASA-MF, and ClASS. Clinicians were given 2 weeks to complete these surveys, and following completion, clinicians were assigned the METRIC training module in February 2023 with 1 month to complete the training. This training was released again from June to July 2023 and June to July 2024 with rolling assignment and individual completion of surveys and METRIC for subsequent new providers. METRIC was created in Articulate Storyline, which is an eLearning software with built-in features to allow reporting of how each learner engaged with the education. The data were sent through the learning management system, where management can pull reports and receive real-time feedback from each learner.

Clinicians had 4 weeks to complete METRIC. One and 3 months after completing METRIC, clinicians were prompted to report attitudes about MBC and were allowed 2 weeks to complete them. Two subsequent email reminders were sent to clinicians to complete each component of the training and surveys. As noted above, two independent cohorts of patients were invited to complete the PaSS: one group in late 2022, prior to training, and a second group in mid-2023 post-METRIC. The pre- and posttraining surveys were distributed electronically via REDCap where responses were recorded electronically.

Data Analysis

PRISM. Results of the study are characterized within the PRISM framework to better understand the broader impact of the training. The primary focus of this evaluation is on the implementation of METRIC as an implementation strategy. Reach. Reach was assessed by examining the proportion of clinicians eligible for METRIC to those who completed the training. Implementation. Training implementation data scales (IAM, FIM, AIM) were scored; and mean, median, and range were calculated. Performance on METRIC was assessed using data from the modules. Responses to qualitative items were extracted and summarized. Since there were few qualitative responses, pragmatic summaries were developed by the first author and reviewed by the study team. Contextual Domains. Changes in contextual domains are evaluated by examining changes in clinician (organizational/staff perspective of MBC) and patient attitudes (patient perspective of MBC) before and after the training. To better understand the impact of METRIC, this study considered changes in attitudes toward MBC, as well as clinicians’ and patients’ perceptions of and experience with MBC. Clinician attitudes were collected with the MFA, ASA-MF, and ClASS. Total and subscale scores were calculated at pretraining, 1-month posttraining, and 3-month posttraining. Repeated measures analyses were conducted for a subset of participants who completed measures at all three timepoints to determine if there was any change in attitudes resulting from implementing METRIC. The Friedman test was selected for clinician attitudes scales as a nonparametric, one-way repeated measures test. Patient attitudes (PaSS) were collected pretraining and posttraining, and a Wilcoxon rank-sum test was conducted to determine changes in attitudes over time. Patient attitudes were analyzed by cohort, rather than individual-level data, to represent clinic trends.

Power Considerations. The repeated measures design with 14 participants limits power to detect small-to-moderate effects. Based on a conventional within-subjects analysis of variance design, our study would have >80% power only to detect relatively large effects (Cohen's f ≥ 0.40). Thus, nonsignificant findings should be interpreted with caution, potentially reflecting insufficient power rather than the true absence of effects.

Results

Module Data. Most items were answered correctly (84.59%), and most clinicians answered at least 75% of the questions correctly (n = 16; Supplemental Table S2). Three participants answered all questions correctly, and two clinicians answered less than 65% correctly. Clinicians encountered challenges interpreting patient scores, defining MBC, and understanding whether MBC could help reduce depressive symptoms.

Practical, Robust, Implementation and Sustainability Model Evaluation

Reach. This study examined the proportionality of clinicians who were able to participate in METRIC to assess reach. A majority (83.9%) of clinicians were able to access METRIC, but there was variable completion of posttraining measures. As such, the data presented only represents a subset of the total sample. Moreover, while METRIC was accessible to most clinicians, feedback indicated enhanced accessibility features for those with visual or auditory impairments.

Implementation. Implementation measure results suggest there was fair acceptability (AIM: M = 3.74; SD = .95), appropriateness (IAM: M = 3.87; SD = .90), and feasibility (FIM: M = 3.91; SD = .91; Table 3) of METRIC. Regarding requests for consultation following METRIC training, 22 providers did not indicate the need for future support. Provider comments regarding information about navigating measure selection (n = 4), visual impairment accessibility (n = 1), encouraging patients to take responsibility (n = 1), and the need for face-to-face training (n = 1) were noted as type of training or support needed in consultation.

Table 3.

Weiner (2017) Implementation Measures Examining the Acceptability, Appropriateness, and Feasibility of the MBC Training Program.

Implementation Measure	Posttraining (n = 34)
Implementation Measure	α	M (SD)	Median	Range
Acceptability of Intervention (AIM)	0.93	3.74 (0.95)	3.88	1.25–5.00
Intervention Appropriateness (IAM)	0.96	3.87 (0.90)	4.00	1.75–5.00
Feasibility of Intervention (FIM)	0.97	3.90 (0.91)	4.00	1.50–5.00

Note. AIM, IAM, FIM total scores can range from 1 to 5.

Clinicians reported whether METRIC seems implementable, possible, and doable; responses ranged from “completely disagree” to “completely agree.” Participants partially to completely agreed that METRIC seems implementable (71%), possible (74%), and doable (76%). The clinician follow-up survey also contained two open-ended response options, including the query, “What do you remember as the key takeaways from the training?” Analysis of the qualitative responses for general themes resulted in trends related to the importance, value, necessity, and efficacy of MBC. Comments were also made regarding accessibility limitations from the digital format for providers with visual impairments.

Contextual Domains: Clinician Attitudes: Clinicians completed the attitudes surveys pretraining (n = 44), 1- (n = 22), and 3-month (n = 21) posttraining (Table 4). Only a subset of this data was analyzed in a repeated measures test for clinicians who had data at all timepoints (n = 14; Supplemental Table S3). There was no statistical difference in MFA Benefit (χ²(2) = 2.33, p = .31) or Harm (χ²(2) = 2.09, p = .35) subscales, though scores increased on the MFA benefit subscale and decreased on the harm subscale over time. There is no evidence to suggest that total ASA Clinical Utility (χ²(2) = 2.38, p = .31), Treatment Planning (χ²(2) = 1.85, p = .40), or Practicality (χ²(2) = 0.04, p = .98) subscales differ among the three time periods. There was no statistically significant difference in total ClASS scores (χ²(2) = 4.42, p = .11), despite some improvement over time. There were no statistically significant differences over time across measures. Effect size estimates suggested small to moderate changes despite insignificance (d = −.37 to .39). For MFA Benefit, the change from baseline to 1-month was small (d = 0.23) and from baseline to 3-month was very small (d = 0.16). The MFA Harm showed a small-to-moderate increase at 1-month (d = 0.39) followed by a small decrease at 3-month (d = −0.19). The ASA-MF Clinical Utility showed negligible change (d = −0.09 at 1 month; d = 0.06 at 3 months). The ASA-MF Treatment Planning and Practicality likewise showed minimal change across time (|d| < 0.20). For ClASS, effect sizes were small (1-month d = 0.19; 3-month d = 0.10).

Table 4.

Clinician Attitudes Across Three Measures: MFA, ASA-MF, ClASS.

Clinician Attitude Scale	Subscale	α	Mean (SD)	α	Mean (SD)	α	Mean (SD)	Repeated Measures Analysis
		Baseline		1 month		3 months
		N = 44		N = 22		N = 21
Monitoring and Feedback Attitudes Scale (MFA)	Benefit	0.95	4.69 (0.52)	0.93	4.72 (0.46)	0.88	4.70 (0.42)	χ²(2) = 2.33, p = .31
Monitoring and Feedback Attitudes Scale (MFA)	Harm	0.90	2.52 (1.08)	0.90	2.63 (1.21)	0.86	2.02 (1.03)	χ²(2) = 2.09, p = .35
Attitudes Towards Standardized Assessment Scales: Monitoring and Feedback (ASA-MF)	Clinical Utility	0.76	3.41 (0.70)	0.85	3.45 (0.85)	0.89	3.58 (0.87)	χ²(2) = 2.38, p = .31
	Treatment Planning	0.88	4.23 (0.72)	0.94	4.32 (0.78)	0.92	4.09 (0.80)	χ²(2) = 1.85, p = .40
	Practicality	0.81	3.42 (0.94)	0.84	3.35 (0.94)	0.79	3.33 (0.88)	χ²(2) = .04, p = .98
Clinician Attitudes and Satisfaction Survey (ClASS)	Total	0.80	29.43 (7.11)	0.85	30.27 (8.60)	0.82	31.38 (5.47)	χ²(2) = 4.42, p = .11

Note. All subscale scores can range from 1 to 5. ClASS is a total sum score that ranges from 7 to 35.

Patient Attitudes. Attitudes did not significantly differ on the PaSS between the pre- and posttraining periods for the frequency subscale (p = .72) or patient experience subscale (p = .15).

Discussion

This study examined the development and implementation of a digital Measurement-based carE Training for Resilient Implementation in a Clinical setting (METRIC) in an ambulatory behavioral health clinic guided by the PRISM framework. The PRISM is designed to enhance the translation of research into practice by considering a comprehensive range of factors that affect implementation and sustainability. Application of PRISM ensures that MBC implementation was practical, feasible, and sustainable. Our general purpose was to provide accessible training and support to clinicians to enhance their knowledge, skills, and confidence in using MBC. Specifically, the application of PRISM to the evaluation of METRIC was focused on reach and implementation (RE-AIM), and the impact of the training on contextual domains (PRISM).

Implementation Outcomes. METRIC diverged from traditional in-person didactics on MBC, requiring critical examination of the program implementation. To evaluate the implementation of METRIC, we used Weiner's (2017) measures of feasibility, appropriateness, and acceptability. Overall, clinicians found METRIC to be highly acceptable, feasible, and appropriate, with over 70% of clinicians agreeing that METRIC seems implementable, possible, and doable—demonstrating strong confidence in its implementation strategy. Considering this support, METRIC and other standardized trainings are a cost-effective and accessible strategy to provide foundational training on MBC in large hospital systems.

As outlined in the training program description section, METRIC utilized multiple interactive learning formats to assess clinicians’ knowledge during the training process (Supplemental Table 1). While many clinicians were practicing MBC prior to completing METRIC, the overall correct response rate was 85%, and 3 out of 19 respondents had perfect performance on METRIC. These findings suggest that (1) there was a measurable level of knowledge demonstrated during METRIC and (2) there was still room for improvement and learning on top of the implementation strategies we had already employed. While many implementation strategies have supported the integration of MBC, the variability of correct responses highlights the need to continue and explore diverse strategies for provider education, supporting the need for METRIC and other brief and standardized trainings.

Clinician and Patient Perspective of MBC. To understand the impact of METRIC, this study considered how training may have influenced the MBC context. Specifically, this evaluation focused on whether clinician and patient attitudes toward the practice improved after implementing METRIC. Clinician attitudes, measured by the MFA, ASA-MF, and ClASS, held steady over time, with small to moderate improvements. The small sample size of clinicians who completed all three surveys may have influenced the power to detect results. Further, high baseline measures may explain the lack of large changes despite high engagement with the METRIC training. Clinicians emphasized the value, necessity, and efficacy of MBC, indicating that the training reinforced their beliefs about its importance. These results demonstrate that METRIC was successful in affirming existing positive views and enhancing clinicians’ confidence in applying MBC.

Similarly, patients’ experience and satisfaction with MBC, as measured by the PaSS, demonstrated that patient attitudes did not improve or worsen over time. A systematic review examining the magnitude of change in outcomes following evidence-based practice training identified knowledge, attitudes, behaviors, and confidence as common outcome variables; synthesized results indicated a small change in attitudes toward evidence-based practice, which aligned with our study findings (Wong et al., 2013). The generally positive baseline perceptions of MBC among both clinicians and patients in our study may have contributed to the stability in attitudes (Table 4).

Sustaining MBC

The PRISM has been used as an overarching framework throughout our implementation process, guiding our understanding of general barriers and facilitators of training efforts (Gatto et al., 2025; Ko et al., 2023 ), determining future implementation strategies, and aiding in the design of METRIC. As a complex clinical process, MBC requires a constellation of strategies to successfully implement on a large scale (Dollar et al., 2020; Jensen-Doss et al., 2018; Lewis et al., 2019; Scott & Lewis, 2015). The MBC requires a combination of structural and contextual support (e.g., champions, incentives, MFS) alongside education (e.g., formal/informal training, supervision). While we have employed myriad implementation strategies to support the broader use of MBC (Table 1), less costly strategies are indicated. For example, relying on clinic champions to volunteer time for training and advocacy may impact education standardization and champion burnout. This is particularly relevant in training clinics with high turnover and new hires that may never interact with MBC champions or receive standardized supervision on MBC. Asynchronous training strategies can mitigate inconsistencies in training due to time or personnel constraints.

Digital trainings like METRIC provide opportunities to standardize knowledge, building a foundation for improving fidelity to MBC processes. However, once MBC has been implemented and integrated into a clinical setting, digital training has great potential to aid in the transition to sustainment (Nevedal et al., 2024). They can serve to provide easily overlooked foundations of a clinical practice that may not be translated to new staff (Matlhaba, 2024; Mosquera & Soares, 2025).

Recommendations for MBC Training

When implementing appropriate training practices, it is necessary to tailor to the unique context (McCreight et al., 2019). METRIC was developed in response to direct feedback and buy-in from clinicians, which may account for the positive reception of this strategy. Further, it is important to launch this type of programming at the right stage of implementation to ensure optimal success (e.g., Chamberlain et al., 2011). Although early implementation of standardized MBC training may optimize consistency and shared proficiency, the ideal timing may vary by clinic based on available resources, baseline MBC knowledge, and existing EBP infrastructure. Tailoring training to these contextual factors can enhance fielding efforts and support higher-quality, sustainable MBC delivery.

The following are recommended for sites developing standardized digital training: (1) collaboration between clinical and institutional education teams, (2) utilizing multimodal learning formats, (3) embedding training in a clinician-accessible, asynchronous learning hub, and (4) engaging in training-related assessment with validated questionnaires.

Strengths, Limitations, and Future Directions

Given the challenges of implementing MBC in mental health clinics, our results serve as a stepping stone to improved MBC sustainability in real-world settings. The digital format allows for standardization of education around MBC and offers an alternative to provider reliance on system champions and academic partnerships to sustain MBC. To our knowledge, this study is the first to evaluate MBC digital training in a mental health clinic as an implementation strategy for sustainment. The implementation of the MBC learning module was bolstered by its digital format and embedding in the institutional hub for other required training. Additional strengths of the study were that it assessed clinicians with a variety of mental health degrees and its use of validated scales to measure feasibility, acceptability, appropriateness, and satisfaction.

This study has several limitations. Due to the pragmatic implementation of this digital training, we did not formally assess changes in competency pre- and post-METRIC implementation, as it was not embedded in the training. This is necessary in future evaluations to determine the effectiveness of training in increasing competency. Moreover, technical barriers did not allow for real-time assessment of clinician attitudes at the time the learning module was completed; however, this was improved in the second administration of METRIC. Not every clinician completed every set of surveys, which made comparison of within-person change challenging. In the future, additional strategies can be applied to incentivize provider completion rates. Since MBC was rolled out in the clinic several years before METRIC, clinicians may have developed habits and attitudes that were deeply ingrained and more difficult to alter. It is recommended that other settings develop digital training earlier on to mitigate such clinical challenges. Early development and implementation of digital training would also allow for investigation regarding the optimal timing of introducing training like METRIC and whether repeat exposure would be beneficial.

Conclusions

Based on these findings, we postulate that by systematically applying the PRISM framework, healthcare organizations can enhance the implementation and sustainability of MBC, ultimately leading to improved patient care. Moreover, we found METRIC to be an acceptable implementation strategy to facilitate MBC in behavioral health settings. Additional components that deserve further study include ongoing follow-up sessions with clinicians as a routine component of training. Further, advanced training modules could be developed and implemented to enhance competencies in these settings. Qualitative feedback from clinician participants indicated an interest in posttraining follow-up support. Previous research has found significant results for the use of online training and posttraining consultation in MBC, though the authors suggest further research on optimal consultation dosages (Lyon et al., 2022). Delivery of consistent MBC completion feedback to clinicians, additional training for those who indicate personal need, improved module delivery for clinicians with visual or hearing impairment, and potentially, accountable care in the form of compensation and/or bonus potential are additional areas for further study.

Supplemental Material

sj-docx-1-irp-10.1177_26334895261456673 - Supplemental material for Assessing the Implementation of a Novel Digital Measurement-Based Care Training in Behavioral Health Using the PRISM Framework

Supplemental material, sj-docx-1-irp-10.1177_26334895261456673 for Assessing the Implementation of a Novel Digital Measurement-Based Care Training in Behavioral Health Using the PRISM Framework by Alyssa J. Gatto, Hayoung Ko, Virginia C. O'Brien, Robert S. Mcnamara, Sydney B. Jones, Hunter D. Sharp, Ashlie R. Phenes, Lee D. Cooper and Anita S. Kablinger in Implementation Research and Practice

Footnotes

Acknowledgments

Dr. Robert L. Trestman has been an advocate for our MBC implementation since 2018 and an integral champion of this work. Owen Fahy was responsible for developing our digital training program in Cornerstone. Martha Tenzer, a data science engineer, has been an integral member of our MBC implementation team and has assisted in several MBC initiatives and projects mentioned in this article.

ORCID iDs

Alyssa J. Gatto

Hayoung Ko

Virginia C. O'Brien

Robert S. Mcnamara

Sydney B. Jones

Hunter D. Sharp

Lee D. Cooper

Anita S. Kablinger

Ethics Approval and Consent to Participate

The Carilion Clinic Institutional Review Board (IRB) fully approved the above referenced study via expedited review under Category 5: Research involving materials (data, documents, records, or specimens) that have been collected, or will be collected solely for nonresearch purposes (such as medical treatment or diagnosis) and Category 7: Research on individual or group characteristics or behavior (including, but not limited to, research on perception, cognition, motivation, identity, language, communication, cultural beliefs or practices, and social behavior) or research employing survey, interview, oral history, focus group, program evaluation, human factors evaluation, or quality assurance methodologies.

Authors’ Contributions

Alyssa J. Gatto was responsible for the organization and structure of the manuscript. Alyssa J. Gatto and Hayoung Ko conceptualized and wrote the background section. Sydney B. Jones, Anita S. Kablinger, Virginia C. O'Brien, Hunter D. Sharp, and Ashlie R. Phenes conceptualized and wrote the method section. Hunter D. Sharp and Alyssa J. Gatto ran the analyses and contributed to the method and results section. Lee D. Cooper, Virginia C. O'Brien, Robert S. Mcnamara, Anita S. Kablinger, and Alyssa J. Gatto conceptualized and wrote the discussion section. All authors read and approved the final manuscript.

Funding

The authors disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: Funding for this project was supported internally through the Department of Psychiatry and Behavioral Medicine.

Declaration of Conflicting Interests

The authors declared the following potential conflicts of interest with respect to the research, authorship, and/or publication of this article: Alyssa J. Gatto and Lee D. Cooper are both selected members of the American Psychological Association Working Group on the Practice Guidelines for Measurement-Based Care (MBC).

Data Availability and Materials

The datasets used and/or analyzed during the current study are available from the corresponding author on reasonable request.

Supplemental Material

Supplemental material for this article is available online.

References

Aarons

G. A.

Hurlburt

Horwitz

S. M.

(2011). Advancing a conceptual model of evidence-based practice implementation in public service sectors. Administration and Policy in Mental Health and Mental Health Services Research, 38(1), 4–23. https://doi.org/10.1007/s10488-010-0327-7

Aboraya

Nasrallah

H. A.

Elswick

D. E.

Ahmed

Estephan

Aboraya

Berzingi

Chumbers

Berzingi

Justice

(2018). Measurement-based care in psychiatry—past, present, and future. Innovations in Clinical Neuroscience, 15(11–12), 13. PMID: 30834167. PMCID: PMC6380611.

Aboraya

A. S.

(2022). Measurement-Based Care (MBC) training curriculum for mental health professionals and psychiatry residency programs. In Manual for the standard for clinicians’ interview in psychiatry (SCIP) a new assessment tool for measurement-based care (MBC) and personalized medicine in psychiatry (PMP) (pp. 193–204). Springer. https://doi.org/10.1007/978-3-030-94930-3_9

Amini

Oemrawsingh

Verweij

L. M.

Lingsma

H. F.

Hazelzet

J. A.

Eijkenaar

van Leeuwen

(2021). Facilitators and barriers for implementing patient-reported outcome measures in clinical care: An academic center’s initial experience. Health Policy (Amsterdam, Netherlands), 125(9), 1247–1255. https://doi.org/10.1016/j.healthpol.2021.07.001

Arbuckle

M. R.

Weinberg

Harding

K. J. K.

Isaacs

A. J.

Covell

N. H.

Cabaniss

D. L.

Essock

S. M.

Sederer

L. I.

(2013). Best practices: The feasibility of standardized patient assessments as a best practice in an academic training program. Psychiatric Services, 64(3), 209–211. https://doi.org/10.1016/j.healthpol.2021.07.001

Barber

Resnick

S. G.

(2023). Collect, share, act: A transtheoretical clinical model for doing measurement-based care in mental health treatment. Psychological Services, 20(Suppl 2), 150–157. https://doi.org/10.1037/ser0000629. PMID: 35201811.

Boswell

J. F.

Hepner

K. A.

Lysell

Rothrock

N. E.

Bott

Childs

A. W.

Douglas

Owings-Fonner

Wright

C. V.

Stephens

K. A.

Bard

D. E.

Aajmain

Bobbitt

B. L.

(2023). The need for a measurement-based care professional practice guideline. Psychotherapy, 60(1), 1–16. https://doi.org/10.1037/pst0000439. PMID: 23885809.

Boyd

M. R.

Powell

B. J.

Endicott

Lewis

C. C.

(2018). A method for tracking implementation strategies: An exemplar implementing measurement-based care in community behavioral health clinics. Behavior Therapy, 49(4), 525–537. https://doi.org/10.1016/j.beth.2017.11.012

Camargo

C. P.

Tempski

P. Z.

Busnardo

F. F.

De Arruda Martins

Gemperli

(2020). Online learning and COVID-19: A meta-synthesis analysis. Clinics, 2(75), e2286. https://doi.org/10.6061/clinics/2020/e2286

10.

Chamberlain

Brown

C. H.

Saldana

(2011). Observational measure of implementation progress in community based settings: The stages of implementation completion (SIC). Implementation Science, 6(1), 116. https://doi.org/10.1186/1748-5908-6-116

11.

Cheung

B. S.

Murphy

J. K.

Michalak

E. E.

Liu

Yang

Wang

Chen

Lam

R. W.

(2023). Barriers and facilitators to technology-enhanced measurement based care for depression among Canadian clinicians and patients: Results of an online survey. Journal of Affective Disorders, 23(320), 1–6. https://doi.org/10.1016/j.jad.2022.09.055

12.

Chick

R. C.

Clifton

G. T.

Peace

K. M.

Propper

B. W.

Hale

D. F.

Alseidi

A. A.

Vreeland

T. J.

(2020). Using technology to maintain the education of residents during the COVID-19 pandemic. Journal of Surgical Education, 77(4), 729–732. https://doi.org/10.1016/j.jsurg.2020.03.018

13.

Dollar

K. M.

Kirchner

J. E.

DePhilippis

Ritchie

M. J.

McGee-Vincent

Burden

J. L.

Resnick

S. G.

(2020). Steps for implementing measurement-based care: Implementation planning guide development and use in quality improvement. Psychological Services, 17(3), 247–261. https://doi.org/10.1037/ser0000368

14.

Edbrooke-Childs

Wolpert

Deighton

(2016). Using patient reported outcome measures to improve service effectiveness (UPROMISE): Training clinicians to use outcome measures in child mental health. Administration and Policy in Mental Health and Mental Health Services Research, 43(3), 302–308. https://doi.org/10.1007/s10488-014-0600-2

15.

Feldstein

A. C.

Glasgow

R. E.

(2008). A practical, robust implementation and sustainability model (PRISM) for integrating research findings into practice. The Joint Commission Journal on Quality and Patient Safety, 34(4), 228–243. https://doi.org/10.1016/S1553-7250(08)34030-6. PMID: 18468362.

16.

Gatto

A. J.

McNamara

R. S.

O’Brien

V. C.

Jones

S. B.

Cooper

L. D.

Kablinger

A. S.

(2025). An EPIC transition: Rapid conversion of a measurement feedback system in behavioral health. Digital Health, 11, 1-6 https://doi.org/10.1177/20552076251359133

17.

Glasgow

R. E.

Vogt

T. M.

Boles

S. M.

(1999). Evaluating the public health impact of health promotion interventions: The RE-AIM framework. American Journal of Public Health, 89(9), 1322–1327. https://doi.org/10.2105/ajph.89.9.1322

18.

Gondek

Edbrooke-Childs

Fink

Deighton

Wolpert

(2016). Feedback from outcome measures and treatment effectiveness, treatment efficiency, and collaborative practice: A systematic review. Administration and Policy in Mental Health, 43(3), 325–343. https://doi.org/10.1007/s10488-015-0710-5

19.

Hailemariam

Bustos

Montgomery

Barajas

Evans

L. B.

Drahota

(2019). Evidence-based intervention sustainability strategies: A systematic review. Implementation Science, 14(57), 1–12. https://doi.org/10.1186/s13012-019-0910-6

20.

Hatfield

D. R.

Ogles

B. M.

(2007). Why some clinicians use outcome measures and others do not. Administration and Policy in Mental Health and Mental Health Services Research, 34(3), 283–291. https://doi.org/10.1007/s10488-006-0110-y

21.

Jensen-Doss

Casline

Woodard

G. S.

Phillips

D. A.

Lane

Palafu

Waye

Ramirez

Cheron

D. M.

Okamura

(2024). Provider use of, attitudes towards, and self-efficacy with key measurement-based care practices in youth mental health treatment: A multi-site examination. Administration and Policy in Mental Health and Mental Health Services Research, 52(1), 146–158. https://doi.org/10.1007/s10488-024-01354-6

22.

Jensen-Doss

Haimes

E. M. B.

Smith

A. M.

Lyon

A. R.

Lewis

C. C.

Stanick

C. F.

Hawley

K. M.

(2018). Monitoring treatment progress and providing feedback is viewed favorably but rarely used in practice. Administration and Policy in Mental Health and Mental Health Services Research, 45(1), 48–61. https://doi.org/10.1007/s10488-016-0763-0. PMID: 27631610. PMCID: PMC5495625.

23.

Kablinger

A. S.

Gatto

A. J.

O’Brien

V. C.

Jones

McNamara

R. S.

Sharp

H. D.

Tenzer

M. M.

Cooper

L. D.

(2022). Effects of COVID-19 on patients in adult ambulatory psychiatry: Using patient-rated outcome measures and telemedicine. Telemedicine and E-Health, 28(10), 1421–1430. https://doi.org/10.1089/tmj.2021.0642

24.

Kim Thi Vu

Nguyen

H. T. T.

Chi Phan

(2023). Enhance teaching methods in the digital environment: A case study of students’s self-study competence. Journal of Advanced Education and Sciences, 3(1), 107–114.

25.

Gatto

A. J.

Jones

S. B.

O’Brien

V. C.

McNamara

R. S.

Tenzer

M. M.

Sharp

H. D.

Kablinger

A. S.

Cooper

L. D.

(2023). Improving measurement-based care implementation in adult ambulatory psychiatry: A virtual focus group interview with multidisciplinary healthcare professionals. BMC Health Services Research, 23(1), 408. https://doi.org/10.1186/s12913-023-09202-3. PMID: 37101134. PMCID: PMC10132409.

26.

Koerner

Levy

Dimeff

L. A.

(2022). Using technology to train and sustain delivery of evidence-based practices. Cognitive and Behavioral Practice, 29(1), 41–49. https://doi.org/10.1016/j.cbpra.2020.12.007

27.

Lewis

C. C.

Boyd

Puspitasari

Navarro

Howard

Kassab

Hoffman

Scott

Lyon

Douglas

(2019). Implementing measurement-based care in behavioral health: A review. JAMA Psychiatry, 76(3), 324–335. https://doi.org/10.1001/jamapsychiatry.2018.3329. PMID: 30566197. PMCID: PMC6584602.

28.

Lyon

A. R.

Liu

F. F.

Connors

E. H.

King

K. M.

Coifman

J. I.

Cook

McRee

Ludwig

Law

Dorsey

McCauley

(2022). How low can you go? Examining the effects of brief online training and post-training consultation dose on implementation mechanisms and outcomes for measurement-based care. Implementation Science Communications, 3(1), 79. https://doi.org/10.1186/s43058-022-00325-y

29.

Matlhaba

(2024). Building a foundation for clinical competence. In Matlhaba

(Ed.), Enhancing clinical competence of graduate nurses (pp. 35–51). Springer Nature Switzerland. https://doi.org/10.1007/978-3-031-81407-5_2

30.

McCreight

M. S.

Rabin

B. A.

Glasgow

R. E.

Ayele

R. A.

Leonard

C. A.

Gilmartin

H. M.

Frank

J. W.

Hess

P. L.

Burke

R. E.

Battaglia

C. T.

(2019). Using the Practical, Robust Implementation and Sustainability Model (PRISM) to qualitatively assess multilevel contextual factors to help plan, implement, evaluate, and disseminate health services programs. Translational Behavioral Medicine, 9(6), 1002–1011. https://doi.org/10.1093/tbm/ibz085

31.

Mellor-Clark

Cross

Macdonald

Skjulsvik

(2016). Leading horses to water: Lessons from a decade of helping psychological therapy services use routine outcome measurement to improve practice. Administration and Policy in Mental Health and Mental Health Services Research, 43(3), 279–285. https://doi.org/10.1007/s10488-014-0587-8

32.

Mosquera

Soares

M. E.

(2025). Onboarding: A key to employee retention and workplace well-being. Review of Managerial Science, 19(12), 3687–3711. https://doi.org/10.1007/s11846-025-00864-3

33.

Murphy

J. K.

Michalak

E. E.

Liu

Colquhoun

Burton

Yang

Wang

Fei

Wang

Zhang

Huang

Yang

Lin

Fang

Chen

(2021). Barriers and facilitators to implementing measurement-based care for depression in Shanghai, China: A situational analysis. BMC Psychiatry, 21(1), 430. https://doi.org/10.1186/s12888-021-03442-5

34.

Nevedal

A. L.

Widerquist

M. A. O.

Reardon

C. M.

Arasim

Jackson

G. L.

White

Burns

Fix

G. M.

DeLaughter

Cutrona

S. L.

(2024). Understanding pathways from implementation to sustainment: A longitudinal, mixed methods analysis of promising practices implemented in the Veterans Health Administration. Implementation Science, 19(1), 34. https://doi.org/10.1186/s13012-024-01361-z

35.

Nkulu-Ily

Y. S.

(2023). Implementation of e-learning curriculum in higher education. European Journal of Open, Distance and E-Learning, 25(1), 62–73. https://doi.org/10.2478/eurodl-2023-0005

36.

O’Brien

V. C.

Kablinger

A. S.

Jones

S. B.

McNamara

R. S.

Phenes

A. R.

Hankey

M. S.

Gatto

A. J.

Tenzer

M. M.

Sharp

H. D.

Cooper

L. D.

(2024). Use of patient-reported outcome measures to assess the effectiveness of hybrid psychiatric visits. Psychiatric Services, 75(12), 1206–1212. https://doi.org/10.1176/appi.ps.20230355. PMID: 38863328.

37.

Øvretveit

Zubkoff

Nelson

E. C.

Frampton

Knudsen

J. L.

Zimlichman

(2017). Using patient-reported outcome measurement to improve patient care. International Journal for Quality in Health Care, 29(6), 874–879. https://doi.org/10.1093/intqhc/mzx108

38.

Patel

Z. S.

Jensen-Doss

Lewis

C. C.

(2022). MFA and ASA-MF: A psychometric analysis of attitudes towards measurement-based care. Administration and Policy in Mental Health and Mental Health Services Research, 49(1), 13–28. https://doi.org/10.1007/s10488-021-01138-2. PMID: 33942200.

39.

Powell

B. J.

Waltz

T. J.

Chinman

M. J.

Damschroder

L. J.

Smith

J. L.

Matthieu

M. M.

Proctor

E. K.

Kirchner

J. E.

(2015). A refined compilation of implementation strategies: Results from the Expert Recommendations for Implementing Change (ERIC) project. Implementation Science, 10(1), 21.

40.

Resnick

S. G.

Hoff

R. A.

(2020). Observations from the national implementation of measurement based care in mental health in the department of veterans affairs. Psychological Services, 17(3), 238–246. https://doi.org/10.1037/ser0000351

41.

Ridout

K. K.

Vanderlip

E. R.

Ridout

S. J.

Alter

C. L.

(2023). Measurement-Based care implementation challenges and opportunities in real-world practice. Psychiatric Annals, 53(9), 405–409. https://doi.org/10.3928/00485713-20230818-02

42.

Scott

Lewis

C. C.

(2015). Using measurement-based care to enhance any treatment. Cognitive and Behavioral Practice, 22(1), 49–59. https://doi.org/10.1016/j.cbpra.2014.01.010. PMID:27330267; PMCID: PMC4910387.

43.

Tudor Car

Soong

Kyaw

B. M.

Chua

K. L.

Low-Beer

Majeed

(2019). Health professions digital education on clinical practice guidelines: A systematic review by Digital Health Education collaboration. BMC Medicine, 17(1), 139. https://doi.org/10.1186/s12916-019-1370-1

44.

van Sonsbeek

M. A.

Hutschemaekers

G. J.

Veerman

J. W.

Vermulst

Tiemens

B. G.

(2023). The results of clinician-focused implementation strategies on uptake and outcomes of Measurement-Based Care (MBC) in general mental health care. BMC Health Services Research, 23(1), 326. https://doi.org/10.1186/s12913-023-09343-5

45.

Van Tiem

Wirtz

Suiter

Heeren

Fuhrmeister

Fortney

Reisinger

Turvey

(2022). The implementation of measurement-based care in the context of telemedicine: Qualitative study. JMIR Mental Health, 9(11), e41601. https://doi.org/10.2196/41601

46.

Weiner

B. J.

Lewis

C. C.

Stanick

Powell

B. J.

Dorsey

C. N.

Clary

A. S.

Boynton

M. H.

Halko

(2017). Psychometric assessment of three newly developed implementation outcome measures. Implementation Science, 12(1), 108. https://doi.org/10.1186/s13012-017-0635-3

47.

Weiner

B. J.

Lewis

C. C.

Stanick

Powell

B. J.

Dorsey

C. N.

Clary

A. S.

Boynton

M. H.

Halko

(2017). Psychometric assessment of three newly developed implementation outcome measures. Implementation science, 12(1), 108.

48.

Williams

N. J.

Marcus

S. C.

Ehrhart

M. G.

Sklar

Esp

S. M.

Carandang

Vega

Gomes

A. E.

Brookman-Frazee

Aarons

G. A.

(2024). Randomized trial of an organizational implementation strategy to improve measurement-based care fidelity and youth outcomes in community mental health. Journal of the American Academy of Child & Adolescent Psychiatry, 63(10), 991–1004. https://doi.org/10.1016/j.jaac.2023.11.010

49.

Williams

N. J.

Ramirez

N. V.

Esp

Watts

Marcus

S. C.

(2022). Organization-level variation in therapists’ attitudes toward and use of measurement-based care. Administration and Policy in Mental Health and Mental Health Services Research, 49(6), 927–942. https://doi.org/10.1007/s10488-022-01206-1

50.

Wong

S. C.

Mcevoy

M. P.

Wiles

L. K.

Lewis

L. K.

(2013). Magnitude of change in outcomes following entry-level evidence-based practice training: a systematic review. International Journal of Medical Education, 4, 107–114.

51.

Wray

L. O.

Ritchie

M. J.

Oslin

D. W.

Beehler

G. P.

(2018). Enhancing implementation of measurement-based mental health care in primary care: A mixed-methods randomized effectiveness evaluation of implementation facilitation. BMC Health Services Research, 18(1), 753. https://doi.org/10.1186/s12913-018-3493-z

52.

Yule

A. M.

Youn

S. J.

Dean

Woodward

D. W.

Firmin

E. S.

Kramer

Stone

Marques

Wilens

T. E.

(2024). Different factors identified by stakeholder group for barriers and facilitators to measurement-based care implementation in behavioral health clinics. Journal of Clinical Psychology, 80(3), 576–590. https://doi.org/10.1002/jclp.23639. PMID: 38230918. PMCID: PMC10923014.

53.

Zhang

K. K.

Thompson

A. W.

(2023). Effectiveness of electronic learning for continuing interprofessional education on behavior change of healthcare professionals: A scoping review. Journal of Interprofessional Care, 37(5), 807–817. https://doi.org/10.1080/13561820.2022.2071850

54.

Zhu

Hong

R. H.

Yang

Wang

Liu

Murphy

J. K.

Michalak

E. E.

Wang

Yatham

L. N.

Chen

Lam

R. W.

(2021). The efficacy of measurement-based care for depressive disorders: Systematic review and meta-analysis of randomized controlled trials. The Journal of Clinical Psychiatry, 82(5), 21r14034. https://doi.org/10.4088/JCP.21r14034

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

5.74 MB

0.00 MB