Using Planned and Unplanned Adaptation to Implement Universal Alcohol Screening and Brief Intervention to Prevent Alcohol-Exposed Pregnancies in Four Primary Care Health Systems

Abstract

Background:

The United States Preventive Services Task Force recommends annual alcohol screening and brief behavioral intervention (alcohol SBI) with general adult and pregnant populations. Implementation of alcohol SBI in primary care has encountered numerous barriers to adapting procedures and infrastructure to support its routine delivery. This collection of case studies describes the implementation strategies used by 4 academic health system teams that were funded by the Centers for Disease Control and Prevention to implement alcohol SBI within healthcare systems to prevent alcohol-exposed pregnancies.

Methods:

We used constructs from the Framework for Reporting Adaptations and Modifications-Expanded (FRAME) to describe planned and unplanned adaptations to implementation strategies, and the SBIRT (Screening, Brief Intervention, and Referral to Treatment) Program Matrix to identify key questions, challenges, and recommendations for improving alcohol SBI implementation. Participating systems were 2 regional affiliates of a national reproductive healthcare organization, an integrated non-profit healthcare system, and an urban medical center and its affiliated network of community health centers.

Results:

Planned adaptations included expanding the target population for brief interventions to include patients drinking at low levels who could become pregnant, modifying workflows and systems to support routine screening, and customizing training content and logistics. Unplanned adaptations included varying site recruitment and pre-implementation awareness-building strategies to enhance local receptivity of systems with decentralized management, and pivoting from in-person to virtual training during the COVID-19 pandemic. Fewer unplanned adaptations were observed for health systems with centralized management structures and practice teams that were fully engaged in implementation planning, training, roll-out, and problem-solving.

Conclusions:

Unplanned adaptations were observed across the 4 cases and emphasized the importance of flexible, adaptive designs when implementing evidence-based practice in dynamic settings. Participation of the health system in planning, including decisions to modify electronic health records and workflows, supported adapting to unplanned circumstances to achieve implementation goals.

Keywords

SBIRT SBI adaptation implementation science fetal alcohol spectrum disorders alcohol women’s health prenatal alcohol exposure

Highlights

Conducting routine alcohol screening and brief intervention (alcohol SBI) within primary and reproductive healthcare has potential to prevent alcohol exposed pregnancies, but implementing practice change in the “real world” requires full involvement from healthcare settings.

Implementation planning that includes the health system in designing alcohol SBI procedures is an effective implementation strategy, but adapting to unanticipated circumstances is key.

Four cases that used planned and unplanned strategies to tailor implementation of alcohol SBI to the needs of diverse healthcare settings are presented, along with recommended strategies for addressing multi-level implementation challenges.

Introduction

Alcohol is a teratogen that can cause miscarriage, low birth weight, and lifelong behavioral, intellectual, and physical disabilities, known as fetal alcohol spectrum disorders (FASDs).¹ Alcohol consumption during pregnancy is the sole risk factor for FASDs.² Because there is no known safe amount, no safe time, and no safe type of alcohol use during pregnancy, clinicians should advise patients who are pregnant or trying to conceive to abstain from using any alcohol.³

The United States Preventive Services Task Force “recommends screening for unhealthy alcohol use in primary care settings in adults 18 years or older, including pregnant women, and providing persons engaged in risky or hazardous drinking with brief behavioral counseling interventions to reduce unhealthy alcohol use.”^4-6 Standardized alcohol screening and brief intervention (alcohol SBI) has been widely promoted to meet this need.^7,8 Adapting alcohol SBI to reach individuals at risk for an alcohol-exposed pregnancy (AEP) could provide a systematic way to identify patients with low levels of alcohol use (ie, those who screen negative for excessive use on standardized screeners) who are or might be pregnant and provide them with the necessary guidance to make informed health choices.⁹ Despite the effectiveness of alcohol SBI in addressing excessive alcohol use for adult patients, and its potential to be used as an FASD prevention strategy, the adoption of alcohol SBI in primary care practice faces numerous barriers, including competing demands and priorities,¹⁰ challenges adapting systems-level infrastructure and workflows,¹¹ clinician beliefs and self-efficacy related to discussing alcohol use,¹² complexities targeting multiple behaviors for AEP prevention (ie, the need to assess any alcohol use and contraception method),¹³ concerns about stigma,¹⁴ and insufficient reimbursement.¹⁵ Technology-supported electronic SBI (e-SBI) addresses some known barriers but still requires adaptation to be integrated within clinical operations.¹⁶

Adapting alcohol SBI to improve its feasibility within healthcare environments^11,15,17 while preserving its effectiveness and fidelity is expected and necessary.^18,19 Proactive (ie, planned) adaptations made in response to anticipated circumstances include modifying content to enhance cultural acceptability; customizing systems and procedures to support delivery; and tailoring training to meet provider and staff needs.¹⁸ When unanticipated circumstances arise, such as the COVID-19 pandemic, planned strategies may need to be paused or modified. The literature acknowledges that reactive (ie, unplanned, but contextually responsive) adaptations can still support implementation aims if they consider data, theory, best practice, and the potential impact on intervention outcomes.^20,21

This paper describes the observations and reflections of 4 multidisciplinary project teams funded by the Centers for Disease Control and Prevention (CDC) to implement alcohol SBI and AEP prevention protocols within structurally diverse primary care systems and the planned and unplanned strategy adaptations used to promote the adoption of alcohol SBI into clinical practice. We used a logic model designed by the CDC to guide the work of the projects and the SBIRT (Screening, Brief Intervention, and Referral to Treatment) Program Matrix²² to describe multi-level strategies used by teams. Constructs from the Framework for Reporting Adaptations and Modifications-Expanded (FRAME)^18,20 were used to describe whether adaptations were planned or unplanned, what was modified (ie, content, context, training, evaluation, or implementation), why it was modified, and lessons learned.

Project Context and Timeframe

Over a 4-year project period (2018-2022), 4 multidisciplinary project teams led by researchers experienced in alcohol SBI training and implementation, partnered with a multi-clinic healthcare system within their respective regions to implement alcohol SBI and AEP prevention protocols. Key strategies for these projects, with their associated outputs and outcomes, are outlined in Figure 1.

Figure 1.

Adapted from the CDC logic model for implementing alcohol screening and brief intervention (alcohol SBI) in primary healthcare settings.

Methods

This study is a descriptive, multiple case study of 4 alcohol SBI implementation projects, led by the authors of this manuscript. Each author reviewed their project team’s original proposals, action plans, and progress reports to develop their case. Cases were structured to describe the 5 components of the SBIRT Program Matrix: (1) performance site type (services provided), location (urban/rural), receptivity to alcohol SBI and AEP prevention; (2) management structure (centralized/decentralized administration and systems, training methods, quality management); (3) providers and staff involved in implementation; (4) patient demographics (if available); and (5) services planned.²²

Applying constructs from FRAME,¹⁸ each team summarized their implementation strategies, adaptations made, reasons for and goal of the adaptation, whether the adaptation was planned or unplanned, and how the adaptation promoted the implementation of alcohol SBI.²⁰ Teams also identified the SBIRT Program Matrix level impacted by the adaptations (ie, structure, provider, patient, or service level).²²

Over a period of 9 months, the authors independently developed and reviewed drafts of their 4 cases and convened monthly on a series of 1-hour, group videoconferences to discuss similarities and differences in implementation contexts, strategies, adaptations, and whether adaptations were planned (proactive) or unplanned (a reaction to unanticipated circumstances).^23,24 Group discussions, which included representatives from the 4 project teams, focused on describing the implementation contexts, understanding and reflecting on each team’s experiences implementing alcohol SBI,²⁵ summarizing planned and unplanned strategies across projects to implement alcohol SBI, and documenting key challenges and strategies by SBIRT Program Matrix levels associated with alcohol SBI implementation.

Results

Cases

Implementing Alcohol SBI in Two Multistate Reproductive Healthcare Systems

Cases 1 and 2: Planned Parenthood Great Northwest, Hawai’i, Alaska, Indiana, Kentucky (PPGNHAIK) and the University of Alaska Anchorage (UAA); Planned Parenthood of Southern New England (PPSNE) and University of Connecticut School of Medicine at UConn Health (UCH).

The 2 participating health systems, PPGNHAIK and PPSNE, are regional affiliates of the Planned Parenthood Federation of America (PPFA), that support the lifelong reproductive and sexual health of individuals through health services, education, and advocacy. Leadership at both of these systems, when approached by their prospective University partners about the alcohol SBI implementation funding opportunity, were highly receptive to implementing alcohol SBI across all health centers, participated in developing the proposal, and agreed to participate fully in adapting alcohol SBI protocols to align with their services and practice philosophy. Each affiliate has its systems that support clinical operations (eg, NextGen^® electronic health record [EHR] at PPGNHAIK and Athena EHR at PPSNE), training, and quality assurance. Strategic direction for operations is provided by the affiliate executive leads. Health center managers provide on-site supervision to clinical assistants and providers. Regional directors of clinical services provide quality assurance oversight and collaborate with the training and education department to maintain patient care standards. Patient care is provided by mid-level clinicians, advanced practice nurses, nurse-midwives, and certified physician assistants.

Implementation Strategies

Strategies for both projects emphasized full participation from affiliate key personnel, with some differences in approach. PPGNHAIK formed a planning team with representatives from leadership, clinical operations, quality assurance, informatics, and training who met semi-monthly. This planning team was tasked with adapting the processes and infrastructure as necessary to fully integrate alcohol SBI into systems, workflows, and practice to assure its long-term sustainability,²⁶ following the steps recommended in the CDC guide: Planning and Implementing Screening and Brief Intervention for Risky Alcohol Use: A Step-by-Step Guide for Primary Care Practices.¹⁷ Protocol development and systems modifications took about 8 months.

The UCH and PPSNE team formed an academic health system partnership for their implementation initiative, “Alcohol Screening in Planned Parenthood to Increase Resiliency (ASPPIRE).” ASPPIRE partners created a semi-monthly working group to assist in health center-specific protocol development and guide the program roll-out. Health center managers served as alcohol SBI champions, who assisted in adapting, organizing, and managing the program at their health centers. ASPPIRE partners prioritized systems-level implementation strategies, which included modifying the Athena EHR system. In addition to deciding which patients would be offered alcohol SBI (adults) and when (annual wellness visits), both PPGNHAIK and ASPPIRE selected the United States Alcohol Use Disorders Identification Test (USAUDIT) as the screening tool,²⁷ which was integrated into the patient visit. Both project teams also worked with the health system’s technology support to add documentation fields (eg, checkboxes, note fields, “order-sets,” referral links) to the EHRs for billing and quality monitoring.

Both PPGNHAIK and PPSNE provided staggered on-site training that was required for all providers and staff. Delivery of training for the 2 projects differed somewhat, with PPGNHAIK using a combination of online modules that were completed prior to an in-person health center training that included skills practice role-playing and was co-led by a PPGNHAIK clinical leader and a UAA trainer. ASPPIRE’s training and technical assistance (TA) was directed by UCH’s SBIRT Training Academy in collaboration with PPSNE staff. Both PPGNHAIK and PPSNE supplemented group training with professional-oriented strategies (eg, in-service education, preceptor training, chart audit, and feedback) and patient-oriented strategies (eg, printed/online educational materials).

Both project teams made iterative adaptations to patient flow and systems based on EHR data and health system feedback. For the PPGNHAIK team, planning questions and process evaluation measures were informed by RE-AIM (Reach, Effectiveness, Adoption, Implementation, Maintenance), which guided criteria for alcohol SBI eligibility (eg, adults presenting for their annual wellness visit); how and by whom the key components of alcohol SBI would be delivered (eg, how would screening be administered; who would deliver the BI); and methods for assessing the consistency of delivery of alcohol SBI’s key components across sites (eg, documentation fields added to the EHR), including methods for obtaining feedback from providers and staff involved in delivering SBIRT.^28,29 System-level implementation fidelity was monitored using visit documentation data extracted from the EHR, to calculate screening and BI rates at the clinic and health system level. In addition, semi-structured questions guided a scheduled post-training call with health center teams to identify problems and improvement suggestions.^26,30 These data informed additional adaptations to enhance feasibility and service quality. In addition to incorporating alcohol SBI procedures into the EHR, strategies to sustain implementation (maintenance) included migrating training to the PPGNHAIK Learning Management System (LMS) and adding alcohol SBI process fidelity metrics into performance quality reports.

For ASPPIRE, continuous program improvement was guided by the SBIRT Program Matrix.²² Progress was reported to health centers and staff via bi-monthly reports and newsletters. Strategies for sustaining alcohol SBI delivery and quality post-funding included certifying PPSNE staff as trainers using a Training-of-Trainers model, embedding alcohol SBI and FASD competencies into PPSNE’s online LMS, and incorporating alcohol SBI into new-hire training.

A Training and Technical Assistance Approach to Implementing Alcohol SBI in an Urban Safety Net System

Case 3: Boston HealthNet and The Boston Medical Center (BMC); The B SMART (Boston Sustainable Models for unhealthy Alcohol use ReducTion) Program.

Boston HealthNet serves minority and underserved populations with culturally sensitive care through a network of community health centers (CHCs) and a hospital (BMC). Receptivity to enhancing alcohol SBI through the B SMART program was mostly enthusiastic. The executive leadership was supportive and facilitated contacts between B SMART and clinic leadership; however, they had little authority over individual CHCs or BMC’s involvement in the program given that BMC and each CHC are managed independently. B SMART reached out to leaders at each primary care clinic whose responses ranged from “highly interested” to “unable to participate,” mostly due to competing priorities. BMC’s General Internal Medicine and Family Medicine departments and 10 out of 12 CHCs participated, with varying degrees and types of engagement.

Implementation Strategies

Strategies were initially targeted to the management and provider levels, providing education to BMC leadership on the topics of alcohol SBI, AEP risks, and FASDs. Since all clinics were implementing some level of alcohol SBI, B SMART offered tailored education, training, and TA to clinics interested in improving SBI services. B SMART emailed medical and behavioral health directors and grand round series course directors and coordinators. Once interest was expressed, B SMART met with clinic leadership and completed an environmental scan to understand the sites’ needs in terms of education and training, and how widely and effectively alcohol SBI was already being done. Staff at participating clinics were then trained together to establish baseline knowledge about excessive alcohol use, AEP, and FASDs and to understand how alcohol SBI fits into their practice. Healthcare teams were trained during existing educational venues (eg, grand rounds). Subsequent trainings were targeted to staff roles (providers, nurses, behavioral health, medical assistants) in delivering SBI and tailored to each clinic’s delivery model and workflow. Educational strategies included didactic and interactive formats, including “brainstorming,” case discussions, skills practice sessions, modeling provider-patient conversations, and a demonstration video. Continuing medical and nursing education (CME/CNE) credits were offered at all trainings. B SMART provided TA to facilitate the enhancement of the EHR to optimize the universal delivery of alcohol SBI. Strategies for promoting alcohol SBI sustainability included offering refresher training and ongoing TA to participating clinics.

Implementing Technology-Delivered Alcohol SBI (e-SBI) in Primary Care and Women’s Health Clinics

Case 4: Henry Ford Health (HFH) and SBI-Tech Michigan.

Henry Ford Health is a non-profit corporation comprised of hospitals and medical centers, including 50 primary care and 21 women’s health clinics. Receptivity for adopting alcohol SBI within HFH was driven by an interest in improving the integration of behavioral health within primary care. HFH stakeholders from primary care, women’s health, and behavioral health acknowledged that universal, standardized alcohol screening was not routinely occurring. Prior to funding, expansion of alcohol SBI efforts was not being actively sought by the HFH administration. However, HFH was receptive to the idea of integrating a technology-based version of alcohol SBI (e-SBI) if funded.

Implementation Strategies

A primary goal of HFH’s SBI-Tech Michigan program was to develop and implement e-SBI to reduce commonly reported provider and staff-level barriers (time, training, confidence) to screening and conducting BIs while facilitating key components of alcohol SBI.^16,31,32 System-level strategies used by the HFH information technology team included building a secure interface between the e-SBI software and EHR and developing a Best Practice Advisory (BPA) and referral order.

While waiting for their appointment, patients access e-SBI using tablets provided by front desk staff, with the screening and BI both being done by the software application. The e-SBI was provided in English only, and patients could opt out of receiving the e-BI after completing screening. Screening scores and a one-page summary for providers are sent from the e-SBI software and filed in the patient’s EHR. For patients scoring above recommended cutoffs, a BPA alerts the provider and includes talking points for initiating a discussion with the patient and the option to submit a referral to behavioral health for additional resources or treatment.

Clinic-level implementation strategies included formal appointment of physicians in leadership to serve as women’s health and primary care champions and assist in planning implementation procedures and troubleshooting problems. Within clinics, informal champions, usually clinic nurse managers, assisted with implementation.

Provider- and staff-level implementation strategies included e-learning modules combined with the use of publications, meetings, and training about alcohol and its effect on health and the importance of alcohol SBI. Quality improvement strategies included piloting in an exemplar clinic prior to scaling to additional clinics. The implementation team also provided regular clinic updates and progress reports. See Table 1 for a summary of key characteristics across the participating health systems. (Table 1).

Table 1.

Characteristics of Participating Health Systems.

Case summaries	Planned Parenthood Great Northwest, Hawai’i, Alaska, Indiana, Kentucky (PPGNHAIK)	Planned Parenthood of Southern New England (PPSNE)	Boston Medical Center (BMC)	Henry Ford Health (HFH)
Intervention goals	System-level implementation	System-level implementation	Alcohol SBI delivery enhancement	Substitute e-SBI; system-level implementation
Purpose (Why?)	Improve the ability of providers and staff to deliver interventions successfully	Improve the ability of providers and staff to deliver interventions successfully	Increase adoption by more settings; increase provider awareness of AEP and FASD	Increase adoption by more settings; address system and provider-level barriers
Performance site	Regional provider of reproductive and primary healthcare services; 35 health centers in 6 U.S. states; urban and rural	Regional provider of reproductive and primary healthcare services; 15 health centers in 2 Northeastern U.S. states; urban and rural	BMC, 12 community health centers; services included family medicine, general internal medicine, ob/gyn clinics; urban	Non-profit hospitals, medical centers, and large group practice; 15 clinics in southeastern and southcentral Michigan; urban/metropolitan area
Management Structure	Centralized leadership and decision-making	Centralized leadership and decision-making	Centralized affiliation with decentralized decision-making	Centralized administrative leadership; operational independence varied by clinics
Provider and staff involved in implementation	Front desk; physicians; advanced practice nurses; medical assistants	Front desk; physicians; advanced practice nurses; medical assistants	Administrators; physicians; nurse practitioners; nurses; medical assistants; behavioral health	Front desk, administrative staff; primary care providers; behavioral health
Patient population	140 000 patients; ~88% female, 90% ages 18-44, ~38% at or below the federal poverty level	72 000 patients; ~90% female, 75% ages of 18-44, ~50% Medicaid or Medicare; 25% uninsured	Patients in ambulatory care; ~55% female	Primary care and women’s health services patients; ~60% female
Primary implementation approach	Multidisciplinary planning team and practice facilitation; continuous quality improvement	Practice-academic partnership; continuous quality improvement	Training targeted to management and provider levels; technical assistance offered to sites	Systems integration of e-SBI; formally appointed physician leader champions; piloting, continuous QI
Implementation framework	RE-AIM; CDC implementation guide; logic model	SBIRT program matrix; logic model	Logic model	Logic model

Abbreviations: Alcohol SBI, alcohol screening and brief intervention; AEP, alcohol-exposed pregnancy; CDC, Centers for Disease Control and Prevention; e-SBI, electronic-screening and brief intervention; FASD, fetal alcohol spectrum disorders; QI, Quality Improvement; RE-AIM, Reach, Effectiveness, Adoption, Implementation, Maintenance; SBIRT, screening, brief intervention, referral to treatment.

Planned and Unplanned Adaptations to Strategies

Throughout implementation, planned and unplanned adaptations were required to respond to dynamic circumstances. Planned adaptations that all projects used included modifying training materials to reflect patient demographics, adding educational content on FASDs for providers, and promoting a universal approach to alcohol SBI that included strategies for preventing AEP (ie, asking about any alcohol use). For example, staff training and patient education materials were adapted to include “case” scenarios for patients who were drinking at lower risk levels, but due to their alcohol use and current contraception practices, were at risk for AEP. Planned systems-level strategies to adapt infrastructure, such as modifying the EHR to include the screening instrument and automated scoring, scripted statements for introducing screening questions to patients, provider alerts for patients that screen positive, fields for documenting alcohol SBI encounters, and codes that support alcohol SBI billing, were accomplished in the health systems where authority to make system-level modifications was centralized. In the health systems with decentralized management structures, modifying the EHR depended on local authority, capacity, and willingness to make the required changes. Strategies for monitoring alcohol SBI quality also varied by health setting, with some healthcare systems establishing performance goals (eg, 80% of eligible patients/visits screened) and then monitoring reach and fidelity using EHR-generated data. Performance data were communicated via standardized dashboards, newsletters, or reports, and used to identify systems glitches, booster training needs, or necessary refinements.

The health systems with decentralized decision-making structures require customization of implementation strategies at the clinic level. For example, in a system like Boston HealthNet, where sites are autonomous, pre-implementation strategies that targeted the network and administrator levels were developed to increase awareness and receptivity to enhancing alcohol SBI protocols. At the clinic level, implementation strategies were informed proactively by conducting environmental scans to identify site-level needs, such as adapting systems and infrastructure to support universal alcohol SBI. Accomplishing such systems-level adaptations was complicated by the variety of EHRs in use by clinics. While most of these systems already contained some universal screening protocols, and all systems used moderate drinking guidelines to define recommended drinking limits,³³ screening practices varied widely. In some cases, screening prompts were added to the EHR for staff and decision support with recommended education, intervention, or referral being added for point-of-care providers.

Evaluations of e-SBI for alcohol use in the perinatal period have shown preliminary but encouraging results.^31,32,34-36 The complexity of integrating e-SBI into visit systems and workflows required the SBI-Tech Michigan team to react with some unplanned adaptations. For example, a planned implementation strategy was to use e-learning to reduce the burden and facilitate consistency of training clinic teams, a method that has repeatedly demonstrated its efficacy in building knowledge and skills, and in reviews has shown better results than face-to-face learning.³⁷ However, providers expressed a lack of interest in e-learning models that required dedicated time to complete. In response, HFH used a multi-pronged training approach with providers—email, grand rounds, and clinic huddles—that did not require separate scheduling, and also offered one-on-one training, consistent with preceptor training models. Also, the initial BPA was designed to be minimally disruptive to providers’ workflow but was not attended to by providers. Following provider feedback, the BPA was adapted to be more “interruptive” in the Epic EHR workflow.

An unanticipated cross-cutting change in implementation circumstances was the COVID-19 pandemic, which disrupted clinical services across all implementation sites due to paused delivery of prevention services for varying periods of time, increased use of telehealth, and staff shortages and turnover. Adaptations made in reaction to this unforeseen change varied by program site. At the reproductive healthcare sites, the continued uncertainty of the pandemic and limited access to sites was addressed by converting live or hybrid training strategies to all-virtual platforms, such as digital modules delivered asynchronously through the health systems’ LMS, synchronous online modules delivered during clinical staff meetings, and facilitated practice/role play sessions delivered via remote videoconferencing.

Since all training ceased for the Boston HealthNet sites during the spring and summer of 2020, B SMART produced a publicly available three-part podcast to continue their education strategy for healthcare teams, providing CME/CNE credits to listeners.³⁸ Once training began again virtually, additional topics of interest and interactive online formats were added.

The pandemic also prompted unplanned adaptations for HFH’s e-SBI. The original design was to have patients complete e-SBI on clinic-owned tablets while in the waiting area. Distancing precautions, along with increased use of telehealth, resulted in fewer in-person appointments and limited the time patients spent in the waiting area. In response, the e-SBI software was adapted for patients to use on personal devices, allowing them to complete the e-SBI from home using a link sent via patient portal message prior to their appointment. Table 2 provides a cross-case summary of planned and unplanned adaptations to strategies.

Table 2.

Planned and Unplanned Adaptations to Implement Alcohol SBI for AEP Prevention in Primary Care Settings, by Project Site and CDC Logic Model.

Implementation strategy	PPGNHAIK	PPSNE	BMC	HFH
Adapt training content and resources to reflect practice and patient needs
Other substances	X	X	X	X+
Gender-affirming/inclusive language	X+	X	X+	X+
COVID-19 and alcohol			X+
Adolescents and alcohol use			X+
Train and provide technical assistance (TA)
In-person SBI training	X	X	X
Coaching and technical assistance	X	X	X	X+
SBI training in new employee onboarding	X	X
Train-the-trainer with in-house educators		X
Recorded training resources	X	X	X	(X)
e-learning (asynchronous)	X	X+	X+	(X)
Remote/virtual training (synchronous)	X+	X+	X+	X+
Adapt and implement alcohol SBI protocols
Engaged health system representatives in planning and implementation (internal champions, leadership, key roles)	X	X	X	X
Adapt health system infrastructure to support alcohol SBI implementation
Embedded screening tools in her	X	(X)	X	X
Added SBI prompts in her	X	X	X	X
Added SBI visit documentation fields	X	X	X
Used SBI billing codes for reimbursement	X	X
Incorporated SBI quality metrics into health system reports	X	(X)	(X)	(X)
Distribution/posting of educational materials	X	X	X+	(X)
Evaluate implementation process and outcomes
Evaluated feasibility of systems-level implementation	X	X	X	X
Monitored fidelity of SBIRT delivery to inform continuous process improvement	X	X	(X)	X
Disseminate key information
Health system use of monitoring data	X	X	(X)	X
Community-level education and awareness		X+	X	(X)
Implement a sustainability plan
Identify sustainability threats and strengths	X	X	X	X
Develop new staff training protocols	X	X		X
Develop online training materials	X+	X	X+	X

Abbreviations: BMC, Boston Medical Center; EHR, electronic health records; HFH, Henry Ford Health; PPGNHAIK, Planned Parenthood Great Northwest, Hawai’i, Alaska, Indiana, Kentucky; PPSNE, Planned Parenthood Southern New England; SBI, alcohol screening and brief intervention; SBIRT, Screening Brief Intervention and Referral to Treatment; X, Planned and implemented; X+, Implemented but not planned (added); (X), Planned but not implemented (dropped)

Supports, Challenges, and Lessons Learned

The 4 project teams identified key questions associated with successful implementation at each SBIRT Program Matrix level, drawing on constructs from the implementation science literature.^23,24 These questions can be used to guide adaptations made to content or implementation methods, whether planned or unplanned, to identify supports and address challenges identified at each level. Table 3 summarizes key questions, challenges, and suggested strategies to pre-empt or address challenges related to alcohol SBI implementation.

Table 3.

Recommended Strategies for Addressing Key Implementation Challenges by SBIRT Program Matrix Levels.

Level	Key questions	Challenges	Strategies for improving receptivity
Performance site receptivity	Is there internal motivation to adopt alcohol SBI now? Is there a perception that alcohol SBI is being “pushed” from outside? Does alcohol SBI align with priorities, culture, and philosophy? What are the competing priorities? Are decision-makers and providers receptive?	If buy-in is perceived as a top-down or external mandate, may encounter resistance. If clinics make training and alcohol SBI screening optional, reliance on provider discretion may result in missed or singled-out patients. Reimbursement may be inadequate to off-set clinic-level costs	• Communicate early and actively• Align project goals with patient health and organizational priorities• Involve providers and staff in brainstorming and planning• Solicit provider and staff feedback to improve procedures• Customize workflows and training to settings based on what is possible• Customize patient educational materials• Integrate information into existing processes and resources
Level	Key questions	Challenges	Strategies to address structural challenges
Management structure	How are decisions made? Is there skills, will, and resources to modify systems to accommodate alcohol SBI?	EHR constraints (eg, cost, expertise, software); inability to customize locally. Combining alcohol SBI and AEP prevention information may be challenging (data are in different places requiring complex programming)	• Customize approaches based on constraints• Involve system administrators and leaders in logistics planning and execution• Program EHR and/or integrate screening into intake protocols, to prompt universal delivery• Create EHR algorithms or flows that prompt providers to counsel on AEP prevention for people who are or might be pregnant• Design procedures; adapt systems, ahead of training• Train on procedures as well as practice changes• Monitor performance quality (EHR and staff feedback)
Level	Key questions	Challenges	Strategies to raise alcohol awareness, knowledge, and confidence of providers and staff
Provider and staff attributes	What do providers and staff know and believe about alcohol use and health? Do providers and staff believe that alcohol SBI is effective?	The belief is that alcohol use is a behavioral health issue. The belief is that addressing alcohol is outside the scope or skill set for the role. The belief is that there is no time to adequately address	• Involve providers and staff in planning, solicit feedback• Address time concerns in training (eg, demonstrate conducting a brief intervention if you only have 2 minutes; remind providers they can bring someone back in for a longer appointment, if necessary)• Incorporate technology solutions into workflows and processes, including adding e-SBI
	Key questions	Challenges	Strategies to raise FASD awareness and knowledge
	What do providers and staff know and believe about FASDs?	Lack of clinician knowledge about FASD prevention, EHR constraints limit integrating multiple behavioral risk factors into the BI (eg, alcohol and contraception)	• Provide FASD prevention education as a way to expand alcohol SBI services• Incorporate FASD prevention into training, workflows, and performance monitoring
	Key questions	Challenges	Strategies to enhance confidence
	Do providers and staff have the skills and confidence to discuss alcohol and pregnancy?	Staff and provider resistance (do not see a need or value). Staff and providers lack confidence; concerns about stigma and privacy	• Address stigma and privacy issues in training, provide guidance language, and model the use of clinically accurate, non-stigmatizing terms and person-first language.• Reinforce universal screening, for example, “we ask all our patients these questions”• Provide opportunities to practice introducing screening and conducting brief interventions• Provide opportunities post-training to discuss successes and challenges• Provide opportunities for booster training• Develop scripts and protocols for discussing alcohol use and contraception to support informed choices• Incorporate information on alcohol and pregnancy into patient education materials
Level	Key questions	Challenges	Strategies to increase patient reach
Patients	Are eligible patients identified and reached to receive alcohol SBI?	Screening is informal or discretionary. Brief interventions are performed inconsistently	• Integrate screening into intake procedures• Inform patients that you ask everybody• Consider technology-delivered SBI (e-SBI), in tandem with electronic patient self-registration
	Key questions	Challenges	Strategies to educate patients on alcohol and pregnancy
	Are patients at risk for AEP consistently identified and reached with prevention information?	Reliance on providers to assess AEP risk based on multiple behaviors. Patients drinking at low levels not flagged for BI by standardized screeners	• Develop patient handouts and educational materials on alcohol use and health including ANY use during pregnancy• Develop waiting room and exam room posters or informational video loops on preventing FASDs

Abbreviations: AEP, alcohol-exposed pregnancy; alcohol SBI, alcohol screening and brief intervention; BI, brief intervention; EHR, electronic health record; e-SBI, electronic-screening and brief intervention; FASD, Fetal Alcohol Spectrum Disorders.

Discussion

The findings of these case studies replicate previously noted supports and challenges associated with implementing alcohol SBI including the need for multi-level buy-in and alignment with priorities,³⁹ the added complexity of working in systems with decentralized authority to make system-level changes,⁴⁰ the constraints related to modifying EHRs,⁴¹ and the concerns about provider and staff burden.¹¹ Although teams planned to adapt implementation strategies to improve SBI fit and feasibility, even planned strategies may require unplanned adaptations to improve receptivity to making the necessary changes or to address unanticipated barriers to implementing them effectively. We discuss some of these examples below.

Adapting alcohol SBI training and content for specific target populations was planned, but doing so required unanticipated system-level adaptations. For example, all project teams expected to adapt alcohol SBI for patients who may be at risk for AEPs, which required modifying the interpretation of screening tools as well as expanded intervention protocols and training content. Although this planned adaptation provided an opportunity to educate healthcare systems and teams about FASD and their role in prevention, it also required providers to address any alcohol use (ie, change the definition of lower-risk use) for the subset of patients who are pregnant or might be pregnant. Thus, addressing the elevated risk of AEP required additional changes to screener interpretation and inclusion of other relevant information, such as sexual history and contraception practices. These adaptations were challenging from a systems perspective, given the need to combine data from different places within the EHR to alert providers. Such complex programming was not easily executed within the participating health systems’ EHRs and workflows and will continue to require future adaptation.

Adaptability of health system infrastructure to support alcohol SBI was influenced by health system management structures. The ability to proactively adapt alcohol SBI so that it is delivered consistently and universally within a setting was facilitated by health system management structures that authorized system-level changes such as modifying the EHR to embed screening tools and procedures, and documentation fields. While centralized management structures promoted the efficiency of planning and adapting alcohol SBI for uniform dissemination across multiple sites, we observed that decentralized management structures required greater site-level participation, local champions, and customization of approaches to enhance acceptance among providers and staff. Regardless of structure, the involvement of key personnel in logistical planning for implementation, including systems administration and quality improvement staff, facilitates the integration of alcohol SBI into routine practice.

Evaluation of implementation process and outcomes also varied by health system structure. Although all 4 of our cases collected data for evaluation purposes, only 3 of the health systems were able to adapt the EHR to document delivery of alcohol SBI key components during the patient visit. The adapted EHR provided these systems with an integrated method for monitoring the reach and consistency of alcohol SBI delivery that could be used to inform continuous quality improvement.

Health systems that authorized adapting their systems and procedures to accommodate alcohol SBI demonstrated fewer unplanned adaptations.¹¹ While receptivity from health system leadership and providers is fundamental to practice change, the willingness or ability of health systems to invest administrator, provider, and staff time to participate in pre-implementation planning and start-up activities, including adapting the EHR and procedures, requires an upfront commitment. Our study found that the number and type of adaptations made to alcohol SBI implementation strategies varied by management structure. Centralized structures using more planned adaptations than unplanned adaptations, dropped relatively few planned strategies, likely due to an emphasis on pre-implementation planning and structural-level changes that promoted consistency of delivery and were maintained when the pandemic struck. Planned adaptations to enhance cultural acceptability or support fidelity of intervention delivery, whether they involve tailoring training or customizing systems, are considered enhancements provided they do not dilute the reach or effectiveness of alcohol SBI.¹⁸ However, unplanned adaptations, sometimes dismissed as improvisation, can still promote the goals of implementing an evidence-based intervention if they relate to the core implementation or pre-implementation strategies. These include responding to unanticipated patient visit protocol changes such as those implemented during the pandemic, which required unplanned adaptations to allow patients to access screening from their personal devices; and responding to unanticipated site-level barriers in adopting alcohol SBI that required pre-implementation FASD education to increase receptivity to implementing or expanding delivery of alcohol SBI.^18-20

The COVID-19 pandemic presented unanticipated challenges across all 4 cases, including competing clinical priorities related to patient care, shutdowns, staff shortages, the shift to telehealth, and changes to the location and length of patient visits. The importance of rapidly reacting and adapting strategies to accommodate the diverse needs of the healthcare systems⁴² provided an opportunity to innovate while still advancing the goals of alcohol SBI implementation and AEP/FASD prevention.

Limitations

While we used a theoretically informed logic model and implementation frameworks to guide the 4 cases, themes, and lessons learned, our data sources were the observations and reflections of the authors, as opposed to standardized measures. Also, we are unable to compare the relative success of the implementation strategies used across projects using a common metric, such as screening and brief intervention rates, given differences in EHR documentation and access to data across the 4 cases. In addition, since implementation approaches were targeted to different levels of the SBIRT Program Matrix depending on each health system’s receptivity, capacity, priorities, and unique challenges, other indicators, such as variation in the use of different types of strategies, whether planned or unplanned, were more aligned with this paper’s focus. Finally, while the 4 cases illustrate implementation approaches used within different types of healthcare settings and management structures, healthcare delivery is context-specific; thus, the lessons from our 4 cases may not be generalizable to all types of settings.

Conclusions

This paper describes the experiences of 4 multidisciplinary project teams and the strategies used to promote the adoption and implementation of alcohol SBI in primary care settings to identify and address excessive alcohol use. A key goal was to adapt alcohol SBI so that it also identified patients at risk for AEPs. The implementation strategies used were multi-level, targeting organizational receptivity, provider and staff knowledge and beliefs, and patient needs for information, as well as system-level factors in alignment with management structures (eg, centralized and decentralized administration and operations support). While most of the implementation strategies used were planned, the COVID-19 pandemic required adjustments to implementation plans. In addition, there were unanticipated internal challenges related to expanding the use of alcohol SBI to identify patients whose combined alcohol consumption and pregnancy prevention practices elevated their risk for AEP. This required further adaptations to brief intervention procedures to address multiple risk behaviors. A key lesson of this work is that unplanned adaptations do not necessarily detract or drift away from implementation goals and may even provide opportunities to enhance the adoption or delivery of the intervention, provided they consider the potential for negative impacts on outcomes. Areas for future study include measuring the short- and long-term impact of planned and unplanned adaptations to implementing alcohol SBI, and whether strategies improve the use of alcohol SBI to address and prevent AEP and FASDs.

Research Data

sj-docx-1-saj-10.1177_29767342241271404 – Supplemental material for Using Planned and Unplanned Adaptation to Implement Universal Alcohol Screening and Brief Intervention to Prevent Alcohol-Exposed Pregnancies in Four Primary Care Health Systems

Supplemental material, sj-docx-1-saj-10.1177_29767342241271404 for Using Planned and Unplanned Adaptation to Implement Universal Alcohol Screening and Brief Intervention to Prevent Alcohol-Exposed Pregnancies in Four Primary Care Health Systems by Diane K. King, Steven J. Ondersma, Bonnie G. McRee, Jacqueline S. German, Amy M. Loree, Amy Harlowe, Daniel P. Alford, Robyn N. M. Sedotto and Mary Kate Weber in Substance Use & Addiction Journal

This article is distributed under the terms of the Creative Commons Attribution 4.0 License (http://www.creativecommons.org/licenses/by/4.0/) which permits any use, reproduction and distribution of the work without further permission provided the original work is attributed as specified on the SAGE and Open Access pages (https://us.sagepub.com/en-us/nam/open-access-at-sage).

Footnotes

Acknowledgements

The authors wish to acknowledge our partners and collaborators at Boston HealthNet; Henry Ford Health; Planned Parenthood Great Northwest, Hawai’i, Alaska, Indiana, Kentucky; and Planned Parenthood of Southern New England.

Authors’ Contributions

DKK, SJO, BGM, and DPA obtained funding and directed the implementation of the projects. MKW provided content expertise, program input, and critical feedback. DKK, SJO, BGM, and JSG developed the methods used in the manuscript. All authors contributed to writing the case presentations, interpreting the results, completing and editing the tables, and reading and approving the final manuscript draft for submission.

Availability of Data and Materials

Yes.

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: All sites were supported by cooperative agreements from the CDC awarded under CDC-RFA-DD18-1802: (grant numbers NU84DD000001 ([Henry Ford Health], NU84DD000002 [Boston University School of Medicine], NU84DD000004 [University of Connecticut School of Medicine], NU84DD000005 [University of Alaska Anchorage]). The findings and conclusions in this report are those of the authors and do not necessarily represent the official position of the Centers for Disease Control and Prevention.

Compliance,Ethical Standards,and Ethical Approval

Ethics approval was carried out in compliance with each project’s respective institution’s review boards: University of Alaska Anchorage Institutional Review Board (expedited), University of Connecticut School of Medicine (exempt), Henry Ford Health Institutional Review Board (expedited), and Boston University Medical Campus and Boston Medical Center Institutional Review Board (expedited).

Consent for Publication

Consent for publication was obtained by participating institutions.

ORCID iD

Diane K. King

References

Centers for Disease Control and Prevention. Alcohol Use During Pregnancy. Updated 2023. https://www.cdc.gov/ncbddd/fasd/alcohol-use.html

May

Gossage

JP.

Maternal risk factors for fetal alcohol spectrum disorders: not as simple as it might seem. Alcohol Res Health. 2011;34(1):15-26.

Centers for Disease Control and Prevention. Advisory on alcohol use in pregnancy: a 2005 message to women from the U.S. surgeon general. Updated 2005. https://www.cdc.gov/ncbddd/fasd/documents/sg-advisory.pdf

USPSTF. Screening and behavioral counseling interventions in primary care to reduce alcohol misuse: recommendation statement. AHRQ Publication No. 12-05171-EF-3; 2013.

Fleming

Mundt

French

Manwell

Stauffacher

Barry

KL.

Brief physician advice for problem drinkers: long-term efficacy and benefit-cost analysis. Alcohol Clin Exp Res. 2002;26(1):36-43.

Esser

Hedden

Kanny

Brewer

Gfroerer

Naimi

TS.

Prevalence of alcohol dependence among US adult drinkers, 2009-2011. Prev Chronic Dis. 2014;11:E206. doi:10.5888/pcd11.140329

Babor

Higgins-Biddle

Dauser

Higgins

Burleson

JA.

Alcohol screening and brief intervention in primary care settings: implementation models and predictors. J Stud Alcohol. 2005;66(3):361-368. doi:10.15288/jsa.2005.66.361

O’Donnell

Anderson

Newbury-Birch

, et al. The impact of brief alcohol interventions in primary healthcare: a systematic review of reviews. Alcohol Alcohol. 2014;49(1):66-78. doi:10.1093/alcalc/agt170

Poole

Schmidt

Bocking

Bergeron

Fortier

The potential for fetal alcohol spectrum disorder prevention of a harmonized approach to data collection about alcohol use in pregnancy cohort studies. Int J Environ Res Public Health. 2019;16(11):2019.doi:10.3390/ijerph16112019

10.

Rahm

Boggs

Martin

, et al. Facilitators and barriers to implementing screening, brief intervention, and referral to treatment (SBIRT) in primary care in integrated health care settings. Subst Abus. 2015;36(3):281-288. doi:10.1080/08897077.2014.951140

11.

King

Gonzalez

Hartje

, et al. Examining the sustainability potential of a multisite pilot to integrate alcohol screening and brief intervention within three primary care systems. Transl Behav Med. 2018;8(5):776-784. doi:10.1093/tbm/ibx020

12.

Rosario

Santos

Angus

Pas

Ribeiro

Fitzgerald

Factors influencing the implementation of screening and brief interventions for alcohol use in primary care practices: a systematic review using the COM-B system and theoretical domains framework. Implement Sci. 2021;16(1):6. doi:10.1186/s13012-020-01073-0

13.

Floyd

Sobell

Velasquez

, et al. Preventing alcohol-exposed pregnancies: a randomized controlled trial. Am J Prev Med. 2007;32(1):1-10. doi:10.1016/j.amepre.2006.08.028

14.

Edwards

Kelsey

Pierce-Bulger

, et al. Applying ethical principles when discussing alcohol use during pregnancy. J Midwifery Womens Health. 2020;65(6):795-801. doi:10.1111/jmwh.13159

15.

Vendetti

Gmyrek

Damon

Singh

McRee

Del Boca

Screening, brief intervention and referral to treatment (SBIRT): implementation barriers, facilitators and model migration. Addiction. 2017;112(Suppl 2):23-33. doi:10.1111/add.13652

16.

USDHHS. Alcohol—excessive consumption: electronic screening and brief interventions (e-SBI). Updated 2012. Accessed April 27, 2022. https://www.thecommunityguide.org/findings/alcohol-excessive-consumption-electronic-screening-and-brief-interventions-e-sbi

17.

Centers for Disease Control and Prevention. Planning and Implementing Screening and Brief Intervention for Risky Alcohol Use: A Step-by-Step Guide for Primary Care Practices. Centers for Disease Control and Prevention, National Center on Birth Defects and Developmental Disabilities; 2014.

18.

Wiltsey Stirman

Baumann

Miller

CJ.

The FRAME: an expanded framework for reporting adaptations and modifications to evidence-based interventions. Implement Sci. 2019;14(1):58. doi:10.1186/s13012-019-0898-y

19.

Chambers

Norton

WE.

The adaptome: advancing the science of intervention adaptation. Am J Prev Med. 2016;51(4 Suppl 2):S124-S131. doi:10.1016/j.amepre.2016.05.011

20.

Kirk

Moore

Wiltsey Stirman

Birken

SA.

Towards a comprehensive model for understanding adaptations’ impact: the model for adaptation design and impact (MADI). Implement Sci. 2020;15(1):56. doi:10.1186/s13012-020-01021-y

21.

Bender

Cvietusa

Goodrich

King

Shoup

JA.

Adapting adaptive design methods to accelerate adoption of a digital asthma management intervention. Transl Behav Med. 2023;13(3):149-155. doi:10.1093/tbm/ibac093

22.

Del Boca

McRee

Vendetti

Damon

The SBIRT program matrix: a conceptual framework for program implementation and evaluation. Addiction. 2017;112(Suppl 2):12-22. doi:10.1111/add.13656

23.

McCreight

Rabin

Glasgow

, et al. Using the practical, robust implementation and sustainability model (PRISM) to qualitatively assess multilevel contextual factors to help plan, implement, evaluate, and disseminate health services programs. Transl Behav Med. 2019;9:1002-1011doi:10.1093/tbm/ibz085

24.

Damschroder

Aron

Keith

Kirsh

Alexander

Lowery

JC.

Fostering implementation of health services research findings into practice: a consolidated framework for advancing implementation science. Implement Sci. 2009;4:50. doi:10.1186/1748-5908-4-50

25.

Lanham

Palmer

Leykum

, et al. Trust and reflection in primary care practice redesign. Health Serv Res. 2016;51(4):1489-514. doi:10.1111/1475-6773.12415

26.

Kirchner

Ritchie

Pitcock

Parker

Curran

Fortney

JC.

Outcomes of a partnered facilitation strategy to implement primary care-mental health. J Gen Intern Med. 2014;29(Suppl 4):904-912. doi:10.1007/s11606-014-3027-2

27.

Higgins-Biddle

Babor

TF.

A review of the alcohol use disorders identification test (AUDIT), AUDIT-C, and USAUDIT for screening in the United States: past issues and future directions. Am J Drug Alcohol Abuse. 2018;44(6):578-586. doi:10.1080/00952990.2018.1456545

28.

Glasgow

Harden

Gaglio

, et al. RE-AIM planning and evaluation framework: adapting to new science and practice with a 20-year review. Front Public Health. 2019;7:64. doi:10.3389/fpubh.2019.00064

29.

Holtrop

Estabrooks

Gaglio

, et al. Understanding and applying the RE-AIM framework: clarifications and resources. J Clin Transl Sci. 2021;5(1):e126. doi:10.1017/cts.2021.789

30.

Baskerville

Liddy

Hogg

Systematic review and meta-analysis of practice facilitation within primary care settings. Ann Fam Med. 2012;10(1):63-74. doi:10.1370/afm.1312

31.

Wouldes

Crawford

Stevens

Stasiak

Evidence for the effectiveness and acceptability of e-SBI or e-SBIRT in the management of alcohol and illicit substance use in pregnant and post-partum women. Front Psychiatry. 2021;12:634805. doi:10.3389/fpsyt.2021.634805

32.

Tansil

Esser

Sandhu

, et al. Alcohol electronic screening and brief intervention: a community guide systematic review. Am J Prev Med. 2016;51(5):801-811. doi:10.1016/j.amepre.2016.04.013

33.

National Institute on Alcohol Abuse and Alcoholism. Drinking Levels Defined. Updated 2022. https://www.niaaa.nih.gov/alcohol-health/overview-alcohol-consumption/moderate-binge-drinking

34.

Ondersma

Beatty

Svikis

, et al. Computer-delivered screening and brief intervention for alcohol use in pregnancy: a pilot randomized trial. Alcohol Clin Exp Res. 2015;39(7):1219-1226. doi:10.1111/acer.12747

35.

Yonkers

Dailey

Gilstad-Hayden

, et al. Abstinence outcomes among women in reproductive health centers administered clinician or electronic brief interventions. J Subst Abuse Treat. 2020;113:107995. doi:10.1016/j.jsat.2020.02.012

36.

Loree

Yonkers

Ondersma

Gilstad-Hayden

Martino

Comparing satisfaction, alliance and intervention components in electronically delivered and in-person brief interventions for substance use among childbearing-aged women. J Subst Abuse Treat. 2019;99:1-7. doi:10.1016/j.jsat.2019.01.007

37.

Pei

Does online learning work better than offline learning in undergraduate medical education? A systematic review and meta-analysis. Med Educ Online. 2019;24(1):1666538. doi:10.1080/10872981.2019.1666538

38.

B Smart Program. Alcohol and pregnancy: the more you know. https://podcasts.apple.com/us/podcast/alcohol-and-pregnancy-the-more-you-know/id1558188509

39.

Goldstein

Vogt

Hamilton

, et al. Practice-based research networks add value to evidence-based quality improvement. Healthc (Amst). 2018;6(2):128-134. doi:10.1016/j.hjdsi.2017.06.008

40.

Monks

Operational research as implementation science: definitions, challenges and research priorities. Implement Sci. 2016;11(1):81. doi:10.1186/s13012-016-0444-0

41.

Cifuentes

Davis

Fernald

Gunn

Dickinson

Cohen

DJ.

Electronic health record challenges, workarounds, and solutions observed in practices integrating behavioral health and primary care. J Am Board Fam Med. 2015;28(Suppl 1):S63-S72. doi:10.3122/jabfm.2015.S1.150133

42.

Thakur

Soklaridis

Crawford

Mulsant

Sockalingam

Using rapid design thinking to overcome COVID-19 challenges in medical education. Acad Med. 2021;96(1):56-61. doi:10.1097/ACM.0000000000003718

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.01 MB