From evidence to practice: Using the RE-AIM framework to adapt the REACHII caregiver intervention to the community

Abstract

This article describes how the multi-step mid-course assessment of the REACH II community translation project in North Carolina was guided by the RE-AIM framework, and summarizes adaptations made to enhance the feasibility of adoption and maintenance while at the same time assuring fidelity to program core elements. The two-stage assessment involved both quantitative (survey) and qualitative (discussion group) components. Results indicated a need to focus primarily on tailoring pre-intervention training, streamlining and clarifying intervention guides and tools, targeting specific participant recruitment messages, addressing issues of session length, and clarifying what flexibilities family consultants could exercise in terms of specific session content addressed and other supportive materials used. The use of the RE-AIM framework and the mixed-method process allowed the program staff to thoroughly assess program satisfaction and areas of concern, and ultimately ensured that the family consultants implementing the intervention had a voice in the adaptation process.

Keywords

dementia caregiver RE-AIM mid-course assessment REACH intervention

Introduction

Over 20 years of caregiver intervention research has generated significant, clinically relevant outcomes and effective strategies for helping dementia family caregivers. Evidence-based interventions have proven effective in mitigating issues such as caregiver depression, stress, family conflict and isolation, reducing care-related illness, improving satisfaction with social supports, enhancing daily coping skills and delaying institutionalization without overburdening family members (Belle et al., 2006; Burgio, Stevens, Guy, Roth & Haley, 2003; Ducharme, Levesque, Giroux & Lachance, 2005; Mittelman, Haley, Clay & Roth, 2006; Roberts & Brown, 2007; Schulz, 2000; Schulz, Martire & Klinger, 2005).

However, there has been criticism that many family caregiver interventions are limited to a single-problem focus rather than the complex interplay of stress, depression, health risk and burden, and further, that many of the intervention studies lack scientific rigor (Belle et al., 2006). To address these issues, the US National Institute on Aging and the National Institute on Nursing Research co-funded a two-phased, multi-site randomized control trial, Resources for Enhancing Alzheimer’s Caregiver Health (REACH). In the first phase, researchers tested several promising approaches with more than 600 Hispanic, black and white caregiver/care recipient dyads across six US sites with the aim of reducing burden and depression. The findings of the most effective strategies served as the basis for formulating the content and protocols for a second phase intervention called REACH II.

The REACH II skill-building and problem-solving program was tested over six months with over 1800 ethnically and culturally diverse dementia caregivers at six US sites. REACH II was designed to help family caregivers manage depression and burden, improve skills for self-care and healthy behaviors; better use social support; reduce risk for care recipients and increase capacity for family care at home. A graduate-prepared interventionist provided nine in-home individual sessions lasting one hour each and three telephone individual sessions lasting up to 30 minutes. An intervention manual included 12 session protocols as well as forms for risk appraisal, problem solving, behavioral prescriptions, and fidelity monitoring. The interventionist focused on skills-based teaching and active learning through role playing, problem-solving practice, and teaching of specific self-care techniques like relaxation and pleasant events. Caregivers received and added to a Caregiver Notebook and were encouraged to practice “homework” between sessions. A separate trained professional group leader guided and facilitated five one-hour weekly structured and moderated telephone support group sessions for five to six participants in the experimental group, reviewing topical information and sharing concerns. The REACH II experimental group experienced significantly greater improvements in quality of life measures compared to controls (Belle et al., 2006).

In 2008, the North Carolina Division of Aging and Adult Services (NC DAAS) successfully applied for a three-year US Administration on Aging (AoA) “Alzheimer’s Disease Supportive Services Evidence-Based Intervention Grant to Better Serve People with Alzheimer’s Disease and Related Disorders.” The overall aim of the state grant is the effective translation of the REACH II intervention for sustained use at the community level. NC REACH II is unique in that it has been embedded as a value-added enhancement to a comprehensive dementia-specific respite program, NC Project C.A.R.E. (Caregiver Alternatives to Running on Empty), developed with previous AoA Alzheimer’s demonstration funding (Kelly & Williams, 2007).

The grant project evaluation has focused on whether selected Project C.A.R.E. “family consultants” and their host agencies could feasibly adopt and deliver the NC REACH II program with fidelity; whether NC REACH II would result in measurable benefits comparable to the original trial; whether low-income rural and minority families could be reached; and whether costs of delivering the program were feasible for sustainable integration into community long-term services and supports beyond the project funding.

NC REACH II was rolled out in successive stages (years 2009 and 2010) in three different regions in the state. As part of our process evaluation, we conducted annual mid-course assessments approximately one year into implementation (in 2010, 2011). Family consultants who were part of the first implementation cohort participated in both years of the mid-course assessments and were joined by a new wave of family consultants in the second year. The purpose of the assessments was to serve as check points to review the implementation process, build on demonstrated strengths, identify problem areas and needs for adaptation to better fit service delivery, and modify operations based on what was not working. We considered this assessment to be pivotal to enhancing adoption of the program by the family consultants and stakeholders and to maintaining NC REACH beyond the funding period. This article describes the assessment steps, how the RE-AIM framework was used, and the programmatic changes made as a result of the findings.

RE-AIM framework

While RE-AIM was originally developed as a framework for consistent reporting of intervention research results (Glasgow, Vogt & Boles, 1999; Glasgow, Whitlock, Eakin & Lichtenstein, 2000), it has more recently been used to help guide program planning to improve the chances of an intervention working in “real-world” settings. The framework guides the delineation of indicators for evaluating processes and outcomes that can help inform decision-making for sustainable program adoption and implementation issues (Green and Glasgow, 2006; Klesges, Estabrooks, Dzewaltowski, Bull & Glasgow, 2005).

The framework addresses and monitors five essential program components: REACH, EFFECTIVENESS, ADOPTION, IMPLEMENTATION, and MAINTENANCE. The REACH component focuses on the population with the health issue being addressed and answers the overall question, “Is the right population being reached?” Specific REACH indicators assess whether target audiences are participating in the program, in what numbers, and the percentage of program completion and attrition. Some typical questions include: “Is the program reaching the targeted number of participants and completers? Is it reaching the population with the targeted health-related characteristics and demographics? Are marketing efforts effective in recruiting participants?”

EFFECTIVENESS focuses on program impact and answers the question, “Does the program produce positive benefits for the health issue being addressed?” Specific EFFECTIVENESS indicators assess both the intended and unintended impact of the program and its value to stakeholders. Examples of key questions include: “Are participants achieving the same outcomes (e.g. improved self-efficacy, health behaviors, symptoms, health care utilization, costs and other outcomes) as in the published research studies? Are there any unanticipated or potentially negative effects? Is there a program value/return on investment for stakeholders?”

ADOPTION focuses on staff, partnering organizations and settings where the program is offered and answers the question, “Can the program be taken to “scale” (offered in all the places it needs to be offered)?” Specific ADOPTION indicators assess the adequacy of the workforce and implementation sites. Typical questions include: “How many partner organizations/ host sites have adopted the program? How many implementation sites are delivering the workshops/ program activities and with what frequency? Have a sufficient number of trained staff and volunteers (workforce) been recruited and trained? Are the right types and right numbers of partners and implementation sites involved who can serve the target population?”

IMPLEMENTATION focuses on the consistency of program delivery no matter how often it is delivered by staff and partners and across settings. This component answers the question, “Can what we see happening in the participants really be attributed to the program – and not something else?” A key question for this component focuses on: “Are the trainings and programs being delivered with fidelity to essential program elements (e.g. specific requirements for number of sessions, length and frequency of sessions, number and type of personnel, use of standardized curricula, etc.)?”

MAINTENANCE focuses on overall impact, that is, sustaining the benefits participants experience AND on sustaining the program over the long-run. Examples of key program questions include: “Are there sufficient staffing/ human resources to sustain the program? Are there adequate financial resources, fees, policies, and/or regulations in place to grow and sustain program delivery and distribution (e.g. grants, rs, insurance reimbursement, etc.)? Are the marketing efforts successfully expanding the number of partners, sites, workforce and participants to bring the program to scale? Is there an adequate pipeline for participant referrals and referral sources (e.g. physician practices, etc.)?”

Methods

Our mid-course assessment was conducted as an eight-step process spanning two years that included the same sequential quantitative and qualitative data collection methods and data analyses for each year. We feature below each step and illustrate how we used the RE-AIM model to help guide the design of our data collection tools.

Step 1: Establish evaluation purpose, team and conceptual approach

Our project team, comprised of an NC DAAS project coordinator, project evaluator and university program and evaluation consultants from three different universities, formulated the operational translation of NC REACH II using the RE-AIM framework. We determined that the purpose of the evaluation was to explore the feasibility of program implementation within the context of community-based service delivery and organizational supports and constraints. We also determined that we would collect evaluation data in a two-stage process starting with an online survey followed by a group discussion with the family consultants.

Step 2: Devise online survey question domains

Guided by the RE-AIM framework, the project team developed an online 30-item self-administered survey that included open- and close-ended items (yes/no, Likert scales). Because we tracked participant characteristics through our enrollment forms, we limited our online survey questions about the REACH component of RE-AIM to issues of recruitment. On a scale from very satisfied to very dissatisfied, we asked the family consultants to rank their satisfaction with recruitment of caregivers into the program, recruitment in rural areas and recruitment within minority communities. We also asked for any recommended changes in recruitment. Separate outcome evaluation forms are used to track changes in caregiver behaviors attitudes and coping, so the focus of the midcourse assessment of EFFECTIVENESS was centered on asking the family consultants to note any unanticipated positive or negative outcomes for the clientele or for themselves, and to rank the value (not at all to extremely) of the REACH II intervention for serving caregivers. For ADOPTION, we asked the family consultants to rank their satisfaction with the pre-intervention meeting, the clarity of the REACH II project goals, the clarity of the REACH II expectations for their roles, the supportiveness of the project partners and the responsiveness of the state-level staff. We also asked for any recommended changes in the pre-intervention meeting content. Because the purpose of the mid-course assessment was to determine the feasibility of program implementation and any needed adaptions, our exploration of the IMPLEMENTATION component was the most extensive. We explored satisfaction with the training workshop all family consultants were required to attend to learn about the NC REACH II program, and any recommended changes in the training format or content; confidence with completing the client individualized intervention plan; satisfaction and challenges with completing the required number of phone calls and visits; satisfaction with tailoring the intervention to meet client needs and writing problem-solving prescriptions; satisfaction with the timeliness and quality of supervision and monthly technical assistance calls and any recommended changes; satisfaction with the program certification requirements; satisfaction and any recommended changes with tools and resources including the intervention forms and the consultant and caregiver notebooks; and any recommended changes in the NC REACH II program protocols and operations. To understand the feasibility of MAINTENANCE of the program beyond the project period, we asked the family consultants whether they would recommend continuing the program, and what strategies related to program operations, costs, and partnerships they would recommend to sustain it.

Step 3: Implement online survey

The online survey was distributed to the family consultants using a Qualtrix platform. The survey took about 20 minutes to complete and respondents could start and stop the survey as often as needed. The family consultants were given two weeks to complete the survey and as the deadline approached, a reminder email was sent to those who had not yet submitted their responses.

Step 4: Summarize survey findings

The evaluation consultant tabulated the frequency, percentage and mean responses for the satisfaction and confidence scales for each of the questions related to the RE-AIM elements. Open-ended responses were also summarized into themes, highlighting direct quotes for illustration. Analyses focused on the identification of program components where there was universal or majority consensus about value and satisfaction and those areas where there was considerable variation. The summarized survey findings were shared with the project team and used as the basis for generating a 15-item moderator’s guide to be used for the group discussion. The moderator guide included five domains related to the RE-AIM components, namely: participant recruitment (REACH), intervention training, number of session that comprise the intervention, intervention components and forms (IMPLEMENTATION), and feasibility of program maintenance (MAINTENANCE).

The purpose of the discussion group was four-fold: to share those ratings that were universal or near-universal, to query about program areas reflecting significant variation in satisfaction level, to collect a deeper understanding of what these findings meant to the family consultants and to explore spontaneous issues that surfaced during the discussion.

Step 5: Conduct group discussion of survey findings

Approximately six weeks after the survey was distributed, a three-hour discussion group with first-year family consultants was conducted by an experienced moderator (first author) paired with a note-taker. The discussion was also audio-recorded.

Step 6: Analyze findings and share results with program advisory board

Our project team shared the results from the online survey and group discussion at a program advisory group meeting that included family consultants, coaches, state-level staff, and the program consultants. At this meeting we explored the “face validity” of the summarized findings and discussed implications for program changes (see the Results section)

Step 7: Putting findings into action – approval of the proposed program adaptations by the program developers and funders

As a key step in the translation/adaptation process, we completed and submitted an “Intervention Modification Request Form” to the funder. We systematically noted each program component that our family consultants recommended be adapted, explained the barriers to implementation that we were addressing and why the proposed modifications were necessary, and explained what changes we would make and the “pro’s” to those adaptations. For all proposed modifications, we verified with one of our program consultants, who was a program developer for REACH II, that we were maintaining fidelity to the intervention. Upon submitting the form, we participated in a conference call with the project officers to discuss each recommended adaptation and to receive approval for making the requested changes.

Step 8: Repeat steps 2–7

In the second year of the mid-course assessment, we added the new group of family consultants to the existing pool of “veteran” family consultants and conducted the same process of inquiry and data collection noted in Steps 2–7. We adapted the prior online survey to account for changes that had been made to the NC REACH II intervention in the prior year. We implemented the survey within the same timeframe and with the same flexibility for completion. We summarized the findings (as an aggregate and also by new and veteran intervention status) looking for areas of consensus and variation in responses, and again used these findings to generate new follow-up questions to be asked in the discussion group. We held separate group discussions with the new and veteran family consultants to maximize sensitivity to the differences in the duration of their intervention experiences and to attain small group size to encourage participation.

Results

A total of 15 REACH II family consultants and coaches from three North Carolina organizations (Department of Social Services, Area Agency on Aging, and an area hospital) participated in the two mid-course assessments. The 15 participants worked in the family consultant role to some capacity for an average of four years.

Collectively, the family consultants expressed satisfaction with program and certification requirements, satisfaction in being able to develop a treatment plan and to tailor the intervention, satisfaction with writing “behavioral prescriptions” (but wishing to de-medicalize the name to “action planning”), and a desire to maintain the program. There was also satisfaction with the technical assistance and coaching, but a preference to conduct such mentoring less frequently than once a month.

However, the family consultants recommended many program adaptations. Chief among these were: re-focusing the pre-intervention training to discuss the family consultant role and flexibilities within the timing and delivery of program content, gearing the intervention training to skill and experience levels of the trainees and using role playing to practice “real life” cases and address problem-solving approaches and dealing with caregiver readiness, streamlining the family consultant and caregiver intervention guides and tools, shifting the mandatory weekly coaching sessions to “as needed”, reducing the number of sessions (12) and length of time (six months) to attract more caregivers and to maximize the caseload of those served, opening program enrollment to related family support programs or caregiver support groups, and tailoring recruitment materials along the continuum of caregiving characteristics (e.g. retired versus working; caring for a relative with mild, moderate or severe dementia).

Both positive and negative unanticipated outcomes were reported. Positive client outcomes focused on the trust and strong relationships built between the caregivers and the family consultants that helped caregivers gain new awareness and skills. However, negative outcomes included perceptions of those caregivers and their families who were “difficult.” This difficulty was not related to the relationship with the consultant, but rather to those caregiving situations challenged by intense family dynamics and interpersonal issues, escalating mental and physical health care needs, and other complex compounding stressors. Unanticipated outcomes for the family consultants’ role pointed to needs for: addressing how to balance participant recruitment with workload management; addressing the time and expense involved in traveling to serve remote clients; providing web-based contact lists and mentoring information; and training “back up staff” on NC REACH II to cover a family consultant’s absence.

Conclusion

Conducting a comprehensive and systematic mid-course assessment, guided by the RE-AIM framework, and utilizing qualitative and quantitative methods provided a rich body of detailed feedback for the feasibility testing, community translation and adoption of an evidence-based caregiver intervention. Our mid-course assessments uncovered a broad range of unanticipated needs for program adaptation toward the goal of external validity within our state’s unique community service delivery context and a “better fit” for the family consultants delivering the intervention. Using the RE-AIM framework to guide development of our data collection tools provided better clarity about the intervention and increased the investment in, and ease of, implementation.

We also found that conducting such an assessment in a staged manner using written surveys followed by group discussion helped us gauge overall program satisfaction while identifying varying concerns and recommendations requiring further elucidation. The group discussion also allowed us to explore additional issues that our administrative team had not anticipated as being salient to the “on the ground” experience of the family consultants.

The staged process also provided two other major benefits. First, it allowed us to assess the changes made with two cohorts of family consultants and within the evolving staffing, program and policy environment (e.g. staff and agency turnover, changes in funder expectations, reduction of state and local funding). It can be tempting to make program adjustments and assume that no more changes are needed, but evolving conditions, such as those we encountered, underscore the need for continuous program evaluation to assure program feasibility. A second key benefit was that making changes to the program as a direct result of the first assessment assured the family consultants that we heard, respected, and followed up with their concerns. This helped to overcome a commonly-held perception by staff that program evaluation is a “gotcha” exercise of criticism, rather than a non-judgmental, inclusive sharing of perspectives about what is working or not. As a result, the family consultants were highly engaged and invested in each mid-course assessment.

While not a limitation of the RE-AIM framework per se, we note that there are potential constraints for resource-strapped agencies to conduct this kind of in-depth mid-course assessment. Our project was supported by grant funds that covered the cost of hiring an outside program evaluation consultant who devoted about 10% time each year to work with the team to establish evaluation aims, design the quantitative survey and group discussion guide, moderate the group discussions, analyze the data, and prepare a result summarizing the recommended program adaptations. Additional expenses included printing of survey materials and reports, as well as travel to the intervention regions to conduct the qualitative data collection. Hence, a precaution to address prior to launching such a comprehensive evaluation is to first determine the availability and adequacy of funding and staff time to assure that the activities can be carried out fully. Despite the fiscal and in-kind costs, we believe the process described demonstrates the value of conducting such assessments using the RE-AIM framework and represents a practical approach to operationalizing translation from clinical trials to community practice.

Footnotes

Acknowledgements

The North Carolina REACH II Translation Project is funded by the Administration on Aging ADSSP Evidence-Based grant (90AE0325). The authors wish to thank Abha Varma, Ph.D. who participated in the tool development, implementation and data analysis of the mid-course assessment. Additionally, they wish to thank Len Erker, Marsha McElroy, Kether Abeles, Sallie Williamson, and Margaret Kelly for their continued leadership and expert guidance as REACH II coaches, and to Wilhelmenia Pledger, Heidi Kimsey, Rosalind Pugh, Halona Locklear, Lavern Weathers, Shenell Henry, Jerry Chabica, Nancy Hogan, Ruth Price, Lynn Hopler, Nicole Jones, Yaisha Robinson, Lynne Hopler, Patricia Mathew, and Cynthia Seeley for their dedication and hard work in delivering the intervention. Lastly, they thank project consultant Lou Burgio for his continued technical assistance and input.

Mary Altpeter is a Research Associate in the Center for Health Promotion and Disease Prevention and Adjunct Assistant Professor in the Department of Health Behavior in the Gillings School of Global Public Health at the University of North Carolina at Chapel Hill. She is also an affiliate of the UNC Institute on Aging and principal investigator of numerous projects related to building infrastructure and testing health promotion programs for older adults.

Lisa P Gwyther is an associate professor in Psychiatry and Behavioral Sciences at Duke's School of Medicine. She is a Senior Fellow of the Duke Center for Aging and directs its Duke Family Support Program, a program funded by the North Carolina Division of Aging and Adult Services to serve as a technical assistance clearinghouse on dementia for the entire state. She also directs education for the Bryan Alzheimer's Disease Research Center at Duke.

Sarah R Kennedy graduated in 2009 from North Carolina State University with her PhD in Lifespan Developmental Psychology. Her primary research foci and interests center around Cognitive Aging with an emphasis on evaluating Everyday Cognition and developing interventions aimed at improving the health and well-being of older adults. She currently works as the Lead Program Evaluator at the North Carolina Department of Health and Human Services in the Division of Aging and Adult Services.

Taryn R Patterson is a doctoral candidate in Lifespan Developmental Psychology at the North Carolina State University. Her interests include cognitive aging and interventions, sustainable translation of evidence-based programs to the community, and public policy. Taryn currently assists with program translation and evaluation for the North Carolina Department of Health and Human Services, Division of Aging and Adult Services.

Karisa Derence earned her Master's degree in Gerontology from Appalachian State University and has over 22 years of experience working with individuals and families affected by Alzheimer's disease and other dementias. She served as the State Alzheimer's Support Specialist and State Project Director for NC REACH with the Division of Aging and Adult Services (2001–2011) and now works as a Program Consultant and Planner for the NC Division of Public Health Preparedness and Response Branch.

References

Belle

S. H.

Burgio

Burns

Coon

Czaja

S. J.

Gallagher-Thompson

Lee

C. C.

(2006) Enhancing the quality of life of dementia caregivers from different ethnic and racial groups: a randomized, controlled trial. Annals of Internal Medicine 145: 727–738.

Burgio

Stevens

Guy

Roth

D. L.

Haley

W. E.

(2003) Impact of two psychosocial interventions on white and African American family caregivers of individuals with dementia. The Gerontologist 43: 568–579.

Ducharme

Levesque

Giroux

Lachance

(2005) Follow-up of an intervention program for caregivers of a relative with dementia living in a long-term care setting: are there any persistent and delayed effects? Aging & Mental Health 9: 461–469.

Glasgow

R. E.

Vogt

T. M.

Boles

S. M.

(1999) Evaluating the public health impact of health promotion interventions: the RE-AIM framework. American Journal of Public Health 89: 1322–1327.

Glasgow

R. E.

Whitlock

E. P.

Eakin

E. G.

Lichtenstein

(2000) A brief smoking cessation intervention for women in low-income planned parenthood clinics. American Journal of Public Health 90: 786–789.

Green

L. W.

Glasgow

R. E.

(2006) Evaluating the relevance, generalization, and applicability of research: issues in external validation and translation methodology. Evaluation & the Health Professions 29: 126–153.

Kelly

C. M.

Williams

I. C.

(2007) Providing dementia-specific services to family caregivers: North Carolina's project C.A.R.E. program. Journal of Applied Gerontology 26: 399–412.

Klesges

L. M.

Estabrooks

P. A.

Dzewaltowski

D. A.

Bull

S. S.

Glasgow

R. E.

(2005) Beginning with the application in mind: designing and planning health behavior change interventions to enhance dissemination. Annals of Behavioral Medicine 29: 66–75.

Mittelman

M. S.

Haley

W. E.

Clay

O. J.

Roth

D. L.

(2006) Improving caregiver well-being delays nursing home placement of patients with Alzheimer Disease. Neurology 67: 1592–1599.

10.

Roberts

Brown

(2007) Specialized continuing care models for persons with dementia: a systematic review of the research literature. Canadian Journal of Aging 19: 106–126.

11.

Schulz

(2000) Handbook on dementia caregiving, New York: Springer Publishing Company.

12.

Schulz

Martire

L. M.

Klinger

J. N.

(2005) Evidence-based caregiver interventions in geriatric psychiatry. Psychiatric Clinics of North America 28: 1007–1038.