Abstract
Caregivers play an important role in the in-home care of community dwelling older adults living with Alzheimer’s disease or related dementias (ADRD); however, many of these caregivers lack training in caring for this vulnerable population. In 2015, we developed and implemented an interactive, community-based, knowledge and skills-based training program for In-Home Supportive Services (IHSS) caregivers. This report shares the results of a process evaluation of this training program as it evolved over the course of three training sessions in Riverside County, California. Our iterative evaluation process reveals the unique needs of training and assessing a population of demographically diverse adult learners and provides guidance for those planning to implement similar training in underserved communities. Factors such as reliance on self-reported abilities, language readability level, and test anxiety may have confounded attempts to capture learner feedback and actual knowledge gains from our caregiver training program.
Introduction
California has an ethnically diverse and rapidly growing aging population, many of whom require long-term in-home care services to be able to remain in the community. In-Home Supportive Services (IHSS) is a personal care program in California that provides eligible low-income older adults, blind, and disabled people to remain in their own homes by providing assistance with activities of daily living, accompaniment to medical appointments, and protective supervision for recipients with a cognitive impairment. This program was developed so that recipients may have an increased quality of life by staying in their homes and avoiding or delaying admission to hospitals or other facilities. County social workers determine eligibility for IHSS services through an assessment of recipient’s specific needs. This process determines the number of caretaking hours provided. IHSS caregivers, who are selected and hired by the recipients, are paid with a combination of county, state, and federal funds. Unlike Certified Nursing Assistants or some Home Health Aides who have mandatory training requirements, IHSS caregivers in California do not have to meet standard training requirements, other than to pass background checks. IHSS workers are compensated the same hourly wage despite prior training, qualifications, or complexity in the types of services they are providing, and most work part-time (California’s Direct-Care Workforce, 2010; Taylor, 2010). IHSS caregivers are made up of both family and nonfamily members, and have increasingly provided care to older persons with complex medical and psychosocial needs (Darragh, Sommerich, & Lavender, 2015).
As of July 2010, 42% of IHSS recipients in California are older adults (Cowles & Logsdon, 2012). However, many in-home caregivers receive little to no training in competent and compassionate care of older adults (National Alliance for Caregiving and American Association of Retired Persons [AARP], 2004). Older adults with Alzheimer’s disease and related dementias (ADRD) demand a considerable amount and complexity of care and supervision. In the case for older adults with ADRD, symptoms and behaviors are often disease-specific and require knowledge of disease progression throughout the caregiving experience. Despite the lack of trainings available, studies have shown that training opportunities for these caregivers has been associated with better care, stronger health outcomes, and lower medical costs (“Care Team Integration and Training of Home Care Workers–Impact Study,” 2016).
As part of a larger initiative to enhance the care of community dwelling older adults living in medically underserved Riverside County, California, we designed and implemented an interactive knowledge and skills-based training program for IHSS caregivers who take care of recipients with ADRD. This article shares the results of our Rapid Cycle Quality Improvement (RCQI) efforts using the Plan Do Study Act (PDSA) model, employed to ensure the quality and appropriateness of the training, as it evolved over the course of three training sessions. Our evaluation process offers insight into developing and implementing an IHSS Caregiver Dementia Training in a large, diverse, and medically underserved community.
An Aging Population
Riverside County is one of the largest counties in Southern California. Roughly the size of New Jersey in total area, it has a population of approximately 3 million people, making it the fourth largest county in California and the 11th largest county in the nation, with more residents than 15 U.S. states. In California, the total population of older adults is expected to double in the next 20 years, and elders of color are projected to become a majority of the older population during the same timeframe (“2014 California Department of Aging Demographic Projections by County and Planning Service Area [PSA] For Interstate Funding Formula,” 2014). In Riverside County, the aging population is growing at a rate that outpaces that of the average rate for California, with a 69% growth rate in 2010. The Health Resources and Services Administration (HRSA) designates 50 out of 58 service areas in Riverside County as a Medically Underserved Area/Population (MUA/P). Geographic areas that meet the MUA/P criteria are defined as areas having too few primary care providers, high infant mortality, high poverty, or a high elderly population. Providing targeted initiatives to improve the care of older adults is important to counteract the health status disparities that exist between non-Latino Whites and different groups of minority older adults for a number of acute and chronic conditions, disabilities, and mortalities (Pollard & Scommegna, 2013).
IHSS in Riverside
According to Riverside County Public Authority (PA) records, there are approximately 32,000 IHSS recipients in Riverside County. To qualify as an IHSS caregiver, the Riverside County PA currently requires caregivers to attend a new provider orientation, which provides information about the IHSS program, basic home safety guidelines, and training on how to complete a timesheet. There is an average of 450 new IHSS caregivers to the PA per month. Although many of these caregivers provide care for older adults with ADRD, there is no specific training available for IHSS caregivers in Riverside County on the disease, its progression, and methods of caring for these individuals. The University of California, Los Angeles (UCLA) Geriatric Workforce Enhancement Program (GWEP) is a federally funded program through the HRSA that aims to improve health outcomes of older adults through workforce education and training. To address this identified caregiver training disparity, the UCLA GWEP partnered with the Riverside PA to develop and implement a training for IHSS caregivers on providing competent care for older adults with ADRD. The training also sought to address caregiver burnout and retention, given the adverse impacts that caregiving has on the general health and mental health of caregivers.
Program Description
We developed and implemented an interactive, community-based, knowledge- and skills-based training program for IHSS caregivers (family and nonfamily) serving older adults living with dementia in Riverside County. The curriculum content was based on self-perceived educational needs and trainings gaps identified through a need assessment survey administered to IHSS workers in Riverside County. Each in-person training consisted of a 2-hr interactive, educational program. The courses covered three broad topics essential to the competent care of a person with dementia: (a) the basics of dementia and managing behavioral issues, (b) promoting home safety and preventing falls, and (c) managing caregiver stress. Academic geriatricians, practicing social workers, and educators with expertise in Alzheimer’s disease and dementia caregiving jointly developed the curriculum. We administered the training program three times over the course of 1 year.
This process evaluation focuses specifically on the IHSS Caregiver Dementia Training, which consists of two modules that provide basic knowledge and skills-based video simulations to enhance learning. The first module consists of an introduction to dementia, including expected disease progression, recognizing characteristics/challenges for each stage of dementia, and distinguishing between delirium and dementia. The second module consists of a series of UCLA Health caregiver training videos on the management of common and challenging behaviors of persons with dementia, including (a) refusal to take medications, (b) repetitive behaviors, (c) sundowning, and (d) wandering. At the end of each video, we provide recommended approaches to each difficult behavior.
Using a train-the-trainer model, 15 representatives from partner organizations in Riverside County (such as Department of Public Social Services [DPSS] social workers, the PA, and the Inland Empire Health Plan) were trained on implementation of the curriculum, and served as “master trainers” in the delivery of the workshops. These “master trainers” consisted of social services supervisors for the PA and care managers from the local health plan. Thus, a UCLA geriatrician (H.B.S.) and a local “master trainer” copresented each of the three IHSS Caregiver Dementia Trainings in Riverside over a 9-month period. Each training session was 2 hr in length and performed at three sites (Hemet, Desert Hot Springs, and Moreno Valley) within Riverside County to maximize trainee participation.
Program Evaluation Method
As part of our RCQI efforts, we implemented a series of PDSA cycles (Figure 1) to ensure optimal design and delivery of the program in real time (“RCQI Resource Guide,” 2017). The PDSA framework consists of four phases: (a) establish a plan based on desired improvement goals or objectives; (b) implement, or do, the plan, all the while collecting data; (c) study the results over a brief period of time; and (d) act based on this information, to sustain improvements. This cycle is then repeated to continuously monitor the improvement of the product or process in question (Deming, 2000). As part of this process, we relied on program logistics analysis, community-organization feedback, leadership feedback, and participant evaluations and satisfaction to inform changes. Our process for evaluating the implementation of the program can serve as an illustrative example for other institutions desiring to develop such a training program, as the objectives of the process evaluation included (a) assessing participant satisfaction and (b) documenting participant knowledge and skill gains. We reveal how to develop and implement this program in other communities by describing our evaluation process for the implementation of our knowledge and skills-based IHSS Caregiver Dementia Training in a large, diverse, and medically underserved community in detail.
GWEP riverside county IHSS training PDSA cycle.
Participants
Participants consisted of IHSS caregivers in Riverside County caring for older adults. Given the large distances between cities in Riverside County, the PA analyzed its IHSS database to selectively invite IHSS caregivers who provided care for older adult clients who were residing within the geographic area where each training was held. The PA recruited both family and nonfamily IHSS caregivers for the trainings. Of the 152 IHSS caregivers invited, 106 (69%) ultimately participated in a training session.
Data Collection
Our first training program provided baseline information and metrics, which we used to inform the first PDSA cycle. Iterative changes made to the program in each stage of the PDSA cycle became the new standard and were used to inform subsequent PDSA cycles.
We collected satisfaction data from all participants, including the instructors, organizational leads from our community partners, and IHSS caregivers who attended the training. We assessed participant knowledge prior to the training and at the conclusion of each training using a true pre–post-test study design. IHSS caregivers also completed an evaluation of the program at the end of each session. We designed the postretrospective, pretest style evaluation to capture caregivers’ self-reported confidence in practicing the skills covered as part of the training and their overall satisfaction with the training. The participants were asked to rate their ability to care for persons with dementia “after the training” and “before the training.” Following each training, we held a debriefing session with our community partners regarding the implementation of the program and conducted semistructured interviews with our master trainers to obtain feedback to inform the next iteration of the training. This study was approved as exempt by the UCLA institutional review board (IRB; 16-000337).
Data Analysis
Analysis of the collected outcome data was directly informed by the PDSA cycle. We used feedback from our master trainers and community partners, and results from the IHSS caregiver evaluations to inform our planning process for the subsequent implementation of the training. We used descriptive statistics and paired samples t-tests to analyze data from our postretrospective–pretests. Results from these analyses were used to adjust the program prior to the subsequent training session to improve knowledge gains and satisfaction with each cycle.
Results
Slightly more than half of the participants served as paid caregivers for nonfamily members (52 of 102, 51%). More than a quarter of participants received funds to care for family members (28 of 102, 27%). A majority of caregivers were age 50 or older (59 of 102, 58%), and nearly three quarters (73 of 102, 72%) self-identified as a person of color. Nearly half (49 of 102, 48%) had a high school education level. Many participants reported to have served as a caregiver for 6 to 10 years (31 of 102, 31%; Table 1). Nearly half of the participants (51 of 102, 48%) stated having received some training in dementia prior to attending our training, but only one third (34 of 102, 32%) said they had received training on caring for clients with difficult behaviors related to dementia or Alzheimer’s disease (Table 2).
Caregiver Demographics and Characteristics (N = 102).
Caregiver Training Characteristics (N = 101).
Tables 3 to 5 demonstrate satisfaction and knowledge attainment data by training session to delineate results from each training session individually, as changes made to a subsequent training were informed by results of the preceding training. Across training sessions, participants were highly satisfied, as indicated by above average to excellent rating for instructors (including master trainers), materials, and organization. A majority of participants (77 of 96, 80%) found all of the information learned helpful to their work as caregivers (Table 3). All participants (98 of 98, 100%), regardless of training session, reported that they would recommend the training to other caregivers.
IHSS Caregiver Perceived Usefulness of Training by Session (n = 96).
Note. IHSS = In-Home Supportive Services.
Bold Values: The bold values represent the majority for each row.
Comparison of Caregiver Self-Identified Abilities Before and After Training by Session.
Note. Responses coded as follows: 1 = very poor; 2 = below average; 3 = average; 4 = above average; 5 = excellent.
p < .0125, p value adjusted using the Bonferonni correction.
Knowledge Gains by Sessions.
ARS precluded our ability to calculate an average participant score and, thereby, perform pre–post comparisons, as data were collected anonymously by item. Results are based on averages over items, not individual responses. ARS = audience response system.
p < .008, p value adjusted using the Bonferonni correction.
To capture potential change in perceived skills, we asked participants after the training to rate their abilities as caregivers after the training and prior to the training. A majority of participants, regardless of training session, reported to have excellent abilities (66 of 95, 70%) after the training (Table 4). A series of paired samples t tests confirmed that these self-reported positive changes in perceived ability to care for a person with dementia were statistically significant, t(94) = 11.47, p < .001. Therefore, adjustments to the content changes made through the PDSA cycles did not affect average participant perceptions of the quality of the training.
Analysis of knowledge scores from our pre–post tests following the first training session revealed nonsignificant participant changes in average knowledge for both Module 1, t(16) = 1.319, p = .206, and Module 2, t(16) = 1.514, p = .150 (Table 5). These results prompted us to revise our knowledge assessments for the subsequent IHSS Caregiver Dementia Training sessions. Changes included simplifying the language level and instructions of the pre- and posttests, reducing the number of answer categories for each multiple-choice item, and minimizing test-anxiety by eliminating language like “test” from the document, and including the response option “I don’t know” for some questions. No changes were made in the content of the questions or the content of the two modules.
Findings from our second training session revealed that with changes to the language of the assessment measures (but no change to the content), the participants demonstrated gains in knowledge (Table 5). However, these knowledge gains were not substantial, particularly for the first module, t(39) = 2.216, p = .003, which provides an introduction to dementia, including signs, symptoms, and stages of the disease. Based on these findings and our observations, we elected to modify how we administered our knowledge assessment in the third session.
For our third training session, we implemented an electronic audience response system (ARS) and interspersed the pretest questions throughout the training, rather than as a true pretest prior to the start of the training. Prior to introducing a particular concept, the instructor would pause to “check everyone’s understanding” using the ARS. The question and answer choices appeared on the screen, and the instructor read the question and answers aloud to the group. This dramatically reduced the amount of time for participants to complete the pretest, while permitting the instructor to gauge participants’ understanding. In addition, the ARS permitted participants to select only one answer option for each item. Participants completed a posttest following the completion of each module. It is important to note that we could not calculate an average participant score for the third training session, as pretest data were collected anonymously per item due to the use of the ARS. Results indicated, however, that average score by item increased at the end of the training program in both modules (see Table 5).
Discussion
Given the growing number of community dwelling older adults living with ADRD in need of complex care at home, there have been efforts toward recognition of caregiving as a profession. Like other health professions, this will require organization of formal training, credentialing, and certification process. Currently, IHSS caregivers in Riverside County do not receive training in dementia, or in any other specific disease or condition prior to providing caregiving services. The GWEP IHSS Dementia Caregiver Training was developed to enhance dementia caregivers’ knowledge of the common signs and symptoms of dementia and to provide them with skills to manage challenging dementia-related behaviors.
Participants scored well on the knowledge pretest in both modules across all three trainings, limiting our ability to capture changes in knowledge. Possible explanations for this include participants gaining knowledge of ADRD outside of formal training, (e.g., learning from past caregiving experiences, reading books/articles). Despite evidence in the literature that caregivers in general lack formal training on how to care for persons with dementia, and results from our needs assessment that indicated a desire among Riverside County IHSS workers for dementia caregiver training, almost half of the participants across all training sessions stated that they had previously received some form of caregiver training. One third stated they had received previous training on managing challenging dementia-related behaviors. Thus, it is possible that the recruitment methods resulted in selection for more knowledgeable and motivated caregivers to attend. In addition, given the number of participants with experience working as a care provider for 10 years or more, it is possible that they compensated for lack of formal training with personal experience working with older adult clients. Given that the current IHSS system does not take into account previous training or experience in determining compensation, there is limited extrinsic motivation for caregivers to receive additional or formal training (Barnes, Sutherland, & Logsdon, 2005). Our participants may represent a subgroup of the IHSS caregiver population who are self-motivated learners or those compelled to seek out training due to a direct need (i.e., currently working with a challenging client or a family member diagnosed with dementia). Whatever the cause, better than anticipated pretest scores highlight the importance of considering not only community-specific IHSS caregiver training needs but also specific targeting of caregivers who may be less likely to attend training programs due to internal or external barriers.
Our iterative evaluation process reveals the unique challenges of assessing understanding within the Riverside County IHSS caregiver population, which comprises largely adult learners who are members of an ethnic minority group and who have several years of caregiving experience. Factors such as reliance on self-reported abilities, language readability level, and test anxiety can confound attempts to capture learner feedback and knowledge gains from the training. In response to these findings, we modified our evaluation measures to provide a more accurate representation of participant understanding of care for persons with dementia.
Findings from our first training indicated no substantial change in participant knowledge, despite positive learner reviews of the sessions. Following the first training, we decided to focus on improving the language of our evaluation measures and eliminating the use of phrases that may cause learner confusion or provoke test anxiety. Many caregivers found the language used in the assessment challenging, as demonstrated by requests for facilitators to read questions aloud and to clarify or define terms during its administration. During our first and second trainings, we observed that participants took considerably longer than expected to complete the assessment, given the length of the measure. We allotted 10 min for this process, but it took more than 20 min to be completed. Many participants did not follow instructions, selecting several response options for each item. Some participants skipped questions or returned blank assessments altogether. Despite our best efforts to design a brief assessment to measure understanding of basic dementia concepts and caring for those with dementia, these observations indicated to us that the assessment itself triggered confusion and/or anxiety among participants. Following this experience, we became more mindful of the questions’ readability levels. Our team reviewed the assessment and made changes to the language.
We also became hyperaware of our references to the assessment as a “test” (originally, we titled the assessment a “pre-” and “post-” test). We worked diligently to remove any reference to a “test,” including adding an option for “I don’t know” to legitimize such a response. In being more sensitive and attentive to our population, we attempted to simplify instructions and reduce any potential text anxiety among participants. By appreciating the unique experiences of our participants, we provided an assessment that more appropriately captured their level of understanding.
By implementing an ARS in our third training, the pretest questions were read aloud and displayed on a PowerPoint slide. Rather than administering a traditional paper pretest, we interspersed these questions throughout the training to precede the content that was to be covered. We informed participants that the questions were being administered to gauge learner understanding and to see what they already knew. Instructors told participants that they would be learning about the topic as a group. The system also allowed for individual respondent anonymity, thereby minimizing potential anxiety around testing. At the end, the instructor presented the same questions one after the other. The UCLA instructor reviewed the correct answers and explained why the incorrect responses were not the best answer (incorrect). Although this approach may have decreased test anxiety among participants, one downside was that the caregivers were not held individually accountable for answering each question (as there was no penalty for not participating). Thus, the total number of responses fluctuated by each question, limiting our interpretation of evaluation results. However, this strategy more importantly validated the needs and experiences of our IHSS caregivers, making the assessment component a more natural and less imposing piece of the training. By employing strategies such as improving readability, minimizing testing language, using ARS technology, and reading the questions aloud to the group, we have consciously adapted our assessment to meet the needs of our caregiver population.
Demonstrating knowledge gains is a widely accepted strategy for demonstrating effectiveness of a training program. Commonly used knowledge assessments, however, may obscure actual understanding among caregivers who are older adult learners, have limited formal education, and are from minority communities. This is an important consideration, especially with the push in recent years to train, certify, or professionalize, in-home caregivers (Pourat, 2013). Implementing a policy based on traditional knowledge acquisition could unfairly disadvantage caregivers from diverse backgrounds, thereby compromising the caregiving workforce in certain underserved areas such as Riverside County, California. Ideally, caregivers participating in such training can demonstrate actual skills when caring for recipients with dementia. Skill assessment, however, is costly and time consuming, which is of concern when attempting to train a large population of caregivers, as in the case of the Riverside County PA. Evaluations of caregiver knowledge must carefully examine the English reading level, readability, and the testing language of the assessment. When participants demonstrate little to no knowledge gains posttraining, evaluators must consider whether the format of the assessment itself precluded the collection of accurate performance scores. We found that caregivers showed a higher level of participation with the second module. This may be explained by the content of the videos, as caregivers most likely felt a stronger, personal connection to the scenes presented. We noticed that participants often engaged in a collaborative discussion of their personal coping methods, which can be an additional form of learning that we did not quantify.
Limitations
The process evaluation of the IHSS Caregiver Dementia Training is limited in that we have no comparison group for participants. The results cannot be examined against the gains that nontrained caregivers might achieve through work and life experience. It is unclear how years of experience working with multiple clients and/or working with clients through the various stages of dementia may influence knowledge. Given the issues that we encountered with using an academic approach to knowledge assessment, and the limited information received on participants’ background, we cannot account completely for knowledge or training obtained prior to attending our trainings. In addition, our training sessions were all conducted in English, and participants who attended were aware of this. Yet we discovered during the training that many participants were adults for whom English was their second language. Thus, the possible role of limited English proficiency in test performance needs to be considered when analyzing our results, as we did not have a comparable group receiving the training in their native language. Last, we did not conduct “in-home” assessments of skills/knowledge gained to truly assess the impact of the training on caregiving competency.
Conclusion
We aimed to improve the in-home care of clients with ADRD who receive in-home care by developing and implementing an interactive, community-based, knowledge- and skills-based training program for IHSS caregivers. Our iterative evaluation process reveals the unique needs of training and challenges to assessing knowledge gains in this population. The IHSS caregivers in our program comprised largely minority adult learners who have several years of caregiving experience, presenting challenges for measuring the extent of the training’s impact. Further research is needed to understand how to best assess learning and evaluate program outcomes for this particular population, especially prior to making any attempt to professionalize the caregiver workforce.
Footnotes
Acknowledgements
We would like to acknowledge the administrative and research support of Maribel Garcia at the University of California, Los Angeles (UCLA) Geriatric Workforce Enhancement Program (GWEP). We also want to acknowledge Jewel Lee, Veronica Hilton, and Laura Gonzalez-Rivera from the Riverside County Department of Social Services, Public Authority for being our partners and collaborators on this project.
Declaration of Conflicting Interests
The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This study was funded by Health Resources and Services Administration (HRSA), Geriatrics Workforce Enhancement Program (U1QHP28725, Tan).
