Cognitive Predictors of Life Skill Intervention Outcomes for Adults With Mental Illness at Risk for Homelessness

Study Design

This study used a longitudinal design with variable exposure to a life skills intervention consisting of four modules: (1) room and self-care management, (2) food management, (3) money management, and (4) safe community participation. Participants completed an initial ACLS–2000 and a Practical Skills Test (PST) and participated in 12 group and individual sessions per module with an occupational therapist. At the completion of these modules and at 3 and 6 mo after the intervention, assessments were repeated to assess life skills knowledge.

Sample

Participants were recruited from either an emergency housing program (EHP) or a single-room occupancy (SRO) program located in a Midwestern city by means of posted flyers and attendance by the occupational therapist at program meetings. Inclusion criteria were intentionally broad to understand the effectiveness of providing services to this population. Participants needed to be in one of the two programs and have a documented mental illness. They also had to identify a life skill need, be able to engage in a 60-min group and individual session each week, and give informed consent. Interested participants were screened to verify their competence to give consent using the MacArthur Competence Assessment Tool for Clinical Research (Appelbaum & Grisso, 2001), which has excellent interrater reliability and construct validity (Dunn, Nowrangi, Palmer, Jeste, & Saks, 2006). Exclusion criteria were inability to understand English or tolerate a 60-min group. Participants received $10 grocery store gift certificates after each study phase. The institutional review boards of all involved institutions approved all aspects of the study.

Participants enrolled in at least one of the four life skills intervention modules (Helfrich, 2006a, 2006b, 2006c, 2006d). Most participants chose to complete all 4 modules. EHP participants (n = 21) completed 2 sessions/wk for 3 wk, and SRO participants (n = 17) completed 6 weekly sessions to accommodate variance in program demands for each group. Modules were repeated sequentially, but participants enrolled in each module only once.

Program Implementation

All study participants completed an initial comprehensive evaluation that included the ACLS–2000 (Allen Conferences, 2000) and 4 PSTs (Helfrich, 2006a, 2006b, 2006c, 2006d) to identify the participant’s current life skills in room and self-care management, food management, money management, and safe community participation. A demographic interview was also administered. Medical and psychiatric diagnoses were obtained from client records. The assessment results identified relevant life skills areas for each participant. Following empowerment theory (Zimmerman, 2000), participants chose the modules they participated in rather than being assigned to them by the occupational therapist.

Group sessions followed a situated learning approach (Lave & Wenger, 1991) using psychoeducational materials and applied activities. Modules began with a topic overview and discussion of how mental illness symptoms interfered with their skills. Topics for the room and self-care module included personal and public hygiene, health, clothing, home cleaning, and home organization. The food management module covered nutrition, food safety, budgeting, obtaining food, microwave cooking, and meal preparation. The money management module included material on budgeting, shopping, advertising schemes and financial protection, banking, credit, and tools for money management. The safe community participation module included community, public, and home safety; health protection; physical safety and protection; and consumer and personal advocacy. After each group session, participants completed a quiz to assess their knowledge and provide feedback.

Participants chose individual session topics to allow for individualized application of the group content. The activities were adapted specifically to each participant’s functional capabilities by considering the activity’s pace and complexity and the participant’s score on the ACLS–2000. Individual sessions were held at a variety of locations, including the program site, the person’s room, or the community for tasks such as learning to use a food pantry, accessing public transportation, accessing banking services, or practicing community safety. After each module, a written practical skills test measuring attained knowledge was administered.

Data Collection Tools

The ACLS–2000 (Allen Conferences, 2000) is used to evaluate functional cognition and the level of assistance needed to safely complete tasks for daily living in the community. It uses a leather-lacing activity to evaluate a person’s ability to learn a new task, identify and correct errors, and problem solve. The final score is used to interpret the individual’s ability to process and use information and capacity to learn or relearn skills and to predict how much assistance the person may need when performing daily activities in both familiar and novel situations (Allen et al., 2007). Although the ACLS–2000 is used widely in mental health practice, the research literature is sparse and does not include people at risk for homelessness.

Velligan, True, Lefton, Moore, and Flores (1995) reported that the ACLS–90’s interrater reliability was high (intraclass correlation = .85) and that it provided a valid and culturally unbiased cognitive functioning measure. Various versions of the ACLS have demonstrated high interrater reliability (rs =.98–.99; Henry, Moore, Quinlivan, & Triggs, 1998). Quantitative studies have used correlations to establish the strength of the association between the Allen Battery (Allen Conferences, 2002) and the ACLS–90 (Allen & Blue, 1998). High interrater reliability was found (r = .99, N = 32, range of levels = 2–6). Although the psychometrics of the ACLS–2000 have not been published, those of the ACLS–90 are strong; because the differences in the two versions are minor (instead of stopping after the first failed stitch in ACLS–90, people are encouraged to try the next stitch in the ACLS–2000), it is likely that the psychometrics of the ACLS–2000 would be similar.

The PSTs were developed by Christine A. Helfrich with input from people who had been homeless and made the transition to living independently in the community (Helfrich & Fogg, 2007). Each test is a 30-question paper-and-pencil assessment of the knowledge people acquired and retained from each module. Participants applied the material they learned through multiple choice, true–false, and short-answer questions. For this study, the PSTs’ reliability, as measured by Cronbach’s α, was good for money management (α = .801) and safe community participation (α = .815), acceptable for food management (α = .665) and self-care (α = .656), and adequate for room care (α = .540). Although room and self-care encompass one module, the PST for this module was split into two sections for content analysis because of the dichotomous information included.

Data Analyses

We examined the data for assumptions of normality and to ensure no outlying values. When sample sizes were discrepant, equal variances were not assumed in analysis.

Data analysis was conducted in 4 phases. First, we used the ACLS–2000 as a scale score in correlation analyses to examine whether the ACLS–2000 score was significantly associated with PST scores at any of the assessment points. We also used the ACLS–2000 score in a series of multiple regressions performed to identify key predictors of PST scores at each time point. Type II partial R ²s were used and are presented to describe the strength of individual predictors. In addition to the ACLS–2000 score, we included other person variables, such as education level, diagnosis, and presence of a dual diagnosis in the model, as predictors of PST score on the basis of their potential impact on cognitive level and performance. Substance use and psychotic features may impair one’s cognitive functioning and judgment and in turn have a negative impact on the performance of life skills and housing stability, whereas higher education level and ACLS–2000 scores may predict higher functioning on knowledge-based testing. We hypothesized that changes in PST scores over time might vary depending on a participant’s ACLS–2000 score. Therefore, we performed separate multiple regressions at each time point.

We examined PST scores over time (baseline, postintervention, 3 mo, and 6 mo) for the sample through linear mixed-effects regression models for longitudinal data (Verbeke & Molenberghs, 2000). The advantages of these methods include the ability to include data from participants with partial follow-up, to control for time-varying covariates, and to better model the correlational structure of longitudinal data.

For subsequent linear mixed-effects regression models, we categorized participants into two groups on the basis of ACLS–2000 score at each time point (The ACLS–2000 measures changes in increments of 0.2.) Participants scoring in the high-ACLS–2000 group demonstrated scores of ≥4.8 at the time of assessment. Participants with scores of ≤4.6 were in the low ACLS category, according to the theory that scores of ≥4.8 are associated with the capacity for new learning (Allen, 1992). Additional analyses examined individual associations between ACLS–2000 groups and PST scores and whether changes in PST scores over time varied by ACLS–2000 group; in those analyses, we used linear mixed-effects regression models with indicator variables representing time, the ACLS–2000 group, and the interaction between the two terms. When a significant interaction was found, indicating differential change in PST scores over time for different ACLS–2000 groups, we repeated the mixed-model regression of changes over time separately for each group and conducted t tests to compare group differences in PST scores at each time point.

Finally, we used effect sizes to describe the magnitude of change in PST scores over time. Effect sizes were calculated by dividing the mean change from baseline to follow-up and then dividing by the standard deviation of the change scores. We used Cohen’s (1988) criteria for small (d = 0.2), medium (d = 0.5), and large (d = 0.8) effects. Differences in effect of the intervention between high and low ACLS–2000 groups are presented descriptively. Some results were suggestive and are discussed to describe the groups, but because of the low ACLS–2000 group’s small sample size, no formal analysis for significant differences was completed to compare groups by effect size.

Participants

We recruited 59 participants for Years 2 and 3 of the intervention study. We report only data from Years 2 and 3 here because changes were made to the PST’s test format after Year 1, and the results were not comparable (they are reported elsewhere; see Helfrich & Fogg, 2007). Of the 59 participants recruited, 8 were assigned to the control group (omitted from analysis because of small sample size), 1 withdrew from the study before baseline testing, and 7 completed the baseline assessment but did not participate in the intervention, representing attrition in the study. Data from 5 additional participants were not included in the analysis as a result of incomplete ACLS–2000 or PST score data at baseline or after the intervention. The final sample consisted of 38 participants. Sample sizes reported for each module at baseline in Table 1 differ from the reported total because the intervention modules were delivered sequentially and because not all participants chose to complete all four modules. In addition, because of the nature of longitudinal data, the number of participants varies from baseline when assessment data are missing for participants at some of the follow-up points. The 38 participants who completed the intervention included 21 from the EHP (55%) and 17 residing in an SRO (45%). We found no significant differences between participants on the basis of housing site at baseline.

Participants were 14 women (37%) and 24 men (63%) ranging in age from 24 to 61 (mean = 46). The racial demographics included White (47%), African-American (39%), Hispanic (2%), multiracial (5%), and other (7%). Nearly 87% had at least a high school education. Participant employment status was unemployed (95%), part time (2.5%), and full time (2.5%). Past and current work occupations varied widely. Most participants had worked in at least three different occupations.

ACLS–2000 and PST Scores

ACLS–2000 scores for participants at baseline ranged from 4.0 to 5.8 (mean = 5.1). The mean scores for each of the PSTs over time are reported in Table 1.

ACLS–2000 and PST Correlations

Correlational analyses reveal a significant association between ACLS–2000 score and performance on the PST, as shown in Table 2. For food management and safe community participation, we found significant moderate positive associations between ACLS–2000 and PST scores at baseline and postintervention with a strong positive association at the 6-mo follow-up. Money management scores showed a similar pattern, with moderate positive associations at baseline and strong associations postintervention and at the 6-mo follow-up. We found no significant correlations at the 3-mo follow-up for these modules. By contrast, ACLS–2000 scores had a strong positive association with self-care PST scores and a moderate positive relationship with room care PST scores at the 3-mo follow-up. ACLS–2000 scores had a moderate positive association with room care scores at baseline and a strong positive association with self-care scores at the 6-mo follow-up.

Regression Analyses

We completed multiple linear regression analyses to identify significant predictors of each of the PST scores for each time point. Predictors included ACLS–2000 score and other factors that could affect cognitive functioning and test performance, such as education level (coded as less than high school, high school or equivalent, or at least some college), diagnosis (coded as affective disorder or psychotic disorder), and presence of a dual diagnosis (yes or no). All results reported were significant when holding all other factors in the model constant.

For the food management module, education level was a significant predictor (F[4, 18] = 3.79, p = .021) of the PST score at baseline, controlling for ACLS–2000 score, diagnosis, and dual diagnosis. A higher level of education predicted higher PST scores at baseline, and the partial R ² indicates that education level explained 20.3% of the variability in score at this time point. We found no other significant predictors for food management PST scores at any other time point.

For money management, linear regression analysis showed significant evidence that education level was the primary predictor of PST score at completion of the intervention (F[4, 8] = 6.80, p = .011) and at the 3-mo follow-up (F[4, 6] = 5.71, p = .030), accounting for 24.6% and 49.1%, respectively, of the variability in scores at this time point. At the 6-mo follow-up, regression results indicated significant evidence (F[4, 7] = 9.14, p = .007) that higher ACLS–2000 score, not education, was a significant predictor of higher money management PST scores, accounting for 39.9% of scoring variability.

Education and ACLS–2000 scores were significant predictors of room care PST scores at the 3-mo follow-up, accounting for 23.4% and 48.6% of the variance, respectively (F [4, 5] = 6.98, p = .028). We found no significant predictors of safe community participation or self-care PST scores at any point, based on the cognitive variables included in the linear regression analysis.

PST Changes Over Time

Table 1 shows that people with higher ACLS–2000 scores had higher mean PST scores over time than those in the lower scoring ACLS–2000 group for all life skill modules. The differences in mean PST scores between groups were significant for all modules over time on the basis of a mixed-model regression analysis: food management (p = .009), money management (p = .002), safe community participation (p < .001), room care (p = .002), and self-care (p = .015).

We also completed independent samples t tests to determine whether significant differences existed between mean PST scores at each time point. Results in Table 1 show that participants with higher ACLS–2000 scores had significantly higher food management and safe community participation PST scores at baseline (p = .001 for both), intervention completion (p = .011 for food management and .007 for safe community participation), and 6-mo follow-up (p = .002 for food management and .008 for safe community participation) than the lower scoring ACLS–2000 group. For room care, PST scores were significantly higher for participants with higher ACLS–2000 scores than for the lower scoring ACLS–2000 group at baseline (p = .010), intervention completion (p = .044), and 3-mo follow-up (p = .012). By contrast, the differences between the high and low ACLS–2000 groups were significant only at baseline for both money management (p = .002) and self-care (p = .021) PST scores.

We also completed mixed-model regression analyses to determine whether mean PST scores changed significantly over time for the high and low ACLS–2000 groups for each module. Results indicate that participants with higher ACLS–2000 scores had significant increases over time in mean PST scores for varying time points for food management (p = .021), money management (p = .039), and safe community participation (p = .02). Mean PST scores showed no significant changes over time for the high ACLS–2000 group for room care (p = .861) or self-care (p = .681).

Participants in the low ACLS–2000 group also had a significant increase in PST scores from baseline to 3-mo follow-up (p = .049) in food management and from baseline and intervention completion to 6-mo follow-up (p = .003 for food management and .044 for safe community participation) in safe community participation. By contrast, self-care scores significantly decreased from intervention completion and 3-mo follow-up to 6-mo follow-up (p = .017 for food management and .049 for safe community participation) for this group. No significant changes over time were found in mean PST scores for the low ACLS–2000 group in money management (p = .656) or room care (p = .847).

Effect Size

Effect sizes for changes in PST scores from baseline to each follow-up point for all modules are reported in Table 1. The greatest effects from baseline to follow-up were found in food management and safe community participation. As seen in Table 1, we found medium effect sizes after intervention and at each follow-up point for food management and noted large to medium effects for safe community participation over time.

Effects for the remaining modules were small and more variable by time point. We saw small effects for self-care after intervention and at 3-mo follow-up and for money management at 3- and 6-mo follow-ups. No effects were seen for room care.

We also completed an exploration of differences in the effect of the intervention between high and low ACLS–2000 groups for possible clinical significance that is reported here for discussion purposes. Trends noted include larger effects among the low ACLS–2000 group after intervention (0.98) and at 3-mo follow-up (1.70) than among the high ACLS–2000 group (0.61 after intervention and 0.48 at 3 mo) in food management and after intervention for self-care (0.64 vs. 0.21). For safe community participation, we saw comparable effects between groups after intervention (low = 0.75, high = 0.79) and at 3-mo follow-up (low = 0.47, high = 0.56); however, we saw larger effects in the low ACLS–2000 group at 6-mo follow-up (1.31 vs. 0.58). Money management and room care modules showed minimal effects in both groups.

Association Between the ACLS–2000 and the PSTs

The ACLS–2000 correlated with all the PSTs at baseline (except self-care), an outcome we expected, because people with higher cognitive functioning should be expected to have more baseline knowledge. The continued correlation with the postintervention test and 6-mo follow-up may be explained by the limited range of ACLS–2000 scores across the sample. Most of the sample scored within a narrow margin of higher scores, and the PST scores followed a similar pattern after intervention and at 6-mo follow-up. The absence of a correlation between the ACLS–2000 score and the PST score 3 mo after intervention is likely to be explained by the disruption people experienced with moving. In related research, some participants in the study also demonstrated increased trauma symptoms at the 3-mo follow-up, which correlated with higher rates of housing transition (Helfrich, Peters, & Chan, 2010).

Residential stability has been shown to benefit people by providing an environment that increases executive functioning skills—specifically, sustained attention—which allows people to use their life skills (Caplan et al., 2006). The cognitive reserves needed to adapt to a new setting and manage the chaos of moving with little social support and the increased social demands of moving, adjusting to new services, setting up utilities, and so forth may have diminished cognitive functioning (Fox, 2001). The rebound in scores to previous levels or higher is encouraging and may further support the residential stability literature. Although new learning would be expected with repetition and support from the environment (Allen, 1992), our participants seemed able to independently relearn by drawing on previously provided support from staff, peers, and written materials.

Outcome Predictors

Education was most frequently seen as a significant predictor of PST scores in three modules, in which participants with more education performed better on the PSTs. The amount of education received may be related to age of onset and severity of mental illness (Isohanni et al., 2001). Person factors related to cognitive functioning accounted for only a few module outcomes at a few time points, however, suggesting that the intervention itself most likely accounts for the changes in outcomes of the PSTs.

Intervention Effects

Over time, people who scored higher on the ACLS–2000 scored higher on most PSTs; however, both groups demonstrated change, as we discuss later. Room and self-care outcomes showed no change for room and decreases for self-care over time. Note that people living in emergency housing were provided with structure and material resources in both of these areas; those supports were lost when they moved out of the EHP.

Although we expected the intervention effects to be larger for the high-scoring ACLS–2000 group than for the low-scoring ACLS–2000 group, the low-scoring ACLS–2000 group demonstrated greater changes in outcomes. Similarly, the effects of the intervention were expected to be larger immediately after the intervention than at the 3- and 6-mo follow-ups; however, with few exceptions, outcomes were maintained for both groups throughout the follow-up period.

The social learning theoretical approach used to deliver the intervention most likely facilitated people at all ACLS–2000 levels to benefit from the intervention and demonstrate changes in life skills outcomes scores. In practice, the participants in the low ACLS–2000 group (ACLS–2000 score = ≤4.6) typically would not be included in an intervention that required new independent learning and abstract thinking.

This intervention was structured to be interactive and immerse participants in their community. After completing the intervention, they used the knowledge they learned in new situations. In follow-up focus groups, participants discussed using the handouts as a reference after they changed living situation or gained access to a microwave or financial entitlements (Helfrich, Sabol, & Simpson, 2009). Larger intervention effects are more likely seen in interactive programs, because participant engagement in program content promotes skill acquisition and attitudinal change (Stice, Shaw, & Marti, 2006).

The amount of change (effect size) observed varied between modules. The money management module produced no effect on performance immediately after intervention but demonstrated a modest effect after 3 mo and a small effect after 6 mo. People in the process of obtaining entitlements may not have found the information relevant until later, when they had money to budget and manage. These follow-up points most closely correspond to the time frame in which most people were able to obtain their entitlements and to move out of EHP, both time points are critical transitions for managing finances.

The greatest changes occurred in the safe community participation module. Many people joined this module to help the occupational therapist teach other group participants; however, participants demonstrated the most growth in life skills knowledge in this module. These results are notable for several reasons. First, this module was the most community based in terms of activities and interaction. Participants were accompanied to police beat meetings, used public transportation, and explored new aspects of their environment. This module was also unique to the intervention because the content of all the other modules was also being addressed in some way by case managers. Finally, this module included the most abstract and diverse information.

The room and self-care modules had the highest baseline scores and little change. Perhaps less new knowledge was to be gained because people already knew the information. Yet, how are deficits in room and self-care explained? One possibility may be that people know how to take care of themselves but do not have the capacity, resources, or motivation to actually do it. This idea needs further exploration; however, the notion is important, considering the emphasis occupational therapists place on basic self-care skills. The emphasis may need to be placed on building routines and follow-through rather than on basic skills.

Another consideration is that although people are living in the EHP, they have daily room checks, are assisted directly when they encounter problems such as bed bugs or lice, and are given rewards for exemplary room maintenance. Although these supports are reinforcing, they may not require people to be as cognizant of the issues related to room care as they must be when they move out. After moving into a less supportive and structured environment, people must take more responsibility and feel more ownership of their living space; this shift may explain the modest to large effect seen at follow-up for participants with higher ACLS–2000 scores.