Abstract
The Practical Skills Test (PST) is a new assessment of individuals’ knowledge of life skills. We evaluated the PST’s reliability, validity, and sensitivity to change among a homeless population. Participants were 123 homeless persons in a longitudinal experimental study who were assessed before and after intervention with the PST, Allen Cognitive Level Screen–2000 (ACLS–2000), and Impact of Event Scale–Revised (IES–R). The PST showed generally good internal consistency, no floor effects, and limited ceiling effects (<20% on each test). Supportive evidence for the PST’s convergent validity was seen in its moderate correlations with the ACLS–2000; we found no significant correlation with the IES–R. Paired t tests indicated that the PST is sensitive to changes in life skills after intervention, but effect sizes were small. The results suggest that the PST has generally good reliability and validity. However, ceiling effects suggest an area for further development.
Keywords
Life skills are the skills that are necessary for everyday life (Kurtz, Jeffrey, & Rose, 2010). In other words, life skills are the basic skills that are necessary to live independently and that support meaningful, productive roles (Muñoz, Garcia, Lisak, & Reichenbach, 2006). However, people with a history of homelessness often lack these basic living skills, leading to difficulties in maintaining stable housing and living independently in the community (Calysn, Morse, Klinkenberg, Yonker, & Trusty, 2002). This phenomenon is likely the result of functional limitations resulting from the physical or psychiatric disabilities that are common in this population (Calysn et al., 2002; Ho & Kehn, 2007; Hung & Nichani, 2002; Muñoz, Reichenbach, & Hansen, 2005; National Institute of Mental Health Behavioral Science Task Force, 1995; President’s New Freedom Commission on Mental Health, 2003).
Given homeless people’s frequent limitations in independent living skills, practical assistance and life skills training for daily activities such as budgeting, shopping, and cleaning are important to help them maintain stable housing and function independently in the community (Calysn et al., 2002). Some studies have suggested that agencies that include life skills training in their services show greater success in helping homeless people with mental illness find and maintain housing and community living than agencies that do not include life skills training (Calysn et al., 2002; Lemaire & Mallik, 2005; Wasylenki, Goering, Lemire, Lindsey, & Lancee, 1993). Training in particular life skills, including money management and housing management, is especially relevant for homeless individuals’ daily living (Muñoz et al., 2005; U.S. Department of Housing and Urban Development, 2009).
Helfrich (2006a, 2006b, 2006c, 2006d) developed a life skills intervention for people at risk for repeated homelessness to assist them with developing or rebuilding their independent living skills and improve their ability to maintain housing. The life skills intervention modules focus on food management, money management, home and self-care, and safe community participation (Table 1). Each module includes six group sessions with an additional 30- to 90-min individual session (per group) tailored to the needs of each individual. To be included in the present analysis, participants needed to complete a minimum of three group-equivalent sessions.
Contents of the Life Skills Intervention Modules
To assess the need for life skills training among homeless people and to evaluate improvement in knowledge after intervention, an effective measurement tool is needed. However, no existing measure fits these purposes and is tailored to the needs of people who are homeless. Most available tools are limited to measuring a specific domain of life skills. For example, the Test of Grocery Shopping Skills (Hamera & Brown, 2000) is considered a sensitive measure of executive function performance in a real-world setting. However, it only measures skills in the area of grocery shopping. Similarly, Duncombe (2004) used a modified version of the Kitchen Task Assessment developed by Baum and Edwards (1993) to measure the effect of intervention on kitchen task performance in people with schizophrenia. The virtual reality environment has also been used with people with brain injuries to assess their skills in meal preparation (Zhang et al., 2003).
Some instruments assess more than a single skill but do not cover the components that are most relevant to the homeless population. For example, Rosen, Hadzi-Pavlovic, and Parker (1989) developed the Life Skills Profile to assess function in people with schizophrenia. Although it contains some self-care items, the skills that relate to home management and community participation, which are especially important for the homeless population, are not addressed. McGourty (1979) developed the Kohlman Evaluation of Living Skills to assess activities of daily living and instrumental activities of daily living in five areas: self-care, safety and health, money management, transportation and telephone use, and work and leisure. Because this instrument was originally designed for patients living in a short-term inpatient psychiatric setting, some domains that are relevant to independent community living and housing maintenance (e.g., food preparation, housing management) are missing. Thus, a measurement tool is needed that covers the full range of life skill domains and specifically addresses the skills necessary for maintaining stable housing and integration into the community, such as budget management, home maintenance, and home safety.
Because of the limitations of existing measures, Helfrich and Fogg (2007) developed a new measurement tool, the Practical Skills Test (PST). The PST was developed to evaluate the knowledge gained and retained from the life skills curriculum. Specific items were identified on the basis of the areas deemed most critical in the curriculum through an iterative process of identifying needs in the literature, consulting an expert panel, and using input from representative individuals who had been homeless and had made the transition into living independently in the community. The format of the questions was designed to test respondents’ application of the material wherever possible, as opposed to memorization. The earliest versions of the test included more open-ended short-answer items but were determined to be too arduous for many participants, thus becoming more a test of endurance than of knowledge. Several equivalent versions of the PST were administered in the process of instrument development; the final version includes 30 items in each of four domains: food management, money management, safe community participation, and home and self-care (Helfrich & Fogg, 2007). An answer key was created with consultation from the interventionists delivering the life skills curriculum to aid in interrater reliability.
The PST has been used to assess life skills intervention outcomes in several studies (Helfrich, Chan, & Sabol, 2011; Helfrich, Peters, & Chan, 2010); however, its psychometric properties have not been fully examined. We undertook the current study for this purpose. Specifically, the study was designed to examine the following properties for each of the four domains of the PST:
Internal consistency
Construct validity (including convergent and divergent validity)
Ability to detect change, including floor and ceiling effects and sensitivity to change.
Method
Research Design
This study was part of a longitudinal repeated-measures study designed to investigate the effect of the life skills intervention on people who were chronically homeless. The larger study involved 96 participants in the intervention group and 27 in the control group and included baseline, postintervention, and 2–3 yr of follow-up data collection. In the current study, we used data collected at two time points: baseline (Time 1 [T1]) and immediately after completion of the 3- to 12-wk intervention (depending on the intervention schedule and how many sessions the participants attended; Time 2 [T2]). The assessments at each time point included a demographic form, the PST, the Allen Cognitive Level Screen–2000 (ACLS–2000; Allen et al., 2007), and the Impact of Event Scale–Revised (IES–R; Weiss, 2004).
After initial assessments were completed, the second author (Helfrich) randomly assigned 2 of every 4 participants to the control and experimental groups. Because the larger, life skills study was also a feasibility study, the research team decided to have a greater number of participants at the expense of a large control group. Thus, when recruitment was low, everyone was assigned to the experimental group. A total of 96 participants participated in at least one of the four life skills intervention modules. Twenty-seven participants did not receive the life skills intervention but completed the same assessments at each time point to be used as comparison data. Participants in both the intervention and the control group were included in all data analyses except for analysis of sensitivity to change, which included only participants in the intervention group. Both the larger study and the current study were approved by Boston University’s institutional review board. Two occupational therapists recruited and obtained written informed consent from participants.
Participants
Participants were recruited into the life skills intervention study through flyers posted in the service settings and direct contact with the interventionist and from seven housing placement programs, including an emergency housing program attached to an academic psychiatric hospital, a Single Room Occupancy program, a Housing First program, two housing search programs, and two housing stabilization programs in two major metropolitan areas in the Midwestern and Eastern United States. Potential participants were oriented to the project and then completed informed consent.
Criteria for participation in the study were (1) ≥18 yr old, (2) understanding of English, (3) ability to identify a life skill need, (4) ability to engage in 60-min group sessions and individual sessions, and (5) ability to give informed consent. All the participants were chronically homeless (including most participants with a documented or self-reported disability) or at risk for repeated homelessness.
Instruments
The three measurement tools for this study were the PST, the ACLS–2000, and the IES–R. The PST examines the person’s practical knowledge of skills required to live independently in the community (Helfrich & Fogg, 2007). It includes four separate tests: Food Management, Money Management, Home and Self-Care, and Safe Community Participation. Each test contains 30 items (10 true–false questions, 10 multiple-choice questions, and 10 open-ended short-answer questions). A dichotomous score (1 = correct, 0 = false) is given for each item; thus, scores can range from 0 to 30. Raters were trained to score the tests, and scoring guides were revised until 90% interrater reliability was achieved to meet standards for test development (Helfrich & Fogg, 2007). An initial examination of the PST’s internal consistency with 51 homeless people with mental illness indicated fair to good internal consistency (Cronbach’s αs = .54–.81; Helfrich & Fogg, 2007).
The ACLS–2000 is an evaluation of functional cognition, including the ability to process and use information and the capacity to learn or relearn skills demonstrated in a leather lacing task. It also predicts the level of assistance that a person will need to perform routine tasks and how he or she will perform in novel situations. Results yield an ordinal score on a 25-point scale ranging from 0 to 5.8. A higher score represents a higher cognitive level (Allen et al., 2007). The ACLS–2000 is similar to the previous version, the ACLS–90, which has high reported interrater reliability (r = .99).
We used the ACLS–2000 to examine the PST’s convergent validity since the functional cognition it measures affects functional performance, living skills, and knowledge acquisition (Allen et al., 2007). Previous studies indicated that the cognitive levels are significantly associated with the accuracy and efficiency of conducting daily tasks, such as shopping and kitchen activities, and impaired cognition is known to interfere with functional performance (Zayat, Rempfer, Gajewski, & Brown, 2011).
We used the IES–R to examine the PST’s divergent validity. The IES–R is a posttraumatic stress disorder (PTSD) screen consisting of a 22-item self-report questionnaire with three subscales that measure the three diagnostic indicators of PTSD: intrusion (eight items), avoidance (eight items), and hyperarousal (six items; Weiss, 2004). Participants rate their level of distress on a Likert scale ranging from 0 to 4. The IES–R total score is indicative of overall traumatic stress symptoms; higher scores indicate higher levels of symptoms. The IES–R has strong internal consistency (Cronbach’s α = .96) and a strong correlation with the PTSD Checklist, a standard measure to assess PTSD symptoms (r = .84) according to the Diagnostic and Statistical Manual of Mental Disorders (4th ed.; American Psychiatric Association, 1994). The subscales also demonstrate strong internal consistency (Cronbach’s αs = .87–.94; Creamer, Bell, & Failla, 2003). Previous literature has reported no significant relationship between PTSD and independent living skills (Davis & Kutter, 1998). A preliminary study (Helfrich et al., 2010) found that people demonstrated a decrease in IES–R scores during the implementation of a life skills intervention; however, the relationship between the PST and the IES–R needs further exploration.
Data Analysis
All analyses were performed using PASW Statistics 18 software (Predictive Analytics SoftWare, Chicago). Descriptive analyses were performed to examine sample characteristics, including demographic variables and psychiatric diagnoses.
We calculated floor and ceiling effects as the percentage of participants (including the intervention and control groups) who achieved the minimum and maximum scores at T1 and T2. Some researchers have suggested that for a measurement to be useful, no more than 20% of scores should be at ceiling or floor (Jette, Warren, & Wirtalla, 2005). We estimated internal consistency reliability for each test at T1 with Cronbach’s coefficient α. An α > .70 is regarded as good internal consistency, and an α < .50 is usually considered unacceptable (Streiner & Norman, 2008).
We examined construct validity through tests of convergence and divergence. Convergent validity is supported by correlation with a tool that measures a related construct, and divergent validity is supported by absence of correlation with a tool whose underlying construct is not believed to be related (Streiner & Norman, 2008). In this study, we expected that the ACLS–2000 would demonstrate moderate to strong positive correlations (r > .40) with the PST (convergence), whereas we expected the relationships between PST and IES–R to be weak (r < .30; divergence).
Finally, sensitivity to change indicates an instrument’s ability to measure degree of change over time. We used paired t tests and Cohen’s effect size (ES) to assess the change in PST scores between T1 and T2 assessments. Cohen’s ES was calculated as (mean at T2 minus the mean at T1) divided by the standard deviation at T1. Cohen regarded an effect size of ≤.20 as small, .50 as moderate, and .80 as large (Portney & Watkins, 2008).
Results
Table 2 shows the baseline demographic characteristics of the sample. A total of 123 participants completed baseline assessments; 112 (91.06%) reported multiple medical issues, with their primary disabling conditions being mental illness, substance abuse, or both. Among the 96 participants in the intervention group (control group n = 27), 49 completed the Food Management module; 59 completed the Money Management module; 45 completed the Safe Community Participation module; 46 completed the Home and Self-Care module; and 27 completed all four modules.
Demographic Characteristics of Participants at Baseline (N = 120)
Note. GAF = Global Assessment of Functioning; GED = general equivalency diploma; M = mean; SD = standard deviation. Table lists only the GAF categories in which participants scored.
American Psychiatric Association (2000). bTotal score = 100.
Ceiling and Floor Effects
Descriptive results are presented in Table 3. No floor effect was noted in any measure at baseline (T1). Ceiling effects ranged from 0.9% (Food Management) to 5.3% (Safe Community Participation). None of the tests had a ceiling effect >20%. After intervention (or, for the control group, 12 wk later at T2), no floor effect was found for any of the tests. More participants were at ceiling at T2; however, they were still within an acceptable range of 3.5% (Home and Self-Care) to 12.7% (Safe Community Participation). At both T1 and T2, the score distributions of all tests were negatively skewed (skewness between −1.68 and −0.79 at T1 and −2.10 and −1.01 at T2), indicating that the average score for each test tended to be high.
Descriptive Statistics for the PST by Time Point and Pearson Correlation Coefficients Between the PST, ACLS–2000, and IES–R at T1
Note. For each test, maximum score = 30; minimum score = 0. ACLS–2000 = Allen Cognitive Level Screen–2000; IES–R = Impact of Event Scale–Revised; PST = Practical Skills Test; SD = standard deviation; T1 = Time Point 1; T2 = Time Point 2.
p < .01.
Reliability
Internal consistencies for Money Management (α = .84), Safe Community Participation (α = .87), and Home and Self-Care (α = .81) were good (αs > .70), whereas the internal consistency of the Food Management test (α = .64) was a little below the .70 threshold but acceptable.
Convergent and Divergent Validity
Pearson correlation coefficient results are presented in Table 3. As expected, all the tests had moderate positive correlations with the ACLS–2000 (r > .40). However, correlations with the IES–R were small and nonsignificant for all of the PST tests, which was consistent with our hypothesis.
Sensitivity to Change
The results of paired t tests and ESs are shown in Table 4. Among participants who completed each life skills module, all test scores but Money Management improved significantly from T1 to T2 (p < .05). The test scores for Money Management improved from T1 to T2 but were at a borderline insignificant level (p = .05). However, all the effect sizes were relatively small (ES = 0.11–0.25).
Sensitivity to Change of the PST
Note. M = mean; PST = Practical Skills Test; SD = standard deviation; T1 = Time Point 1; T2 = Time Point 2.
Discussion
The results of this study provide support for the reliability and validity of the PST. In general, items on each of the four tests are congruent and appear to be measuring the same construct, as indicated by acceptable α levels. In addition, results demonstrated the expected convergent and divergent associations with other measures. As hypothesized, the PST was positively correlated with the ACLS–2000. This finding is consistent with previous evidence of the association between cognition and life skills among people with mental illness. As summarized by Green, Kern, Braff, and Mintz (2000), studies have consistently indicated that cognition (including executive functioning, working memory, and vigilance) is moderately correlated with functional outcomes (including skills for daily and community activities). More recent studies have also confirmed the finding that individuals’ daily functioning is positively associated with their cognition (Keefe, Poe, Walker, Kang, & Harvey, 2006; Kurtz, Wexler, Fujimoto, Shagan, & Seltzer, 2008; Schultz, Ellingrod, Turvey, Moser, & Arndt, 2003).
However, scores on the PST were not significantly correlated with the IES–R. This finding is also consistent with the literature indicating that life skills have no notable correlation with trauma symptoms (Davis & Kutter, 1998). Likewise, previous research has verified that psychiatric symptoms are generally not significantly associated with daily functioning, even though this finding may not fit with clinical impressions (Green, 1996; Kurtz, 2006). As noted earlier, this area needs further exploration.
Regarding floor and ceiling effects, we found no floor effect for all PST tests, and only Safe Community Participation had more than a minor ceiling effect at T2. The result for Safe Community Participation suggests this test may have less sensitivity to capture change or improvement. Although other tests did not have much of a ceiling effect, the responses were all negatively skewed with high mean scores, suggesting that the tests may not be sufficiently challenging for the participants. Thus, the tests may not discriminate well across the full range of life skills knowledge.
Although all test scores except Money Management showed significant change after life skills intervention, none of the tests demonstrated a substantial effect size. This result may also indicate insufficient difficulty of the tests, which was reflected in the high mean scores of each test at baseline. This limitation may affect the tests’ ability to detect improvement because there is not much room for scores to improve.
The results suggest some areas of potential improvement for the PST. Several approaches may be possible. The first approach is to adjust the difficulty of the tests. Although life skills deficiency has been a significant concern with homeless people, the participants obtained relatively high scores on the PST. This finding suggests a need to expand the test by adding more difficult items. Because expanding the length of a test can present problems of feasibility, application of item response theory (IRT) methods, especially with a computer adaptive test (CAT), could be explored. IRT methods can help to identify a pool of items that cover the full range of difficulty and to examine the match between the range of performance in the relevant population and the range covered by the items. Subsequent incorporation of the item pool into a CAT yields a measure that can be short to administer but very precise (Haley et al., 2006). The difficulty of the tests can also be adjusted by removing the items that are too easy for all participants or by changing some of the response formats from true–false options to multiple choice or open-ended short answer format to reduce the possibility of being correct just by chance.
Limitations and Future Research
This study has some limitations that should be considered. First, although the sample size was sufficient for each analysis, a larger sample would provide more reliable statistical estimates. Because the population studied is unstable and vulnerable in coping with mental illness and ongoing substance use, recruiting a large number of participants into the study and keeping them participating is especially difficult. Second, the study sample may limit the generalization of the results. The participants came from two metropolitan cities. Therefore, whether the study results are generalizable to people who live in rural areas or people with other types of living situations is questionable. Clinicians applying the PST to evaluate improvement of individuals’ life skills after treatment should keep in mind that the change of score on the PST may be in the small to moderate range.
This study’s findings may be used to revise the PST. Future research should also investigate use of the PST with other populations with life skills needs. For example, examining the validity of the PST for people with traumatic brain injury or cognitive disabilities would be useful. In addition, it will be important to examine differences on the PST among subgroups because life skills performance may differ depending on different housing situations, clinical conditions, or cognitive levels. Finally, because the PST is an assessment of life skills knowledge, validating the tests’ results against actual performance of skills in real-life situations will also be critical.
Implications for Occupational Therapy Practice
The results of this study have the following implications for occupational therapy practice:
The PST is a valid measure of life skills in food management, money management, safe community participation, and home and self-care for people who are homeless.
Evidence has shown that the PST is moderately associated with a client’s cognitive level but not with trauma symptoms.
The PST is a valid and useful measurement tool for measuring outcomes of occupational therapy life skills interventions.
Conclusion
This study fills a gap in both practice and research with this client population by providing preliminary evidence of the PST’s reliability and validity and by providing some directions for further development and application of the instrument. The findings suggest that more investigation and adjustment in terms of the difficulty and complexity of the items would be beneficial. The PST is the first instrument shown to be reliable and valid for evaluating the life skills of people who are chronically homeless or at risk for repeated homelessness, including those with significant mental health issues. The PST is most applicable for use in community settings in which clinicians need to understand clients’ life skills needs or examine effects of interventions that target life skills.
Footnotes
Acknowledgment
This research was supported by the National Institute on Disability and Rehabilitation Research Grants H133G090046 and H133G060180, the U.S. Department of Education, and the 2011 Dudley Allen Sargent Research Fund, Doctoral Student Research Grant, College of Health and Rehabilitation Sciences: Sargent College, Boston University.