Abstract
This study examined reliability of the Health Enhancement Lifestyle Profile–Screener (HELP–Screener), a 15-item self-report questionnaire serving as an initial screening of lifestyle behaviors for older adults. The internal consistency of HELP–Screener was analyzed using data derived from a sample of 483 older adults. The resultant Cronbach’s α coefficient of .74 indicated an acceptable level of internal consistency. The test–retest reliability study with 90 participants yielded an intraclass correlation coefficient of .93, indicating a high degree of temporal stability of the instrument at the scale level. Moreover, good to excellent degrees of agreement found through the κ statistic (k s = .76–.96) and percentage of agreement (96%–99%) between the test and retest scores of each item further supported HELP–Screener’s test–retest reliability at the item level. Future studies should determine other aspects of psychometric properties and clinical suitability of this new instrument.
The Health Enhancement Lifestyle Profile—Screener (HELP–Screener) is a 15-item self-report questionnaire designed to screen older adults for health-risk behaviors and lifestyles. Conceptually excerpted from the original 56-item Health Enhancement Lifestyle Profile (HELP; Hwang, 2010b), HELP–Screener can help to quickly identify older adults who may require a more comprehensive evaluation through HELP and who may, as a result, benefit from a lifestyle intervention. Echoing the manifold constructs and subscales embedded in the original HELP, HELP–Screener also encompasses diverse aspects of health-related lifestyle behavior such as exercise, diet, socialization, leisure, and spirituality. Some sample items are as follows:
I exercise more than twice a week.
I frequently avoid those foods high in fat, cholesterol, sodium, or sugar (e.g., red meat, butter, eggs, canned soup, desserts).
I engage in my religious/spiritual activities at least once a week.
I pursue my hobbies at least once a week.
Both HELP and HELP–Screener were designed to be self-report questionnaires. Although apt to be more client centered, self-report instruments often introduce subjectivity in that they rely on respondents’ interpretation of questions and selection of responses (Czaja & Blair, 2005). Given the inevitability of such subjectivity, establishing psychometric properties for self-report instruments is particularly important. HELP–Screener’s antecedent, the HELP, has been tested for the different areas of reliability and validity as well as for clinical utility through multiple procedures and studies (Hwang, 2010a, 2010b, 2010c; Peralta-Catipon & Hwang, 2011).
More recently, several studies were proposed to determine HELP–Screener’s psychometric properties. A preliminary study has been conducted to determine the internal validity of HELP–Screener and to establish the cutoff score of this instrument (Hwang, 2012). HELP–Screener’s unidimensionality and data–model fit were largely supported through Rasch analyses of principal components of residuals, fit statistics, local dependency, differential item functioning, and item hierarchy. The central tendency measure and standard deviation of the HELP–Screener scores generated from a major study sample were used to calculate the cutoff so as to help delineate clinical significance of the screening results (Hwang, 2012). Further studies are to be performed to corroborate the psychometric and clinical soundness of this new instrument.
The purpose of this study was to explore HELP–Screener’s reliability with a sample of community-dwelling (noninstitutionalized) older adults. Specifically, I sought to examine HELP–Screener’s internal consistency (the extent to which the items measure the same attribute, namely, scale homogeneity) and to measure its test–retest reliability (consistency of the scores from two administrations) at both scale and item levels. An acceptable to good level of reliability was expected for each line of inquiry.
Method
Research Design
This study was part of the main project aimed to develop and validate HELP–Screener (Hwang, 2012). Two types of reliability were examined in this study: internal consistency and test–retest reliability.
Participants and Sampling
The study sample consisted of older adults residing in Southern California who met the following criteria: age ≥ 55 yr, community dwelling (noninstitutionalized), and adequate cognitive and English- or Spanish-language capabilities to respond to a questionnaire. The project was approved by the institutional review board of the California State University, Dominguez Hills. Several nonprobabilistic sampling methods, including convenience, network and snowball, and quota sampling, were used in such a way as to maximize the representativeness of the study sample. Participants were recruited from a wide range of community sites (including senior citizen centers, senior residential communities, regular residential houses and apartments, adult day health care centers, and local senior social and activities groups). Details of the sampling and recruiting procedures were described in Hwang (2012). It is noteworthy that the examination of test–retest reliability was specifically proposed as an ancillary study to the main project, in which prospective participants were informed of two administrations of HELP–Screener in a 2-wk time interval.
Instrument
HELP–Screener is a self-report questionnaire consisting of 15 items with yes (1) and no (0) response categories, which yield a score range of 0–15; higher scores are indicative of healthier lifestyles. All items are positively worded. The length of time needed to complete HELP–Screener is <5 min (Hwang, 2012). A Spanish version of HELP–Screener was also developed using the translation and back-translation method (Sperber, DeVellis, & Boehlecke, 1994) by two bilingual occupational therapy graduate students.
Data Collection
HELP–Screener was administered by means of three methods: (1) on-site paper-and-pen administration to an individual or a small group of participants; (2) a direct interview in which an occupational therapy graduate student or a staff member at the site read out the questions and recorded the participant’s responses (a commonly used method for participants with visual or reading difficulties); or (3) hand-delivered, postal, or electronic mail with simple instructions enclosed. For the test–retest reliability study, participants completed HELP–Screener twice within a 2-wk interval. The 2-wk interval is commonly used in test–retest reliability studies to avert recall bias (Portney & Watkins, 2009). All participants were provided with a numeric identification code, and all returned questionnaires were checked for completeness before data entry.
Data Analysis
HELP–Screener’s internal consistency was analyzed with Cronbach’s α, a statistic calculated from the item–total correlations (i.e., the correlation of each item with the sum of the remaining items). In general, for a behavioral scale, a Cronbach’s α > .70 is indicative of an acceptable internal consistency, α > .80 is indicative of a good internal consistency, and α > .90 may suggest redundancy among test items (DeVellis, 2003).
Test–retest reliability was first examined using an intraclass correlation coefficient (ICC) that reflects degree of correspondence and agreement between the total scores of two test administrations (Portney & Watkins, 2009). The analysis was based on the two-way random model, ICC(2, 1), in which respondents and scores were considered to be random effects. As a general guideline, an ICC between .75 and 1.00 demonstrates good test–retest reliability; however, to secure reasonable validity, an ICC > .90 is usually recommended (Portney & Watkins, 2009). Given that the ICCs, as with other point estimates, are subject to variability around the population’s true parameter value, the 95% confidence interval (CI) estimator was also obtained.
As a further line of inquiry, the test–retest reliability of HELP–Screener was also assessed at the item level by using the κ statistic, a commonly used chance-corrected measurement of agreement between raters or test occasions (Portney & Watkins, 2009). It was decided a priori that a κ value of ≤.40 would be considered fair to poor agreement; .41–.60, moderate agreement; .61–.80, good agreement; and ≥.81, excellent agreement (Landis & Koch, 1977; Portney & Watkins, 2009). However, because the scoring of HELP–Screener items is dichotomous (either yes or no), a deflated κ value could result even when the agreement between two test administrations is high (Cicchetti & Feinstein, 1990; Gwet, 2008). Therefore, the percentage of agreement between the test and retest scores of each item was also reported despite the fact that it can be inflated by chance agreement.
All of these statistical analyses were performed with IBM SPSS Version 19 software (IBM Corporation, Armonk, NY) and Stata Release 12 software (StataCorp LP, College Station, TX).
Results
A study sample from the main HELP–Screener project mentioned earlier (N = 483) was used for the internal consistency study. Participants’ ages ranged from 55 to 97 yr, with a mean (M) of 70.3 and a standard deviation (SD) of 9.9. Of note, this major study sample included 90 participants who were recruited purposefully for the test–retest reliability study. The ages of the 90 participants ranged from 56 to 88 yr (M = 68.4, SD = 9.3). Table 1 provides a summary of participants’ demographics, including gender, ethnicity, marital status, education, and employment status.
Participant Demographic Characteristics
Note. Percentages may not add to 100 due to rounding.
The analysis of internal consistency (N = 483) revealed a Cronbach’s α coefficient of .74, indicative of an acceptable level of scale homogeneity within HELP–Screener’s 15 items. The corrected item–total correlations of the 15 items ranged from .19 to .51 (M = .35). That is, most of the items demonstrated either a low correlation (rs = .21–.40) or a moderate correlation (rs = .41–.60) with the remaining items in HELP–Screener.
The test–retest reliability study with the 90 participants who completed HELP–Screener twice within a 2-wk interval resulted in ICC(2, 1) = .93, p < .0001. The 95% CI of the ICC(2, 1) ranged from .90 to .96. The result indicated good test–retest reliability of HELP–Screener at the scale level.
Furthermore, the κ statistic (k) conducted to examine HELP–Screener’s test–retest reliability at the item level revealed ks ranging from .76 to .96 throughout the items, suggesting good to excellent agreement between the test and retest scores of each HELP–Screener item. Likewise, the percentage of agreement between the two test scores for each item ranged from 96% to 99%. Table 2 provides the detailed results of the percentage of agreement, k, and standard error and 95% CI of k for each HELP–Screener item.
Percentage of Agreement and κ Statistics (Test–Retest Reliability) for the HELP–Screener Items
Note. Only item key words presented. CI = confidence interval; HELP–Screener = Health Enhancement Lifestyle Profile–Screener; SE = standard error.
Discussion
This study was conducted to examine the reliability of the newly developed HELP–Screener. This screening tool provides a means to facilitate communication and raise awareness of potential concerns pertaining to health-related lifestyles of older adults and, subsequently, identify a need for at-risk older adults to undergo additional evaluations. Drewnowski et al. (2003) claimed that screening tools that address multiple lifestyle factors will assist public health programs in facilitating the successful aging of older adults. Conceptually akin to the multiple subscales included in the original HELP, HELP–Screener defines healthy lifestyle as a broader term that encompasses the physical, dietary, psychological, social, spiritual, and occupational aspects of health-promoting behavior. The preliminary Rasch analyses have proved the psychometric legitimacy of the inclusion of these seemingly distinct but pertinent lifestyle factors in the 15-item HELP–Screener (Hwang, 2012). The results of this study further support the internal consistency and test–retest reliability of this new instrument.
The analysis of Cronbach’s α coefficient suggested an acceptable level for HELP–Screener’s internal consistency (α = .74). This resultant α resembles the previous finding of the Rasch principal components analysis of standardized residuals that evidenced a moderate amount of variance (67.5%) of HELP–Screener scores being accounted for by the primary Rasch factor (Hwang, 2012). Given the all-inclusive feature of HELP–Screener described earlier, a high level of internal consistency (homogeneity) within the scale was not expected. That is, the goal in designing the 15-item HELP–Screener was for the items to be conceptually related and for each to contribute some unique information as well. Moreover, as mentioned previously, a very high coefficient (α > .90) or item–total correlation (r > .60) is not necessarily desirable, because it may introduce item redundancy, which in turn contradicts the prerequisite of briefness for a screening tool (DeVellis, 2003).
HELP–Screener’s test–retest reliability was confirmed at both scale and item levels by the analyses of ICC and κ statistic–percentage of agreement, respectively. When determining the time frame between HELP–Screener administrations, consideration was given to the construct assessed as well as participants’ memories. As mentioned, this study used a 2-wk interval that was thought to be long enough for memory of the questions and responses to fade among the participating older adults, thus counteracting recall bias (Portney & Watkins, 2009). Another attempt was made to avoid having the test and retest administrations performed across the time points for seasonal changes (e.g., fall to winter) or major holiday celebrations (e.g., Thanksgiving, Christmas) when people’s lifestyle behaviors (e.g., physical, social, or spiritual activities; eating habits) may alter accordingly. With control of the potentially confounding factors, the resultant satisfactory findings lent support to the temporal stability of HELP–Screener.
Limitations and Future Directions
Several limitations of this study warrant the need for caution in the use of HELP–Screener as well as the need for future research. First, despite its potentially sound psychometric properties, HELP–Screener is a self-report instrument that necessitates clients’ recognition of the evaluation purpose along with their truthful response to secure useful and accurate results. Second, given the dynamic, multifaceted nature of the lifestyle context, other factors—such as motivation, self-efficacy, functional status, and environmental supports and barriers—should be incorporated into the screening process through other methods or instruments. Similarly, because a person’s lifestyle can be affected by changes in functional and health status and personal or environmental factors (e.g., personal budget, life events, community resources, weather), periodic readministration of HELP–Screener is recommended. Last, because this study was conducted using data derived from a sample of older adults in Southern California, the generalizability of its psychometric utility and related findings is limited. Future studies should be conducted by other interested researchers to replicate the existing evidence of HELP–Screener’s validity and reliability or to explore new aspects of the instrument’s psychometric properties and clinical suitability.
Implications for Occupational Therapy Practice
The results of this study have the following implications for occupational therapy practice:
The 15-item HELP–Screener (Hwang, 2012) was designed to serve as a quick initial screen that would indicate whether further evaluation with the original 56-item HELP (Hwang, 2010b, 2010c) is necessary.
Occupational therapy practitioners working with older adults in various clinical settings can benefit from using such a short, wide-ranging lifestyle screening as part of their routine evaluation.
The further evaluation using the 56-item HELP can provide a more in-depth understanding of particular areas of a respondent’s life warranting lifestyle modifications or regimens. Consequently, a variety of health-promoting occupations can be facilitated among older clients to provide opportunities for enhanced levels of health and wellness.
Conclusion
This study provides evidence of reliability for the newly developed HELP–Screener, a 15-item health-related lifestyle screening questionnaire for community-dwelling older adults. The analysis of data derived from a sample of 483 community-dwelling older adults revealed a Cronbach’s α coefficient of .74, indicating an acceptable level of internal consistency for HELP–Screener. The test–retest reliability study with 90 participants resulted in ICC(2, 1) = .93 (p < .0001, 95% CI = [0.90, 0.96]), suggesting good test–retest reliability of HELP–Screener at the scale level. Additionally, the resultant κ statistics ranging from .76 to .96, along with the percentage of agreement ranging from 96% to 99% between the test and retest scores on each item, further supported HELP–Screener’s good to excellent test–retest reliability at the item level. Future studies contributing to cumulative evidence regarding the psychometric properties of this new instrument will be highly valued.
Footnotes
Acknowledgments
I thank all the facilities and participants who contributed their time and effort to complete HELP–Screener and related materials. I also thank the group of occupational therapy graduate students for their enthusiasm and participation in the training and data collection for the study.