Abstract
Examination of the eight- and four-domain scores of the Stroke Impact Scale 3.0 may not be valid; scores therefore may not provide a valid assessment of health-related quality of life among clients with stroke.
Health-related quality of life (HRQOL) addresses all domains of health that are influenced by health events (e.g., stroke) and are of concern to individuals (Karimi & Brazier, 2016). HRQOL is one of the major aspects of client-reported outcomes (Paz Ruiz, 2017), and it provides unique information about clients’ subjective feelings in related domains (e.g., satisfaction, self-perceived difficulties in mobility function) that cannot be obtained by objective measures (e.g., ability, performance of mobility function; Lynch et al., 2008). Assessing HRQOL helps clinicians detect clients’ difficulties in various domains, plan treatment programs according to domains of concern to clients, and monitor subjective feelings of treatment effects (Paz Ruiz, 2017). Thus, a client-reported measure of HRQOL is a prerequisite for promoting client-centered health care.
The Stroke Impact Scale 3.0 (SIS 3.0) is a HRQOL measure widely used by both clinicians and researchers in stroke rehabilitation (Pistoia et al., 2016). The SIS 3.0 was designed especially for clients with mild to moderate stroke to sensitively capture changes in their HRQOL (Duncan et al., 2003). The SIS 3.0 contains 59 items that conceptually assess eight domains important to clients: Strength, Hand Function, Activities of Daily Living/Instrumental Activities of Daily Living (ADL/IADL), Mobility, Communication, Emotion, Memory and Thinking, and Participation (Duncan et al., 2003). The eight-domain structure was validated by the developers using unidimensional Rasch modeling (Duncan et al., 2003). With the eight-domain structure, the SIS 3.0 has good test–retest reliability, concurrent validity, and responsiveness (Chou et al., 2015; Lin et al., 2010). Moreover, good psychometric properties have been found in the different language versions of the SIS 3.0 with cultural adaptions (e.g., changing the utensils used for eating; Chang et al., 2005; Kamwesiga et al., 2016). Thus, the SIS 3.0 appears to be a promising measure of HRQOL for users in both clinical and research settings in various cultural contexts.
However, the factorial validity of the SIS 3.0 has rarely been examined. To the best of our knowledge, only three studies have examined its factor structures (Duncan et al., 2003; Mohammad et al., 2014; Vellone et al., 2015). The developers’ Rasch analysis indicated that the items in each of the eight domains were unidimensional (Duncan et al., 2003). The other two studies, however, found that the eight-domain structure was not supported by confirmatory factor analysis (CFA). Mohammad et al. (2014) deleted 43 items with poor discriminative validity to achieve good model fits. Vellone et al. (2015) then proposed a four-domain structure—Physical, Cognitive, Emotional, and Social Participation—as an alternative solution for the SIS 3.0 on the basis of exploratory factor analysis. These findings suggest that the eight- and four-domain structures may both reasonably reflect clients’ HRQOL. However, because the two structures have not been compared simultaneously, the factorial validity of the SIS 3.0 remains unclear.
The unclear factorial validity limits the calculation of SIS 3.0 scores and interpretation of the results in two respects. First, it remains unknown which of these factor structures is more robust. Given that a robust structure is the foundation for calculating domain scores to represent clients’ HRQOL in each domain, the lack of a robust structure may lead to unclear interpretations of the domain scores. Second, the replicability of the structures also remains unknown. The four-domain structure was proposed and validated using the same sample, a method that tends to overestimate the model fit (Matsunaga, 2010). If an underlying structure cannot be replicated, the meanings of the domain scores are not consistent across different samples, and the utility of the SIS 3.0 is thus highly degraded. To confirm the underlying structure of the SIS 3.0, we compared the currently available eight- and four-domain structures simultaneously.
Method
Participants
We extracted responses to the SIS 3.0 from a previous study of psychometric comparisons of four HRQOL measures (Chou et al., 2015). In that study, 263 patients were recruited from five general hospitals in northern and southern Taiwan by convenience sampling. The participants were recruited from either rehabilitation wards (inpatients) or the clinics of neurology and rehabilitation departments (outpatients) from August 2008 to June 2010. The inclusion criteria were as follows: (1) diagnosis of stroke according to the Diagnostic and Statistical Manual of Mental Disorders (4th ed.; American Psychiatric Association, 1994), (2) hemiplegia caused by stroke, (3) age ≥18 yr, (4) sufficient ability to complete the assessments, and (5) sufficient cognitive function to follow simple instructions (Chou et al., 2015). No participating patients were excluded. Moreover, the data set of the previous study had no missing data.
Data Collection
The patients’ characteristics, stroke severity, self-care ability, and cognitive function were evaluated by licensed occupational therapists. The HRQOL measures (including the SIS 3.0 and two other HRQOL measures that were validated in the previous study) were completed by the patients in random order within 3 days. The study received ethical approval from the five hospitals. All participants provided informed consent.
Measures
The SIS 3.0 is an HRQOL measure developed by Duncan et al. (2003). The items were generated on the basis of the major concerns of clients with stroke, caregivers, and health professionals with stroke expertise (Duncan et al., 1999). The SIS 3.0 can be administered by face-to-face interview, completed by clients themselves, or completed by clients’ proxy if clients have insufficient ability to answer the questions (e.g., unable to follow a three-step instruction; Kline, 2005). This study used the Mandarin version of the SIS 3.0, which was translated through rigorous processes including forward–backward translation and cultural adaption (Chang et al., 2005). The Mandarin version of the SIS 3.0 has good test–retest reliability, convergent and predictive validity, and responsiveness in Taiwanese clients with stroke (Chou et al., 2015; Lin et al., 2010).
The National Institutes of Health Stroke Scale (NIHSS) was used to assess the patients’ stroke severity (Brott et al., 1989; National Institute of Neurological Disorders and Stroke, 2011). The total score ranges from 0 to 42; higher scores indicate greater stroke severity. The NIHSS has excellent test–retest and interrater reliability and good factorial validity in clients with stroke (Dewey et al., 1999; Goldstein & Samsa, 1997; Zandieh et al., 2012).
The Barthel Index (BI) was used to assess the patients’ self-care ability. The BI is composed of 10 items; the total score ranges from 0 to 100 (Mahoney & Barthel, 1965), with higher scores indicating less disability. The BI has excellent interrater reliability and concurrent validity in clients with stroke (Hsueh et al., 2002).
The Mini-Mental State Examination (MMSE) was used to assess the patients’ cognitive function (Burns et al., 1998; Folstein et al., 1975). The MMSE has 11 items in the domains of Orientation, Attention, Memory, Language, and Construction. The total score ranges from 0 to 30; a score of ≤24 is commonly accepted to indicate cognitive impairment. In general, the MMSE has acceptable psychometric properties in clients with stroke (Cumming et al., 2013).
Data Analysis
We first examined whether the eight- and four-domain structures of the SIS 3.0 were valid. For comparison with the previous study by Vellone et al. (2015), we examined the four-domain structure by item parceling. If neither structure was supported, we examined the unidimensionality of each domain in both structures.
We used CFA to examine whether the eight- and four-domain structures of the SIS 3.0 could be supported by the data. Four fit indices were used: (1) χ2/df, which represents the χ2 value with adjustment based on the model complexity (e.g., number of parameters needed for estimations); (2) comparative fit index (CFI), which shows how the data can be better explained by the given model when compared with a baseline model (e.g., a model with no correlations among items); (3) root mean square error of approximation (RMSEA), which indicates the estimated approximation of the error by the given structure; and (4) standardized root mean square residual (SRMR), which indicates how well the data can be explained by the given model (Hsueh et al., 2011). Given that the fit indices tend to be affected by the sample size (Taasoobshirazi & Wang, 2016), the criteria of good model fit were selected on the basis of a sample size of 250 patients: χ2/df ≤3.0, CFI ≥.93, RMSEA ≤.05, and SRMR ≤.10 (Sivo et al., 2006). To avoid reporting a model that cannot be replicated, only models that showed good model fit in all fit indices were considered strongly supported (Dion, 2008; Kline, 2005). The CFA was performed using EQS 6.1 (Bentler & Wu, 2005). The maximum likelihood estimators were adopted to estimate the parameters (e.g., factor loadings, interdomain correlations) in this study (Bentler & Wu, 2005).
Item parceling is an analytic strategy that uses the average score to replace the individual item scores (Rhemtulla, 2016). This strategy is commonly used to reduce the number of parameters in CFA, which is helpful when the sample size is relatively small (Rhemtulla, 2016). However, given that replacing item scores with average scores can reduce the number of parameters needed for estimations (e.g., factor loading; Rhemtulla, 2016), item parceling tends to mask some misfit findings and yield a better model fit (Bandalos, 2002; Rhemtulla, 2016). Therefore, this strategy is not recommended for validations of factorial validity. Nevertheless, for comparison with the findings of the previous study (Vellone et al., 2015), we used the same methods of item parceling (i.e., the same composition of item parcels using the average scores) to examine the four-domain structure.
Results
Table 1 shows the characteristics of the 263 patients with stroke (183 men, M age = 59.8). About half (48%) were inpatients (onset within 3 mo), and the rest were outpatients. In general, the patients were in the subacute to chronic stages of mild stroke (M NIHSS score = 4.4) and had moderate levels of self-care disability (M BI score = 79.7).
Participant Characteristics (N = 263)
Note. BI = Barthel Index; MMSE = Mini-Mental State Examination; NIHSS = National Institutes of Health Stroke Scale.
Table 2 provides descriptive information about the domain scores of the SIS 3.0. In general, the internal consistencies of the eight- and four-domain scores were both good (Cronbach’s αs = .80–.98). However, floor and ceiling effects were found in some domains in both structures, including a mild floor effect in the Hand Function domain (26.2%); mild ceiling effects in the Hand Function (23.6%), ADL/IADL (22.1%), and Cognitive (30.8%) domains; and a moderate ceiling effect in the Memory and Thinking (36.9%) and Communication (59.7%) domains.
Descriptive Information About the Scaling of the Stroke Impact Scale 3.0
Table 3 shows the overall data–model fits of the eight- and four-domain structures and those of each domain. The overall model fits of both structures were poor (χ2/df = 2.7 and 5.0, CFI = .79 and .86, RMSEA = .08 and .12, SRMR = .08 and .08, respectively). In addition, poor model fits were found in every one of the eight domains (χ2/df = 3.3–65.8, CFI = .66–.95, RMSEA = .09–.50, SRMR = .02–.15) and two (Physical and Cognitive) of the four domains (χ2/df = 66.3 and 102.7, CFI = .88 and .80, RMSEA = .22 and .48, SRMR = .06 and .07, respectively). The model fits of the Emotional and Social Participation domains in the four-domain structure were not estimated because of an insufficient number of parcels for model identification (i.e., ≤3 parcels).
Model Fits of the Eight- and Four-Domain Structures
Note. ADL/IADL = Activities of Daily Living/Instrumental Activities of Daily Living; CFI = comparative fit index; RMSEA = root mean square error of approximation; SRMR = standardized root mean square residual.
Fit index criteria: χ2/df, ≤3.0; CFI, ≥.93; RMSEA, ≤.05; SRMR, ≤.10.
The model fits of the Emotional and Social Participation domains were not estimated because of the insufficient number of parcels (i.e., 3 and 2, respectively).
Table 4 shows the factor loadings of the two structures. Generally, the factor loadings of the items were in the acceptable range (i.e., .54–.97). However, in the eight-domain structure, the Emotion and ADL/IADL domains each had three items with lower factor loadings (.27–.44 and .27–.43, respectively) than those of the other items (.59–.73 and .65–.91, respectively). Regarding the four-domain structure, the Physical and Emotional domains each had one parcel with a low factor loading (.38 and .39, respectively).
Factor Loadings of the Eight- and Four-Domain Structures
Note. P1 indicates the first parcel in the Physical domain, C2 indicates the second parcel in the Cognitive domain, and so forth. ADL/IADL = Activities of Daily Living/Instrumental Activities of Daily Living.
Item with reverse wording.
Discussion
We found that the eight-domain structure had poor model fits, which is consistent with previous findings (Mohammad et al., 2014; Vellone et al., 2015). The model fits might be poor for three reasons.
First, the composition of each domain might be complicated. In particular, poor model fits to the one-factor structure were found in each of the eight domains, indicating that these domains are not unidimensional. The nonunidimensional findings may have two explanations:
Other domains might be embedded in the eight domains—for example, the Participation domain appears to contain items in the Social Roles, Family Roles, and Work and Productivity domains of another well-known HRQOL measure (Stroke-Specific Quality of Life Scale; Williams et al., 1999).
Some correlations among items cannot be explained by the corresponding domains—for example, two items in the Strength domain appear to have unexplained item correlations because they both assess patients’ strength in their upper extremities (arm and wrist).
Second, items with reverse wording within a domain might have confused the patients (Suárez-Álvarez et al., 2018). For example, in the Emotion domain, the factor loadings (.27–.44) of the three items assessing patients’ positive feelings (e.g., “Feel that life is worth living?”) were lower than those (.59–.73) for the other six items, which assess patients’ negative feelings (e.g., “Feel quite nervous?”). The meanings of the response categories are determined by the content of the items (e.g., “all the time” indicates good for positive emotion but bad for negative emotion). Thus, patients might have made errors when items asking about positive and negative emotions were alternated, and such errors may have reduced the factor loadings. Thus, the wording of the three positive-feelings items might have been confusing for patients.
Third, items assessing toileting function might not be indicators of the ADL/IADL domain. Factor loadings for the three items assessing toileting function (i.e., “Control your bladder?” “Control your bowels?” and “Get to toilet on time?”) were lower than those (.65–.91) for the other items in the ADL/IADL domain. The low loadings for the three toileting items are consistent with those of a previous study (Leung et al., 2007), suggesting that the toileting items might not be appropriate indicators of the ADL/IADL domain.
We found that the model fits of the four-domain structure were poor, a finding not consistent with that of the previous study by Vellone et al. (2015). The inconsistent results have two possible explanations. First, the previous findings may have been overfitted, as commonly occurs when the same sample is used in exploratory factor analysis and CFA (Matsunaga, 2010). Second, the patients’ characteristics were different; specifically, more patients in the previous study were female (45.2% vs. 30.0%), and they were older (M age = 71.2 vs. 59.8 yr), were at earlier stages of recovery (about 20 days after stroke vs. generally at subacute to chronic stages), and had more severe stroke (M NIHSS score = 7.2 vs. 4.4) and self-care disability (M BI score = 44.0 vs. 79.7). These differences in the patients’ characteristics may have affected the correlations among items and led to inconsistent model fits.
Regarding the reasons for the poor model fits for the four-domain structure, the same reasons as for the eight-domain structure may be applicable. Similar problems were found in the two structures. For example, the two low-loading parcels in the four-domain structure were composed of the three positive-feelings items in the Emotional domain and the three toileting items in the Physical domain. One additional reason may be that the four-domain structure cannot comprehensively reflect the domains assessed by the items of the SIS 3.0; these four domains are combinations of the domains in the eight-domain structure (e.g., the Cognitive domain consists of the original Memory and Thinking and Communication domains), which may have resulted in the issue of complicated compositions within a domain.
We used item parceling to validate the four-domain structure, which may raise concerns about the impact of this strategy on the results. However, the adoption of item parceling may not be a critical issue in this study because item parceling tends to improve model fits. Specifically, unexplained correlations among items (which may result in poor model fits) are masked when the scores of these items are combined as an averaged score (i.e., a parcel; MacCallum et al., 1999). Although we used item parceling, however, the results for the four-domain structure were still poor. Accordingly, our findings strengthen the point that the four-domain structure does not appear to be a robust structure for the SIS 3.0.
The floor and ceiling effects in some domains may also be a concern for this factorial validation. In particular, the model fit of the eight- and four-domain structures may have been underestimated because of the reduced interitem correlations caused by floor and ceiling effects. However, unacceptable model fit was found in the domains without floor and ceiling effects (e.g., the Emotion domain in the eight-domain structure, the Physical domain in the four-domain structure). Therefore, although the model fits of some domains may be somewhat underestimated, the factorial validity of the eight- and four-domain structures seems not to be strongly supported.
Further modification and confirmation of the underlying structure would be helpful to strengthen the validity of the SIS 3.0. Such work can be based on the reasons we have suggested for the poor model fits. For example, regarding the apparent complexity of some domains, dividing the Participation domain into the domains of Role Function and Social Activities may be helpful. We did not modify the eight- and four-domain structures because we could not confirm the modified structure with robust evidence. Modifying and confirming a structure with the same sample are very likely to result in overestimation (Matsunaga, 2010). Thus, we have provided only the possible reasons for the poor model fits of the two structures. Future studies can consider this information for further modification and confirmation of the structures.
Limitations
This study has four limitations that may hamper the generalizability of our findings. First, we used a convenience sample. Second, information on stroke chronicity was unavailable because time after stroke was not recorded in the previous study, although the patients were generally in the subacute or chronic stages. Third, the number of patients who did not meet the inclusion criteria was not recorded in the previous study (Vellone et al., 2015). Fourth, the data were collected about 10 yr ago. However, because the SIS 3.0 is still commonly used by clinicians and researchers in stroke rehabilitation (Pistoia et al., 2016), validations of the underlying structures of the SIS 3.0 remain crucial to remind users that the domain scores for patients’ HRQOL should be interpreted cautiously. More validation of the underlying structure of the SIS 3.0 is warranted.
Implications for Occupational Therapy Practice and Research
The results of this study have the following implications for occupational therapy practice and research:
Neither the eight- nor the four-domain score of the SIS 3.0 seems to validly represent clients’ HRQOL. Therefore, until more supporting evidence is developed, the scores of the SIS 3.0 should be interpreted cautiously; alternatively, other stroke-specific HRQOL measures can be used.
Because the eight-domain structure had a better model fit than the four-domain structure, further modifications and confirmations of the SIS 3.0 based on the eight-domain structure are needed to improve the assessment and interpretation of HRQOL in clients with stroke.
Conclusion
We found that neither the factorial validity of the overall eight- and four-domain structures nor the unidimensionality of each domain was supported in this study. These findings indicate that the currently available eight- and four-domain scores may not validly reflect the HRQOL of clients with stroke, which may mislead users regarding clients’ HRQOL (e.g., over- or underestimated treatment effects). Moreover, because both structures could not be replicated, interpretations of results across studies will be difficult. Therefore, until more supportive evidence is established, scores on the SIS 3.0 should be interpreted with caution.
Footnotes
Acknowledgment
Ching-Lin Hsieh and Chia-Yeh Chou contributed equally to this work and serve as corresponding authors.