Abstract
Study results partially support the use of the Stroke Impact Scale–16 (SIS–16) in clinical and research settings for patients with chronic stroke.
Stroke is a leading cause of long-term disability and mortality worldwide, and its burden is expected to rise over future decades as the population ages (Virani et al., 2020). The effects of stroke vary from person to person depending on the lesion and side of the brain affected. Although physical and cognitive impairments are the most common consequences of stroke, other impairments such as aphasia, dysphagia, urinary incontinence, and depression affect a substantial proportion of stroke survivors (Lawrence et al., 2001). Collectively, these deficits lead to activity limitations and participation restrictions. As a result, measuring health-related quality of life is crucial in assessing stroke outcomes and as an overall evaluation of the impact of stroke.
The Stroke Impact Scale (SIS), a stroke-specific quality of life measure, is becoming one of the most commonly used health status measures in evaluating stroke outcomes (Duncan, Bode, et al., 2003; Duncan, Lai, et al., 2003; Duncan et al., 1999). Published in 1999, the initial SIS 2.0 included 64 items in eight domains of functional status (strength, hand function, activities of daily living/instrumental activities of daily living [ADL/IADL], mobility, communication, emotion, memory/thinking, and participation; Duncan et al., 1999). In 2003, Duncan, Lai, et al. (2003) selected a subset of 16 items from the SIS 3.0 (Duncan, Bode, et al., 2003) for the SIS–16 to be used to assess self-reported physical function across the continuum of stroke severity.
Considerable research has addressed the reliability (Carod-Artal et al., 2008; Gonçalves et al., 2012; Mohammad et al., 2014; Ochi et al., 2017; Vellone et al., 2010, 2015), validity (Carod-Artal et al., 2008; Choi et al., 2017; Geyh et al., 2009; Gonçalves et al., 2012; Kamwesiga et al., 2016; Kwon et al., 2006; Lin, Fu, Wu, Hsieh, et al., 2010; Mohammad et al., 2014; Ochi et al., 2017; Vellone et al., 2010, 2015), and responsiveness (Lin, Fu, Wu, Wang, et al., 2010) of the SIS 3.0. Knowledge of the psychometric properties and applications of the SIS–16, however, is relatively scanty (Chou et al., 2015; Duncan, Lai, et al., 2003; Edwards & O’Connell, 2003; Kwon et al., 2006; Ward et al., 2011). The majority of the psychometric research on the SIS–16 was conducted during the initial development, in which Duncan, Lai, et al. (2003) used Rasch analysis to select the 16 items from the SIS 3.0 in patients 1 to 3 mo poststroke. Duncan, Lai, et al. found that compared with the Barthel Index, the SIS–16 contains more difficult items that can differentiate patients with less severe limitations, has less pronounced ceiling effects, and may be suitable for assessing a wide range of physical function limitations of patients.
Studies by Chou et al. (2015; n = 263, Taiwan) and Edwards and O’Connell (2003; n = 74, Australia) supported the convergent validity of the SIS–16. Chou et al. found excellent internal consistency (Cronbach’s α = .94), but the item-to-total correlation in the studies varied: .33 to .86 in Chou et al. and .37 to .77 in Edwards and O’Connell. Moreover, Chou et al. found that the SIS–16 had a mild ceiling effect (14%). In a small sample with 30 stroke patients, Ward et al. (2011) demonstrated that the SIS–16 could detect change (standardized response mean = 1.65) and predict length of stay. Because limited evidence supporting the use of SIS–16 is available from these studies, we aimed to validate the psychometric properties of the SIS–16 using Rasch analysis with a larger sample of patients with subacute or early chronic stroke.
Method
Data Source
In this secondary data analysis study, we extracted data from the published database of the Field Administration of Stroke Therapy–Magnesium (FAST–MAG) Trial. The purpose of the FAST–MAG Trial was to evaluate the effectiveness and safety of field-initiated magnesium sulfate in improving the long-term functional outcomes of patients with stroke.
The FAST–MAG Trial started in January 2005 and concluded in March 2013. Emergency medical services personnel screened patients for study entry when they arrived at the hospital. Inclusion criteria were (1) suspected stroke identified by the Los Angeles Prehospital Stroke Screen; (2) age 40–95 yr; (3) treatment initiation within 2 hr of symptom onset; and (4) presence of deficits >15 min. Among the many exclusion criteria were coma, rapidly improving neurological deficit, head trauma in the past 24 hr, recent stroke within the past 30 days, and inability to give informed consent.
As a secondary outcome measure, FAST–MAG Trial participants completed the self-reported SIS 3.0 at 3 mo poststroke. For this study, we used their answers to the 16 items that form the SIS–16 for analyses. The SIS–16 includes 7 ADL/IADL items, 8 mobility items, and 1 hand function item. Each item is rated on a 5-point Likert scale from 1 (could not do at all) to 5 (not difficult at all). The institutional review board at the University of Wisconsin–Milwaukee waived approval because this secondary data analysis used deidentified data free of personal identifiers.
Rasch Analysis
We analyzed the SIS–16 data using the partial credit model (PCM) with ACER ConQuest Version 2.0 (ACER Press, Camberwell, Victoria, Australia). Part of item response theory (IRT), the Rasch model is a probabilistic mathematical model that assumes the probability of passing an item depending on the relationship between a person’s ability and an item’s difficulty. We used the PCM because the SIS–16 has a polytomous rating scale structure. The PCM enables the location and relative distance between rating scale thresholds to be estimated separately for each item, allowing partial correctness to vary across items. In addition, we had a sufficiently large sample to justify using the PCM. Using the Rasch model, we examined several psychometric properties of the SIS–16, including item difficulty hierarchy, item fit, person–item match, separation index, person reliability coefficient, and ceiling and floor effect.
Item Difficulty Hierarchy
Item difficulty hierarchy is the ordering of items from least to most difficult to perform. The empirical item difficulty hierarchical order produced by Rasch analysis can be used as evidence of construct validity of the theoretical base of the scale (i.e., whether the item difficulty hierarchical order of the SIS–16 items reflects the clinical reasoning of the physical function construct).
Item Fit
Item fit indicates how well items fit a unidimensional latent trait (Linacre, 2002). The Rasch model assumes that people have a higher probability of successfully performing easier items (compared with their physical functional ability) and a lower probability of successfully performing more difficult items. A person is considered to misfit the Rasch model if their responses or performances do not follow the expected response pattern (e.g., a person with poor physical functional ability reports being able to run a marathon but unable to walk around the house).
There are two types of fit statistics, infit and outfit. Infit is an information-weighted residual and is more sensitive to item fit around the person’s estimated ability level. Outfit is an unweighted mean square and therefore is more susceptible to unexpected responses away from the person’s estimated ability level. Items with fit statistics greater than 1.40 or less than 0.60 are considered misfit items for clinical questionnaires using a Likert scale (Wright, 1994).
Person–Item Match
Person–item match is the extent to which the items are of appropriate difficulty for the study sample. In Rasch analysis, both person ability and item difficulty are expressed with a common metric, logits. Whether the items are of appropriate difficulty for the sample can be examined by comparing the score distributions of person ability measures and item difficulty estimates.
Separation Index
The separation index (G) indicates the extent to which the items distinguish distinct levels of functioning within the sample (Wright, 1982). The separation index is an estimate of how well the scale can differentiate people into statistically distinct person strata using the following formula: strata H = (4*G + 1)/3. For example, a stratum H equal to 3, equivalent to a reliability value of .80, indicates that the scale can statistically distinguish low-, moderate-, and high-functioning examinees (Fisher, 1992).
Person Reliability Coefficient
The person sample reliability indicates the reproducibility of person ordering. The person reliability coefficient ranges from 0 to 1, with values closer to 1 indicating greater consistency in the person ability estimates (Linacre, 1997). A reliability coefficient is classified as excellent (≥.90), good (≥.80), or acceptable (≥.70; Linacre, 2007).
Ceiling and Floor Effects
Ceiling and floor effects are the extent of score clusters toward the high and low end of the scale. Ceiling and floor effects limit the range of data reported by the scale and thus reduce the chance to detect true change. A ceiling effect is considered present if >20% of examinees achieve the best or maximum score, and a floor effect is considered present if >20% of examinees achieve the worst or minimum score (Salter et al., 2013).
We made a scatter plot with 95% confidence intervals (CIs) of the item measures with the entire sample and without extreme scores. We used the plot to check whether the item difficulty parameters were invariant regardless of the ceiling and floor effects (Bond et al., 2007; Wright, 1977; Wright & Stone, 1999).
Unidimensionality and Local Independence
To assess IRT assumptions of unidimensionality, we conducted exploratory factor analysis (EFA) of latent trait variables, followed by confirmatory factor analysis (CFA) with one-factor extraction, using Mplus (Muthén & Muthén, 2004). A unidimensional scale has only items that represent the same construct. To test for unidimensionality, we analyzed the factor loadings and variances explained by each factor. To test for local independence, we analyzed (1) the residual correlation matrix, (2) the magnitude of the standardized coefficients, and (3) the percentage of absolute residual correlations (>.20 suggests local dependence). As suggested by Nunnally (1978), items with factor loadings above .40 in one factor are preferred. Model fit was evaluated using the comparative fit index (CFI), the Tucker–Lewis index (TLI), and the root mean square error of approximation (RMSEA). The TLI and CFI range from 0 (poor fit) to 1 (good fit); values greater than .90 are indicative of good model fit. RMSEA values less than .08 suggest adequate fit (Wang et al., 2014).
Results
Analytic Sample
Of 1,700 patients who were enrolled in the FAST–MAG Trial, 1,010 completed the SIS 3.0. Their mean age was 68 yr (SD = 13; range = 40–95); 40% were female; and 78% were White, 14% Black, and 7% Asian. About 80% of these participants had ischemic stroke. The mean National Institutes of Health Stroke Scale (NIHSS) score was 8.0 (SD = 7.4), and the mean Barthel Index score was 98.8 (SD = 5.4), suggesting that most of the study sample had a slight dependence for personal care and mobility assistance. Table 1 summarizes the characteristics of the FAST–MAG study sample.
Characteristics of FAST–MAG Trial Participants
Note. df = degree of freedom; FAST–MAG = Field Administration of Stroke Therapy–Magnesium; SIS = Stroke Impact Scale 3.0.
t test for continuous variables; χ2 test for categorical variables.
To examine selection bias, we compared patients who completed the SIS to patients with no SIS data using independent t tests for continuous variables and χ2 tests for categorical variables. Patients who completed the SIS were younger, had lower NIHSS scores, and were more likely to be male, to be Black, and to have ischemic stroke compared with those who did not have SIS data.
Item Difficulty Hierarchy
Table 2 presents the item difficulty parameter estimates of the SIS–16; items are sorted by difficulty from least difficult (bottom) to most difficult to perform (top). “Climb a flight of stairs” was the most difficult item, followed by “walk fast,” “heavy household tasks,” and “carry heavy objects.” “Sit without losing balance” was the easiest, followed by “control bowels,” “control bladder,” and “get to toilet on time.” Although step calibrations (i.e., the Rasch–Andrich threshold) should increase monotonically, 16 items had disordered thresholds.
Item Difficulty Parameters and Step Calibrations of the Stroke Impact Scale–16
Note. MnSq = mean square fit statistic. Items are listed according to their degree of difficulty (in logits). Disordered rating scale categories are marked in bold. “Step” is the Rasch–Andrich threshold.
Item Fit
Two items had infit statistics greater than 1.40: “control bladder” and “control bowels.” Four items had outfit statistics greater than 1.40: “carry heavy objects,” “get to toilet on time,” “control bladder,” and “control bowels.” One item, “move from bed to chair,” had outfit statistics less than 0.60.
Person–Item Match
Figure 1 shows the distribution of the person ability measures (left) along with the item difficulty estimates (right). Participants with poor physical functional abilities appear at the bottom, and those with better physical functional abilities appear at the top. Similarly, items are sorted by difficulty from least difficult (bottom) to most difficult (top) to perform. Compared with the mean item difficulty estimate of 0.0 (SD = 1.1), the patient sample had a higher mean of 2.1 (SD = 2.0).
The person–item map of the Stroke Impact Scale–16.
Separation Index and Person Reliability
The separation index of 2.85 indicates that the SIS–16 differentiated participants into 4.1 statistically distinct strata. The person reliability coefficient was .89.
Ceiling and Floor Effects
A total of 244 (24.2%) participants obtained the maximum score, whereas only 2 (0.2%) obtained the minimum score. Figure 2 shows that the item difficulty parameters were not significantly different for all participants and those without extreme scores (ceiling and floor effects), except for six items (“climb a flight of stairs,” “walk fast,” “heavy household tasks,” “carry heavy objects,” “control bowels,” and “sit without losing balance”).
Scatter plot showing 95% confidence intervals (parallel lines) of the item measures (logits) with the entire sample (x-axis) and without extreme scores (y-axis).
Unidimensionality and Local Independence
Our analysis supports the unidimensionality of the SIS–16. The first factor explained 84% of the total variance, the second factor 5%, and the third factor 3%. The factor loadings ranged from .87 (“sit without losing balance”) to .96 (“walk without losing balance”). Out of 16 × (16 − 1)/2 = 120 item pairs, 0 item pairs had absolute correlation residuals higher than desired (>.20). The CFI was .98 and the TLI was .99, suggesting a good model fit. The RMSEA was .18, suggesting a fair model fit.
Discussion
The results of this study partially support good psychometric properties of the SIS–16 in a clinical stroke setting. The SIS–16 has a clear item difficulty hierarchy and a high person reliability coefficient and can discriminate at least four levels of physical function. However, the inherent misfit to a unidimensional model and disordered thresholds (i.e., disordered rating scale steps or reversed deltas) are concerning. In addition, the ceiling effect suggests that the number of items addressing the upper levels of physical function is insufficient.
Fit statistics indicate that most items measured the intended construct and were unidimensional. Four items misfit the model and had fit statistics greater than 1.40. The finding of misfit of bladder and bowel items was not unusual and has previously been found in studies of the FIM® (Pretz et al., 2016) and the Minimum Data Set (Wang et al., 2008). For the SIS–16, “control bowels” had the lowest item-to-total correlation of .37 (Edwards & O’Connell, 2003). Bowel control has an inherent component of involuntary neurological muscle control and the involvement of both autonomic and somatic nervous systems, so this item does not fit the measurement model with other physical functioning items (Fisher, 1997). Although bowel and bladder management is an important aspect in measuring functional independence in people with stroke, control of bowel and bladder functions is fundamentally neurological. Constructing a unidimensional scale is one of the most challenging aspects in developing patient-reported outcome measures in health care. Future studies are warranted to determine whether removing or revising the wording of the bowel and bladder items would improve the reliability, validity, responsiveness, and predicting power of the SIS–16 in measuring the physical functioning for stroke outcomes.
The EFA showed that a dominant factor explained 84% of the total variance. The CFA demonstrated that both the CFI and the TLI exceeded .95, suggesting a good model fit, whereas the RMSEA was .18, suggesting a fair model fit. Previous studies have used different cutoff criteria for fit indexes. Pinto et al. (2020) adopted the values of CFI >.90, TLI >.90, and RMSEA <.06. Schuller et al. (2019) and Hu and Bentler (1999) recommended more stringent criteria with values of CFI >.95, TLI >.95, and RMSEA <.06. Nonetheless, the RMSEA values for SIS–16 items were .18, .14, .08, and .06 for factor analysis based on one-, two-, three-, and four-factor solutions, respectively, suggesting that further revisions of the items to improve the unidimensionality may be needed. Accordingly, the findings of EFA and CFA generally support the unidimensionality of the SIS–16 items.
The item difficulty hierarchy in this study is similar to that of a study of the SIS physical function domain by Duncan and colleagues (Duncan, Bode, et al., 2003; Duncan, Lai, et al., 2003), with the exception that “climb a flight of stairs” was not ranked in the top four most challenging items in that study. Implicitly, the high replicability in the item difficulty hierarchy supports the construct validity of the SIS–16. Moreover, Figure 2 shows that the item parameters were not significantly different between all participants and those without extreme scores (ceiling and floor effects), except for six items. The 95% CI was used to examine the estimated item difficulties in samples with and without extreme scores. This index was suitable because the standard errors of item difficulties are not affected by item difficulty, so the standard errors would be similar for both difficult and easy items (Wright, 1977; Wright & Stone, 1999). The findings suggest that the item parameters are generally invariant and that the SIS–16 can be used for assessment of most patients in clinical settings.
Although we found the item difficulty hierarchy of the SIS–16 to be consistent with that of other physical functioning domains, there were disordered thresholds (Adams et al., 2012) among the items of SIS–16. The criteria for rating scale category utility suggested by Linacre (1999) indicate that step calibrations should increase monotonically. Put differently, as a person’s physical functional ability improves, the odds of observing rating category k + 1 rather than k should increase. The disordered thresholds may be ascribed to patients’ inability to clearly distinguish the descriptions of the rating scale categories (e.g., “somewhat difficult” vs. “a little difficult”). Duncan, Bode, et al. (2003) also found disordered thresholds in the SIS–16 and opted to keep five levels of scoring to develop this instrument. Collapsing the categories into three groups may be a potential solution. However, more rating scale categories increase the information gathered and improve person reliability.
Limitations
This study has several limitations. First, because the study was a secondary analysis of prospectively collected data, we were not in control of the data collection procedures. Second, nearly 60% of enrolled patients (1,010 out of 1,700) completed the SIS. Missing data are inevitable for research studies taking place in routine hospital settings. Selection biases associated with patients who completed the SIS 90 days after their stroke are unknown and might have affected the generalizability of results. Third, because this study sample consisted of patients who had experienced a stroke 3 mo previously, the results are not generalizable to patients with early-onset acute or chronic conditions. Fourth, model selection is often a subjective process, and results may vary depending on the model used. Finally, the Rasch model is a one- parameter logistic IRT model with constrained item discrimination parameters (Bond et al., 2007). Investigations of psychometric properties such as item discrimination and multidimensionality are needed using more complicated models.
Implications for Occupational Therapy Practice
To provide the most effective care in stroke rehabilitation with the aim of improving patient outcomes, occupational therapy practitioners must understand the psychometric properties of disease-specific outcome measures such as the SIS–16 to select the best measure. The findings of this study have the following implications for occupational therapy practice: • Results from this study partially support sound psychometric properties of the SIS–16 in a clinical stroke setting. • The SIS–16 has a clinically meaningful item difficulty hierarchy and a high person reliability coefficient, and it can discriminate four levels of physical function. • Further research is required to address the inherent misfit to a unidimensional model, disordered thresholds, and ceiling effect.
Conclusion
Our results partially support the validity and clinical use of the SIS–16 in subacute or early chronic stroke clinical settings. Further research is warranted to examine the psychometric properties of the SIS–16 in samples of people with stroke at different acuity stages.
Footnotes
Acknowledgments
The authors declare that there is no conflict of interest. Chia-Yeh Chou and Ching-Lin Hsieh contributed equally to this work and serve as the corresponding authors. This study was partially supported by a grant from the National Science and Technology Council in Taiwan (MOST 110-2314-B-030-007).