Validation of the Machine Learning–Based Stroke Impact Scale With a Cross-Cultural Sample

Abstract

Importance: The machine learning–based Stroke Impact Scale (ML–SIS) is an efficient short-form measure that uses 28 items to provide domain scores comparable to those of the original 59-item Stroke Impact Scale–Third Edition (SIS 3.0). However, its utility is largely unknown because it has not been cross-validated with an independent sample.

Objective: To examine the ML–SIS’s comparability and test–retest reliability with that of the original SIS 3.0 in an independent sample of people with stroke.

Design: Comparability was examined with the coefficient of determination (R ²), mean absolute error, and root-mean-square error (RMSE). Test–retest reliability was examined using the intraclass correlation coefficient (ICC).

Setting: Five hospitals in Taiwan.

Participants: Data of 263 persons with stroke were extracted from a previous study; 144 completed repeated assessments after a 2-wk interval.

Results: High R ² (.87–.95) and low mean absolute error or RMSE (about 2.4 and 3.3) of the domain scores, except for the Emotion scores (R ² = .08), supported the comparability of the two measures. Similar ICC values (.39–.87 vs. .46–.87) were found between the two measures, suggesting that the ML–SIS is as reliable as the SIS 3.0.

Conclusions and Relevance: The ML–SIS provides scores mostly identical to those of the original measure, with similar test–retest reliability, except for the Emotion domain. Thus, it is a promising alternative that can be used to lessen the burden of routine assessments and provide scores comparable to those of the original SIS 3.0.

Plain-Language Summary: The machine learning–based Stroke Impact Scale (ML–SIS) is as reliable as the original Stroke Impact Scale–Third Edition, except for the Emotion domain. Thus, the ML–SIS can be used to improve the efficiency of clinical assessments and also relieve the burden on people with stroke who are completing the assessments.

This study examined the comparability and test–retest reliability of the machine learning–based Stroke Impact Scale with that of the original Stroke Impact Scale–Third Edition in an independent sample of people with stroke.

The Stroke Impact Scale–Third Edition (SIS 3.0) is a well-known outcome measure used to assess health-related quality of life of stroke patients. The SIS 3.0 consists of 59 items assessing eight domains, such as Activities of Daily Living (Duncan, Bode, et al., 2003). To improve the feasibility of administration, three short-form versions of the SIS 3.0 have been developed, including two 8-item versions (Jenkinson et al., 2013; MacIsaac et al., 2016) and a 16-item version (Mohammad et al., 2014). The two 8-item versions each consist of 1 item for each domain (Jenkinson et al., 2013; MacIsaac et al., 2016), and both generate a single total score to represent the overall level of health-related quality of life (Jenkinson et al., 2013; MacIsaac et al., 2016). The 16-item version, however, contains 2 items for each domain (Mohammad et al., 2014), and eight domain scores are calculated to reflect the quality of life for each aspect of life. These short-form versions and the original SIS 3.0 have been widely used in research to evaluate the effectiveness of stroke rehabilitation.

The machine learning–based Stroke Impact Scale (ML–SIS) is a newly developed short form of the SIS 3.0 (Lee et al., 2022). The ML–SIS was developed to estimate eight domain scores, comparable to those of the SIS 3.0, using a few selected items from each domain of the SIS 3.0. The domain scores are calculated using pretrained deep neural networks established in a previous study (Lee et al., 2022). Compared with the aforementioned short-form versions (Duncan, Lai, et al., 2003; Jenkinson et al., 2013; MacIsaac et al., 2016; Mohammad et al., 2014), the major advantage of the ML–SIS is the comparability of its scores to those of the original SIS 3.0. Because the ML–SIS was developed to estimate scores for the eight SIS 3.0 domains (Lee et al., 2022), the estimated scores have the same scaling as those of the SIS 3.0 (Lee et al., 2022). In contrast, the other short forms of the SIS 3.0 report either a single total score only (Duncan, Lai, et al., 2003; Jenkinson et al., 2013; MacIsaac et al., 2016) or eight domain scores on which the range of scores differs from those of the SIS 3.0 (two items for each domain vs. four to nine items for each domain; Duncan et al., 1999; Hamza et al., 2012; Mohammad et al., 2014). Therefore, these short-form scores are hard to compare with those of the SIS 3.0, and the short forms cannot be used interchangeably with the SIS 3.0. On the basis of these considerations, the ML–SIS appears to be a promising short form that can lessen the burden of assessments, particularly in time-pressed clinical settings.

However, some uncertainties have limited the utility of the ML–SIS. First, the ML–SIS has not been cross-validated. Thus, its comparability to the original domain scores remains largely unknown. Second, its test–retest reliability has not been examined. Therefore, whether the ML–SIS is as reliable as the SIS 3.0 in repeated assessments is in doubt. Third, the ML–SIS was developed using data collected 90 days after stroke (Lee et al., 2022), so its usefulness with patients at other stages of stroke, such as the chronic stage, is unknown. Fourth, the ML–SIS has yet to be examined with people of different languages and cultures. As a result, whether the ML–SIS can be used in place of versions of the SIS 3.0 in other languages is unknown.

Taken together, although the ML–SIS has merits as an alternative to the SIS 3.0, cross-validation is warranted; it can be conducted with a cross-cultural sample with varying stroke chronicity to compare ML–SIS scores and test–retest reliability with those of the original SIS 3.0. If the ML–SIS shows comparable domain scores and test–retest reliability in such a sample, the results will provide strong evidence supporting the utility of the ML–SIS.

The current study aimed to cross-validate the ML–SIS in a cross-cultural sample with varying stroke chronicity for comparability and test–retest reliability. The research questions were as follows: (1) Can the ML–SIS can provide scores comparable to those of the original SIS 3.0 and (2) is the test–retest reliability of the ML–SIS similar to that of the SIS 3.0?

Method

Participants

The data were extracted from a previous study that examined the psychometric properties of three Mandarin Chinese versions of commonly used health-related quality-of-life measures (Chou et al., 2015). In that study, people with stroke were recruited from rehabilitation wards and outpatient departments of five general hospitals from August 2008 to June 2010 (Chou et al., 2015). The inclusion criteria were as follows: diagnosis of stroke, hemiplegia due to stroke, older than age 18 yr, ability to complete the self- reported health-related quality-of-life questionnaire, and ability to follow simple instructions (Chou et al., 2015). Data collection was approved by the institutional review boards of the five general hospitals. All participants provided informed consent before participating in that study.

Procedures

The demographic data were collected through interviews and confirmed through medical charts. The clinical data, including cognitive function (assessed with the Mini-Mental State Examination [MMSE]; Burns et al., 1998), stroke severity (assessed with the National Institutes of Health Stroke Scale [NIHSS]; Brott et al., 1989), and level of disability (assessed with the Barthel Index [BI]; Mahoney & Barthel, 1965), were compiled by licensed occupational therapy practitioners at the first assessment. The SIS 3.0 was completed by the participants themselves 2 wk apart (2–4 wk is considered an adequate interval for examining test–retest reliability; Chiang et al., 2023; Marx et al., 2003). In addition to the standard scoring process of the SIS 3.0, item scores were extracted to simulate administration of the ML–SIS and to produce corresponding domain scores.

Measures

The SIS 3.0 was developed to monitor health-related quality of life among people with stroke (Duncan, Bode, et al., 2003). As noted earlier, the SIS 3.0 consists of 59 items divided into eight domains: Strength, Hand Function, Activities of Daily Living/Instrumental Activities of Daily Living, Mobility, Communication, Emotion, Memory and Thinking, and Participation/Role Function. Items are rated on a 5-point Likert-type scale for self-perceived difficulty (1 = not difficult at all, 5 = extremely difficult) in performing specific activities (e.g., think quickly) and the frequency (1 = none of the time, 5 = all of the time) of specific conditions (e.g., feel sad). Item scores for each domain are summed as domain scores to indicate the level of satisfaction with these life aspects. A higher score indicates higher satisfaction. The SIS 3.0 has generally good reliability and validity among people with stroke. In this study, the Mandarin Chinese version of the SIS 3.0 was adopted (Chou et al., 2015; Lin et al., 2010) because it has also shown good reliability and validity with people with stroke (Chou et al., 2015; Lin et al., 2010).

The ML–SIS was developed to improve the efficiency of the original SIS 3.0 (Lee et al., 2022). The ML–SIS uses about 50% of the items (i.e., 28 items) to achieve scores comparable to those of the original measure (Lee et al., 2022). Moreover, the ML–SIS shows improved interpretability because it provides a single score for each domain and a range of estimated scores, just like confidence intervals. Such advantages are achieved by the 50 parallel models, which were trained using the development set of data randomly sampled by a bootstrap approach (Lee et al., 2022). With the 50 estimated scores for each domain, the mean of the scores (i.e., the ML–SIS domain score) is a robust approximation of the raw domain score, and the range of the 50 scores represents an interval including the raw domain score. In a previous simulation, the ML–SIS scores—the mean scores of the 50 estimated scores—were consistent with those of the original SIS 3.0, including the correlations among the eight domains (one kind of convergent and divergent validity; Lee et al., 2022). Note that, despite the apparent complexity of ML–SIS scoring, the administration and scoring can be completed by an online system. Thus, the ML–SIS can still be feasible for clinical practice, just like other computerized self-report questionnaires.

The MMSE was used to assess the cognitive function of the participants in this study (Burns et al., 1998). The MMSE has 11 items that assess commonly affected domains of cognitive functions, such as orientation, memory, and language. The MMSE total score ranges from 0 to 30. A higher score indicates better cognitive function. The MMSE has good reliability and validity among people with stroke (Cumming et al., 2013).

The NIHSS was used to determine the participants’ severity of stroke (Brott et al., 1989). The NIHSS consists of 11 items, with a total score ranging from 0 to 42. A higher score represents a more severe stroke. The severity of stroke is classified into four levels: minor (total score <3 points), mild (4–6 points), moderate (7–15 points), and severe (≥16 points). The NIHSS has shown acceptable reliability and validity with people with stroke (Dewey et al., 1999; Goldstein & Samsa, 1997; Zandieh et al., 2012).

The BI was used to assess the participants’ overall level of disability (Mahoney & Barthel, 1965). The BI consists of 10 items assessing common self-care activities. The total score ranges from 0 to 100, and a higher score indicates less disability in self-care activities. The BI has demonstrated good reliability and validity among people with stroke (Hsueh et al., 2001, 2002).

Data Analysis

The comparability of ML–SIS and SIS 3.0 scores was examined using three indices: the coefficient of determinant (R ²), mean absolute error (MAE), and root-mean-square error (RMSE). R ²is the proportion of shared variance among the total variances. A higher R ² means a higher similarity of information provided by two scores. In general, R ² > .90 is considered to indicate excellent comparability.

We used the MAE and RMSE to demonstrate the magnitude of discrepancy between the eight domain scores of the ML–SIS and the SIS 3.0. Given that these domains consist of 4 to 10 items, both indices were acceptable if values were smaller than the range of an item (i.e., 4 points; Lee et al., 2022). MAE% and RMSE% were also calculated by dividing the MAE and RMSE by the mean domain scores to demonstrate the proportion of discrepancies in patients’ scores. However, because there is currently no consensus on the acceptable criteria for MAE% and RMSE%, we considered percentages smaller than 10% to be good (Smidt et al., 2002).

We examined the test–retest reliability of the ML–SIS with the intraclass correlation coefficient (ICC), using the two-way random model with the absolute agreement of single-score mode (Koo & Li, 2016). A higher ICC value indicates better consistency between scores from repeated assessments. In general, ICC values of .70 and higher are good and sufficient for group-level comparisons, and values of .90 and higher are excellent and sufficient for individual-level comparisons (Aaronson et al., 2002).

The random measurement error of the ML–SIS was examined using the standardized error of measurement (SEM). The SEM was estimated using the formula SEM = SD_pool × $\sqrt{1 - ICC}$ , where SD_pool was the standard deviation of scores from the first and second assessments combined (Norkin & White, 2016). To evaluate the magnitude of random measurement error, the SEM value was divided by the means of the participants’ scores on the first and second assessments (as SEM%). If SEM% was less than 10% of the mean, the scores were acceptable (Flansbjer et al., 2005).

Systematic bias was examined using Cohen’s d. Cohen’s d is a standardized score representing the magnitude of difference between the mean scores obtained from the first and second assessments relative to the pooled standard deviation of the two scores (Portney & Watkins, 2009). A larger d indicates a larger difference: d < 0.2 is trivial, d ≥ 0.2 is small, d ≥ 0.5 is moderate, and d ≥ 0.8 is large (Portney & Watkins, 2009). The paired t test was adopted to examine whether the differences in mean scores obtained from the first and second assessments were larger than standard errors. In this study, trivial and insignificant differences were expected because the participants’ health-related quality was assumed to be stable across the repeated assessments.

Results

The 263 Taiwanese participants had a mean age of 59.8 yr (Table 1), with more men (69.6%) than women, mild stroke severity (mean NIHSS score of 4.4), intact cognitive function (mean MMSE score of 26.2), and limited disability (mean BI score of 79.7). About half (54.8%) of the participants completed both assessments. The characteristics of the participants who completed the first and second assessments were similar to those of the participants who did not, except that their mean age was younger (58.3 yr vs. 61.1 yr, respectively; p = .042) and overall disability was milder (mean BI scores 85.5 vs. 72.7, respectively; p < .001).

Table 1.

Participant Characteristics

Characteristic	n (%)			p (Test–Retest vs. Missing)
Characteristic	Whole Sample (N = 263)	Test–Retest Sample (n = 144)	Missing (n = 119)	p (Test–Retest vs. Missing)
Age, M (SD)	59.8 (13.0)	58.3 (11.6)	61.1 (14.3)	.042*
Male	183 (69.6)	98 (68.1)	85 (71.4)	.544
Education level				.613
Below elementary school	84 (31.9)	46 (31.9)	38 (31.9)
Junior high school	43 (16.3)	24 (16.7)	19 (16.0)
Senior high school	85 (32.3)	50 (34.7)	35 (29.4)
Above college or university	51 (19.4)	24 (16.7)	27 (22.7)
MMSE	26.2 (3.6)	26.5 (3.4)	25.8 (3.9)	.155
BI	79.7 (24.8)	85.5 (21.2)	72.7 (27.1)	<.001*
NIHSS, M (SD)	4.4 (4.2)	4.5 (3.9)	4.3 (4.4)	.675
Stroke severity
Minor (< 4)	133 (50.6)	69 (47.9)	64 (53.8)
Mild (4–6)	54 (20.5)	32 (22.2)	22 (18.5)
Moderate (7–15)	71 (27.0)	42 (29.2)	29 (24.4)
Severe (≥ 16)	5 (1.9)	1 (0.7)	4 (3.4)

Note. BI = Barthel Index; MMSE = Mini-Mental State Examination; NIHSS = National Institutes of Health Stroke Scale.

*p < .05.

Overall, the ML–SIS yielded scores that were generally identical (R ²s = .87–.95; Table 2) to those of the SIS 3.0, except those of the Emotion domain. The discrepancy between the two scores in each domain (MAEs = 0.72–2.40; RMSEs = 1.08–3.28) was smaller than the score for an item (i.e., 5 points) and made up about 10% of the mean domain score (MAE%s = 0.02–0.09; RMSE%s = 0.04–0.14), except for the Emotion domain.

Table 2.

Comparability of Scores on the ML–SIS and the Original SIS 3.0

Domain	R ²	MAE	MAE%	RMSE	RMSE%
Strength	.95 (.94–.96)	0.72 (0.65–0.81)	0.07 (0.06–0.07)	1.08 (0.99–1.17)	0.10 (0.09–0.11)
Memory and Thinking	.92 (.90–.92)	0.95 (0.89–1.05)	0.03 (0.03–0.03)	1.46 (1.38–1.57)	0.05 (0.04–0.05)
Emotion a	.08 (.02–.12)	5.61 (5.47–5.81)	0.18 (0.18–0.19)	6.77 (6.62–6.98)	0.22 (0.22–0.23)
Communication	.90 (.89–.92)	0.75 (0.67–0.83)	0.02 (0.02–0.03)	1.28 (1.18–1.35)	0.04 (0.04–0.04)
ADLs/IADLs	.91 (.90–.92)	2.27 (2.12–2.46)	0.06 (0.06–0.07)	3.12 (2.95–3.31)	0.08 (0.08–0.09)
Mobility	.91 (.90–.92)	2.33 (2.13–2.46)	0.07 (0.06–0.07)	3.14 (2.92–3.32)	0.09 (0.08–0.09)
Hand Function	.95 (.94–.96)	1.28 (1.15–1.42)	0.09 (0.08–0.10)	1.86 (1.73–2.01)	0.14 (0.13–0.15)
Participation	.87 (.85–.88)	2.40 (2.27–2.54)	0.09 (0.08–0.10)	3.28 (3.12–3.44)	0.12 (0.12–0.13)

Note. N = 263. ADLs/IADLs = Activities of Daily Living/Instrumental Activities of Daily Living; ICC = intraclass correlation coefficient; MAE = mean absolute error; ML–SIS = machine learning–based Stroke Impact Scale; RMSE = root-mean-square error; R ² = coefficient of determination; SIS 3.0 = Stroke Impact Scale–Third Edition.

^aThe extremely low R ²s for this domain indicates that the emotion scores provided by the ML–SIS were different from and uncorrelated with those of the SIS 3.0. Thus, the emotion scores were not further examined.

Regarding psychometric performance in repeated assessments, the performance of the ML–SIS was similar to that of the SIS 3.0, including test–retest reliability (ICCs = .39–.87 vs. .46–.87, respectively; Table 3), random measurement error (SEMs = 2.37–5.16 vs. 2.40–5.56, and SEM% = 0.09–0.22 vs. 0.08–0.22, respectively), and systematic bias (Cohen’s ds = 0.00–0.12 vs. −0.06–0.14, respectively). The results did not include the Emotion scores because their comparability was unsatisfactory (a low R ² was found). We note that the ICC values for some domains of the ML–SIS and the SIS 3.0 were below the acceptable criterion of .70: the Memory and Thinking (.58 vs. .60), Communication (.39 vs. .46), and Participation (.60 vs. .55) domains.

Table 3.

Test–Retest Reliability of the ML-SIS and the SIS 3.0

Domain	First Assessment		Second Assessment		ICC (Min–Max)	SEM (Min–Max)	SEM% (Min–Max)	Cohen’s d (Min–Max)	p (Min–Max)
Domain	M (Min–Max)	SD (Min–Max)	M (Min–Max)	SD (Min–Max)	ICC (Min–Max)	SEM (Min–Max)	SEM% (Min–Max)	Cohen’s d (Min–Max)	p (Min–Max)
ML–SIS
Strength	10.8 (10.6–11.0)	4.4 (4.3–4.6)	11.1 (10.9–11.3)	4.1 (3.9–4.2)	.69 (.66–.71)	2.37 (2.31–2.44)	0.22 (0.21–0.23)	0.05 (0.03–0.07)	.313 (.142–.474)
Memory and Thinking	30.8 (30.6–31.0)	4.7 (4.5–4.9)	30.7 (30.5–31.0)	4.8 (4.6–5.0)	.58 (.54–.61)	3.07 (2.97–3.18)	0.10 (0.10–0.10)	0.00 (-0.03–0.01)	.895 (.631–1.000)
Emotion a	30.6 (30.2–30.9)	5.7 (5.5–5.9)	30.8 (30.6–31.0)	5.2 (4.9–5.3)	.45 (.41–.48)	4.06 (3.94–4.19)	0.13 (0.13–0.14)	0.03 (0.01–0.06)	.620 (.356–.841)
Communication	32.3 (32.1–32.5)	4.1 (3.9–4.3)	32.7 (32.5–32.9)	3.6 (3.4–3.7)	.39 (.35–.43)	2.99 (2.85–3.11)	0.09 (0.09–0.10)	0.08 (0.05–0.09)	.253 (.135–.409)
ADLs/IADLs	36.9 (36.4–37.3)	10.1 (9.7–10.4)	38.3 (37.9–38.7)	9.5 (9.1–9.7)	.84 (.83–.86)	3.89 (3.73–3.98)	0.10 (0.10–0.11)	0.10 (0.09–0.12)	.002 (.000–.005)
Mobility	34.2 (33.7–34.7)	9.4 (9.0–9.8)	35.7 (35.4–36.2)	8.6 (8.2–8.9)	.78 (.76–.80)	4.22 (4.04–4.38)	0.12 (0.12–0.13)	0.12 (0.10–0.14)	.002 (.000–.008)
Hand Function	13.3 (12.8–13.7)	7.4 (7.2–7.7)	13.7 (13.3–14.1)	7.5 (7.3–7.7)	.87 (.87–.88)	2.65 (2.58–2.73)	0.20 (0.19–0.21)	0.04 (0.02–0.06)	.176 (.035–.464)
Participation	26.0 (25.6–26.3)	8.0 (7.8–8.2)	27.0 (26.6–27.3)	8.3 (8.1–8.5)	.60 (.56–.63)	5.16 (5.02–5.28)	0.19 (0.19–0.20)	0.09 (0.07–0.10)	.106 (.050–.214)
SIS 3.0
Strength	10.6	4.6	10.9	4.3	.70	2.40	0.22	0.06	.331
Memory and Thinking	30.6	5.0	30.3	5.1	.60	2.92	0.10	−0.06	.407
Emotion	32.7	7.4	33.8	7.5	.68	4.05	0.12	0.15	.026
Communication	32.4	4.3	32.8	3.6	.46	2.51	0.08	0.10	.272
ADLs/IADLs	37.7	9.8	38.6	9.2	.84	3.66	0.10	0.10	.041
Mobility	34.4	10.0	35.7	8.9	.78	4.40	0.13	0.14	.013
Hand Function	13.2	8.1	13.5	8.2	.87	2.91	0.22	0.03	.432
Participation	27.0	8.2	27.4	8.6	.55	5.56	0.20	0.04	.575

Note. Min–max indicates the minimal and maximal scores provided by the 50 models of the ML–SIS that were developed to demonstrate the variations in estimated scores. The p value indicates the results of a paired t test. ADLs/IADLs = Activities of Daily Living/Instrumental Activities of Daily Living; ICC = intraclass correlation coefficient; ML–SIS = machine learning-based Stroke Impact Scale; SEM = standard error of measurement; SIS 3.0 = Stroke Impact Scale–Third Edition.

^aThe performance of the ML–SIS emotion scores could reasonably be ignored for their unsatisfactory comparability.

Discussion

The ML–SIS yielded scores that were very similar to those of the SIS 3.0, except for the Emotion domain. These findings indicate that the ML–SIS works well in an independent sample, regardless of the differences in culture between where it was original developed, the United States, and the current cross-validation in Taiwan. With the Emotion scores excluded, the ML–SIS seems to be a robust short-form measure that can provide scores that are valid and comparable to those of the original SIS 3.0. The ML–SIS can be a promising alternative for clinicians to achieve efficient assessment with scores comparable to those of the original SIS 3.0.

The Emotion domain was the only one that showed unsatisfactory comparability. This finding may have two explanations. First, the emotional responses of people with stroke may differ between Americans and Asians; this proposition may be partially supported by a previous study showing that “being respected” is recognized as a unique requirement for Asians–Taiwanese (Yao et al., 2002). Second, emotional status may be inconsistent between people with different stroke chronicity. Given that the ML–SIS was developed for people in a subacute stage (i.e., 90 days after stroke onset; Lee et al., 2022), it may not generalize to people with stroke in a chronic stage (about half of our participants were outpatients, but we could not separate them out because the time elapsed since stroke onset was not recorded; Chou et al., 2015). Regardless of the reasons for the negative results, the current findings demonstrate that the Emotion scores yielded by the ML–SIS are invalid. If the Emotion scores are needed, given that the ML–SIS includes all but one (“having nothing to look forward to”) of the Emotion items (Lee et al., 2022), users can further administer that item to obtain the Emotion score through the original scoring process. By doing so, prospective users can obtain Emotion scores for the SIS 3.0.

The test–retest reliabilities between the ML–SIS and the SIS 3.0 were very similar in all the domains, except for the Emotion domain. Test–retest performance between the two measures was also similar for the indices of random measurement (SEM values) and systematic bias (Cohen’s ds). The near-identical performances can be explained by the comparability of the ML–SIS scores to the original SIS 3.0 scores. In summary, the current findings suggest that the ML–SIS is as reliable as the original SIS 3.0. Thus, the findings lend support for the utility of the ML–SIS as an efficient alternative to the SIS 3.0.

The relatively low ICCs in some domains of the original SIS 3.0 may be a concern. For example, three of the eight domains (Memory and Thinking, Communication, and Participation) had ICCs lower than .70, which is a criterion for group-level comparisons (Aaronson et al., 2002). The findings may be explained by the nature of the participants. These participants were recruited from rehabilitation wards (subacute patients) and outpatient departments (chronic patients; Chou et al., 2015). Because the subacute patients’ quality of life might have increased over time because of the effects of rehabilitation and natural recovery, their scores on the second assessments would be higher than those on the first assessments and lead to low test–retest reliability (Bernhardt et al., 2017). However, given that the time since stroke was not recorded, we could not separate the participants who might have had improvement (subacute patients) from those who might have a stable quality of life (chronic patients). As a result, the current results for test–retest reliability may have been underestimated. Note that this study aimed to compare the test–retest reliability between the two measures, and the results showed that their test–retest reliabilities were similar. Thus, despite the limitation of the mixed sample, the current findings still support that the ML–SIS is as reliable as the SIS 3.0.

Interestingly, the ICC values for the SIS 3.0 were inconsistent with those from a previous study (Chou et al., 2015) that used the same dataset (current study, ICCs = .46–.87; previous study, ICCs = .67–.96). This inconsistency may be related to the sample size, the ICC models adopted, or both. Regarding the sample size, our sample was slightly larger than that in the previous study (144 vs. 121 people with stroke; Chou et al., 2015). In this study, we included only participants who had complete SIS 3.0 data for both the first and the second assessments, regardless of missing data for other variables, such as demographic characteristics. However, the previous study included participants only if they had complete data for all variables (Chou et al., 2015). As for the issue of ICC models, the two-way random model with the absolute agreement of the single score mode was used in this study; this model reduces the ICC values for any inconsistency in each pair of scores (Koo & Li, 2016). However, in the previous study, the adopted model was not specified (Chou et al., 2015). If the consistency mode was used in that study, higher ICC values are reasonable because that model targets the linear relationships but not the actual differences between the pairs of scores. Nevertheless, this issue does not affect the conclusions of the current study (Koo & Li, 2016).

Limitations

Some limitations may affect the generalizability of the current study. First, the participants were recruited through convenience sampling. Thus, the participants may not represent the whole population of people with stroke. Second, the characteristics of the participants in the current study differ from those of the development study on numerous elements, such as language (Mandarin Chinese vs. English) and chronicity (mixture of subacute and chronic patients vs. subacute patients exactly 90 days after stroke). Therefore, the causes of some differences in the ML–SIS’s psychometric properties could not be identified. Third, the data were collected about 15 yr ago. Accordingly, the findings may be slightly different from those in recent years. Fourth, the participants who completed the repeated assessments were younger and had milder disabilities than those who did not. Therefore, the current findings may have been overestimated. Fifth, this study used the Mandarin Chinese version of the SIS 3.0 items. As a result, the findings may not generalize to other versions.

Implications for Occupational Therapy Practice

This study has the following implications for occupational therapy practice:

▪ The ML–SIS can be used to improve the efficiency of clinical assessments, particularly for repeated assessments, because it is as reliable as the original SIS 3.0.

▪ ML seems to be a promising method to shorten assessments without sacrificing reliability.

Conclusion

Our findings suggest that the test–retest reliability, random measurement error, and systematic bias of the ML–SIS are similar to those of the SIS 3.0. Thus, the ML–SIS can be useful for clinicians to monitor the quality of life in the critical aspects of life for persons with stroke. Other important psychometric properties, such as responsiveness, should be examined in future studies.

Footnotes

Acknowledgments

We have no conflicts of interest to disclose. All authors agree with the stated authorship of and contributions to this article. This research was supported by the Ministry of Science and Technology (MOST 111-2811-B-002-066). Tzu-Yi Wu and Ching-Lin Hsieh contributed equally to this work and serve as corresponding authors.

References

Aaronson

Alonso

Burnam

Lohr

K. N.

Patrick

D. L.

Perrin

, & Stein

R. E.

(2002). Assessing health status and quality-of-life instruments: Attributes and review criteria. Quality of Life Research, 11, 193–205. https://doi.org/10.1023/A:1015291021312

Bernhardt

Hayward

K. S.

Kwakkel

Ward

N. S.

Wolf

S. L.

Borschmann

et al. Cramer, S. C. (2017). Agreed definitions and a shared vision for new standards in stroke recovery research: The Stroke Recovery and Rehabilitation Roundtable taskforce. International Journal of Stroke, 12, 444–450. https://doi.org/10.1177/1747493017711816

Brott

Adams

H. P.

Jr ., Olinger

C. P.

Marler

J. R.

Barsan

W. G.

Biller

, . . . Hertzberg

(1989). Measurements of acute cerebral infarction: A clinical examination scale. Stroke, 20, 864–870. https://doi.org/10.1161/01.STR.20.7.864

Burns

Brayne

, & Folstein

(1998). Key papers in geriatric psychiatry: Mini-Mental State: A practical method for grading the cognitive state of patients for the clinician. Folstein

Folstein

and McHugh

, Journal of Psychiatric Research, 1975, 12, 189–198. International Journal of Geriatric Psychiatry, 13, 285–294. https://doi.org/10.1002/(SICI)1099-1166(199805)13:5<285::AID-GPS753>3.0.CO;2-V

Chiang

H. Y.

Chen

P. T.

Lee

S. C.

Shieh

Y. J.

Hsueh

I. P.

, & Hsieh

C. L.

(2023). Test–retest reliability and responsiveness of the Computerized Adaptive Test of Social Functioning in persons with stroke. Archives of Physical Medicine and Rehabilitation, 104, 1432–1438. https://doi.org/10.1016/j.apmr.2023.03.017

Chou

C. Y.

Y. C.

, & Chiang

T. R.

(2015). Psychometric comparisons of four disease-specific health-related quality of life measures for stroke survivors. Clinical Rehabilitation, 29, 816–829. https://doi.org/10.1177/0269215514555137

Cumming

T. B.

Churilov

Linden

, & Bernhardt

(2013). Montreal Cognitive Assessment and Mini-Mental State Examination are both valid cognitive tools in stroke. Acta Neurologica Scandinavica, 128, 122–129. https://doi.org/10.1111/ane.12084

Dewey

H. M.

Donnan

G. A.

Freeman

E. J.

Sharples

C. M.

Macdonell

R. A.

McNeil

J. J.

, & Thrift

A. G.

(1999). Interrater reliability of the National Institutes of Health Stroke Scale: Rating by neurologists and nurses in a community-based stroke incidence study. Cerebrovascular Diseases, 9, 323–327. https://doi.org/10.1159/000016006

Duncan

P. W.

Bode

R. K.

Min Lai

, & Perera

; Glycine Antagonist in Neuroprotection Americans Investigators .(2003). Rasch analysis of a new stroke-specific outcome scale: The Stroke Impact Scale. Archives of Physical Medicine and Rehabilitation, 84, 950–963. https://doi.org/10.1016/S0003-9993(03)00035-2

10.

Duncan

P. W.

Lai

S. M.

Bode

R. K.

Perera

, & DeRosa

(2003). Stroke Impact Scale-16: A brief assessment of physical function. Neurology, 60, 291–296. https://doi.org/10.1212/01.WNL. 0000041493.65665.D6

11.

Duncan

P. W.

Wallace

Lai

S. M.

Johnson

Embretson

, & Laster

L. J.

(1999). The Stroke Impact Scale Version 2.0. Evaluation of reliability, validity, and sensitivity to change. Stroke, 30, 2131–2140. https://doi.org/10.1161/01.STR.30.10.2131

12.

Flansbjer

U.-B.

Holmbäck

A. M.

Downham

Patten

, & Lexell

(2005). Reliability of gait performance tests in men and women with hemiparesis after stroke. Journal of Rehabilitation Medicine, 37, 75–82. https://doi.org/10.1080/16501970410017215

13.

Goldstein

L. B.

, & Samsa

G. P.

(1997). Reliability of the National Institutes of Health Stroke Scale. Extension to non-neurologists in the context of a clinical trial. Stroke, 28, 307–310. https://doi.org/10.1161/01.STR.28.2.307

14.

Hamza

Nabilla

, & Loh

(2012). Evaluation of quality of life among stroke survivors: Linguistic validation of the Stroke Impact Scale (SIS) 3.0 in Hausa language. Journal of the Nigeria Society of Physiotherapy, 20, 52–59.

15.

Hsueh

I. P.

Lee

M. M.

, & Hsieh

C. L.

(2001). Psychometric characteristics of the Barthel Activities of Daily Living Index in stroke patients. Journal of the Formosan Medical Association, 100, 526–532.

16.

Hsueh

I. P.

Lin

J. H.

Jeng

J. S.

, & Hsieh

C. L.

(2002). Comparison of the psychometric characteristics of the Functional Independence Measure, 5 item Barthel Index, and 10 item Barthel Index in patients with stroke. Journal of Neurology, Neurosurgery, and Psychiatry, 73, 188–190. https://doi.org/10.1136/jnnp.73.2.188

17.

Jenkinson

Fitzpatrick

Crocker

, & Peters

(2013). The Stroke Impact Scale: Validation in a UK setting and development of a SIS short form and SIS index. Stroke, 44, 2532–2535. https://doi.org/10.1161/STROKEAHA.113.001847

18.

Koo

T. K.

, & Li

M. Y.

(2016). A guideline of selecting and reporting intraclass correlation coefficients for reliability research. Journal of Chiropractic Medicine, 15, 155–163. https://doi.org/10.1016/j.jcm.2016.02.012

19.

Lee

S. C.

Wang

Lin

G. H.

P. C.

Lee

Y. C.

Chou

C. Y.

et al. Hsieh, C. L. (2022). Development of a short-form Stroke Impact Scale using a machine learning algorithm in patients at subacute stage. American Journal of Occupational Therapy, 76, 7606205070. https://doi.org/10.5014/ajot.2022.049136

20.

Lin

K. C.

C. Y.

Hsieh

Y. W.

Chen

C. L.

, & Lee

P. C.

(2010). Psychometric comparisons of the Stroke Impact Scale 3.0 and Stroke-Specific Quality of Life Scale. Quality of Life Research, 19, 435–443. https://doi.org/10.1007/s11136-010-9597-5

21.

MacIsaac

Ali

Peters

English

Rodgers

Jenkinson

et al. Quinn, T. J. ; VISTA Collaboration. (2016). Derivation and validation of a modified short form of the Stroke Impact Scale. Journal of the American Heart Association, 5, e003108. https://doi.org/10.1161/JAHA.115.003108

22.

Mahoney

F. I.

, & Barthel

D. W.

(1965). Functional evaluation: The Barthel Index. Maryland State Medical Journal, 14, 61–65.

23.

Marx

R. G.

Menezes

Horovitz

Jones

E. C.

, & Warren

R. F.

(2003). A comparison of two time intervals for test-retest reliability of health status instruments. Journal of Clinical Epidemiology, 56, 730–735. https://doi.org/10.1016/S0895-4356(03)00084-2

24.

Mohammad

A. H.

Al-Sadat

Siew Yim

, & Chinna

(2014). Reliability and validity of the Nigerian (Hausa) version of the Stroke Impact Scale (SIS) 3.0 index. BioMed Research International, 2014, 302097. https://doi.org/10.1155/2014/302097

25.

Norkin

C. C.

, & White

D. J.

(2016). Measurement of joint motion: A guide to goniometry. F. A. Davis.

26.

Portney

, & Watkins

(2009). Foundations of clinical research: Applications to practice. Prentice Hall.

27.

Smidt

van der Windt

D. A.

Assendelft

W. J.

Mourits

A. J.

Devillé

W. L.

de Winter

A. F.

, & Bouter

L. M.

(2002). Interobserver reproducibility of the assessment of severity of complaints, grip strength, and pressure pain threshold in patients with lateral epicondylitis. Archives of Physical Medicine and Rehabilitation, 83, 1145–1150. https://doi.org/10.1053/apmr.2002.33728

28.

Yao

Chung

C. W.

C. F.

, & Wang

J. D.

(2002). Development and verification of validity and reliability of the WHOQOL-BREF Taiwan version. Journal of the Formosan Medical Association, 101, 342–351.

29.

Zandieh

Kahaki

Z. Z.

Sadeghian

Pourashraf

Parviz

Ghaffarpour

, & Ghabaee

(2012). The underlying factor structure of National Institutes of Health Stroke scale: An exploratory factor analysis. International Journal of Neuroscience, 122, 140–144. https://doi.org/10.3109/00207454.2011.633721