Abstract
Telerehabilitation interventions (i.e., the delivery of rehabilitation via an internet-connected telecommunication platform) are increasingly being used for a wide range of purposes (Charvet et al., 2017; Cotton et al., 2017; Cottrell et al., 2017; Seidman et al., 2017; Theodoros et al., 2016), including stroke rehabilitation (Chen et al., 2015; Linder et al., 2015). The growth of telerehabilitation is driven by the pressing need to provide care for patients who cannot access traditional in-clinic rehabilitation because they lack the resources (e.g., have limited insurance coverage; Pastora-Bernal et al., 2017) or live in rural areas (Cason, 2009). With this growth in telerehabilitation comes a concurrent need to develop methods to enable therapists to obtain patient evaluation data via telecommunication. These data are needed to appropriately plan and progress the rehabilitation program and to measure outcomes. However, very few standardized methods of tele-evaluation are available (Richardson et al., 2017; Truter et al., 2014; Ward et al., 2013). The purpose of this study was to test an objective evaluation method of assessing a stroke patient’s level of arm movement impairment, using the Kinect sensor (Microsoft Corp., Redmond, WA) to facilitate tele-evaluation.
The Kinect sensor is a low-cost motion tracking device (Zhang, 2012) that can measure upper limb movements (Hawi et al., 2014). Many telerehabilitation systems already use the Kinect sensor to detect and encourage limb movements (Bao et al., 2013; Brokaw et al., 2015; Chang et al., 2011; Lange et al., 2011; Lee, 2013; Levac et al., 2015; Seo, Arun Kumar, et al., 2016), thereby making the addition of a Kinect-based assessment feasible. A therapist can review Kinect-based assessment scores without the need for in-clinic, in-person observation, which thus increases access to therapy.
Whether the Kinect can provide valid assessment data for tele-evaluation of the upper limb is unknown. To date, the psychometric properties of Kinect kinematic measures have been both supported and questioned; specifically, Kinect measures such as reachable workspace volume and arm movement velocity have been shown to have strong test–retest reliability, with intraclass correlations (ICCs) of .96–.98 (Lowes et al., 2013). However, the Kinect is limited in measurement accuracy, and its agreement with measurements of active range of motion using hand-held goniometry is low (mean ICC = .48; Hawi et al., 2014). Compared with research-grade three-dimensional motion capture systems, the Kinect has been shown to have an approximately 10° error in estimating shoulder elevation and elbow angles (Bonnechére et al., 2014; Choppin & Wheat, 2013; Hawi et al., 2014; Kitsunezaki et al., 2013; Kurillo et al., 2013; Seo, Fathi, et al., 2016), and this error is higher for other joint angles (Seo, Fathi, et al., 2016). The Kinect has also been reported to overestimate movement velocity by 18%–23% (Lowes et al., 2013). Despite this limited accuracy, Kinect measures correlate with measures obtained from a research-grade 3-D motion capture system (Kurillo et al., 2013), can distinguish people with neuromuscular disorders from those without such disorders (Han et al., 2013; Lowes et al., 2013), and correlate with standard clinical assessments (Han et al., 2013, 2015; Lowes et al., 2013). Therefore, although the Kinect may not be sufficiently accurate for precise motion analysis, it may be sufficient to classify people into categories of movement ability.
Classification systems are widely used for many diagnoses. For example, in stroke rehabilitation, classification categories have been used to describe the overall independence of rehabilitation patients (Functional Independence Staging; Stineman et al., 2003), the amount of walking assistance needed (Functional Ambulation Categories; Mehrholz et al., 2007), or the level of poststroke arm motor impairment (Fugl–Meyer Assessment for the upper extremity (FMA–UE; Fugl-Meyer et al., 1975). To be specific, on the FMA–UE, ratings of 0, 1, and 2 are given for no, partial, and near-normal movement ability, respectively. These ordinal ratings are, in essence, categories of patient ability. Using the Kinect to determine an item rating category may require less measurement precision and may thus take advantage of Kinect technology.
Two studies have used rating scales in examining the use of Kinect for categorizing the upper limb movements of patients with stroke (Kim et al., 2016; Olesh et al., 2014). In both studies, participants with stroke performed several FMA–UE movement items while being tracked with the Kinect sensor. Their performance was also rated by therapists. Good agreement was observed between the Kinect and therapist ratings (65%–87% agreement in Kim et al. [2016] and a strong linear relationship in Olesh et al. [2014]). Together, these studies provide preliminary evidence that Kinect measures can reliably assign rating categories to arm movements. However, because the Kinect cannot track finger motions, movements that involve object interaction, or movements out of its sight (e.g., with the hand behind the trunk), the Kinect can replicate only 10–13 of the assessment’s 33 items, and in-person observation is still needed for the remaining part of the assessment to formulate treatment targets and interpret treatment response (Woodbury et al., 2013). Thus, the burden of patient travel for assessment and therapist workload is not reduced.
The purpose of this study was to determine feasibility of using the Kinect sensor to perform a standard clinical assessment in its entirety to objectively classify level of arm gross motor impairment and enable tele-evaluation. Specifically, we used the Mallet system (de Luna Cabrai et al., 2015) to classify stroke survivors with the Kinect. The Mallet system is a standardized method commonly used to assess shoulder movement ability in people with brachial plexus injury. It consists of five movement items—shoulder abduction, shoulder external rotation, hand to head, hand to back, and hand to mouth—and each item is given an ordinal rating (Figure 1). The Mallet is not traditionally a stroke rehabilitation assessment. However, the five motions that comprise the items of this assessment are impaired in people with stroke (Beebe & Lang, 2008; Cirstea & Levin, 2000), and they are critical for recovery of functional tasks such as reaching high, opening a door, brushing hair, adjusting a zipper in the back, and eating. Because the Mallet is based on kinematic measurements and the movements tested do not require object handling, it is suitable for measurement with the Kinect. In this study, we tested the idea that a computerized assessment using the Kinect can categorize a patient’s upper limb impairment level as well as a therapist can. If this is the case, the results will add support for the use of Kinect assessment in telerehabilitation.
Method
Development of a Computerized Assessment
We developed a computerized assessment program that used the Kinect sensor by means of the OpenNI Software Development Kit (SDK; http://structure.io/openni) and a custom-developed C++ program. This computer program prompted the user to perform each of the five tasks one by one while it obtained the user’s upper body position data in a three-dimensional space from the Kinect sensor. Specifically, the Kinect sensor detected the positions of the user’s hand, wrist, elbow, shoulder, center of torso, and head. Using these joint position data, the program computed joint angles and distances that were relevant to the scoring criteria (described in the next paragraphs) and determined the user’s score according to these criteria. The program monitored the user’s body positions for 10 s. The program displayed a mirror view of the user to clearly show the upper limb being monitored. The best score observed during the 10-s period was used as the score for a task. On completion of a task, the program prompted the user to perform the next task. On completion of all tasks, a summary of scores for all five tasks were displayed on the computer screen.
The scoring criteria (see Figure 1) were constructed to be consistent with the description of the Mallet classification (de Luna Cabrai et al., 2015). Specifically, on Task 1, shoulder abduction, a shoulder abduction angle <30° resulted in a score of I, a shoulder abduction angle of 30°–90° resulted in a score of II, and a shoulder abduction angle of >90° resulted in a score of III. For Task 2, shoulder external rotation, shoulder external rotation angles of <0°, <20°, and ≥20° were assigned scores of I, II, and III, respectively. A shoulder external rotation score was assigned only if the distance between the elbow and the torso was less than the width of the torso.
For the remaining three tasks, an exact angle or distance criteria that the Kinect can follow was not available from the Mallet classification description. Thus, we developed specific criteria to best match the pictorial description of the scoring criteria (Figure 1). Specifically, for the hand-to-head task (Task 3), a distance from the hand to the head of >100%, 100%–60%, and <60% of the forearm length led to scores of I, II, and III, respectively. For the hand-to-back task (Task 4), elbow angles >100°, 100°–70°, and <70° resulted in scores of I, II, and III, respectively. The distance between the hand and torso had to be less than the torso width to obtain any score. For the hand-to-mouth task (Task 5), shoulder abduction angles >90°, 90°–40°, and <40° resulted in scores of I, II, and III, respectively. A shoulder abduction score was not assigned unless the distance between the hand and head was less than the width of the torso.
We obtained position data for the hand, wrist, elbow, shoulder, head, and torso from the Kinect OpenNI skeleton tracking library. These position data were used to compute vectors for body segments. Then we computed shoulder abduction angle as the angle between the upper arm and the vertical axis, projected to the frontal plane. The shoulder external rotation angle represented a rotation of the forearm about the axis of the upper arm according to the literature (Soechting et al., 1995; Wu et al., 2005). The elbow angle was the angle between the upper arm and the forearm.
Evaluation of the Computerized Assessment
Participants.
Two experiments were conducted. Participants in the first experiment were 10 healthy adults with no history of neurological or orthopedic disorders affecting the upper limbs (age range = 20–37 yr). They used the right arm for evaluation. Participants in the second experiment were 7 stroke survivors with upper limb hemiparesis (age range = 34–81 yr) whose stroke had occurred 5–58 mo before the study. Two had the left side affected, and 5 had the right side affected. Their impairment level ranged from severe to mild, with FMA–UE scores of 12–55 out of 60 (Woodbury et al., 2013), representing a range of upper limb impairment levels. They used their affected arm for evaluation.
None of the participants had a language barrier or cognitive impairment that precluded them from following instructions. Evaluations were conducted at the Medical University of South Carolina and the University of Wisconsin–Milwaukee. The study protocols were approved by the institutional review boards of these universities. All participants provided informed consent.
Procedure.
We conducted two experiments to evaluate scoring by the computerized assessment. In the first experiment, healthy participants intentionally aimed to achieve each score for each task (see Figure 1). Thus, healthy participants performed the computerized assessment three times to obtain scores of I, II, and III on each of the five tasks. To help healthy participants move their arm correctly to achieve the target score, pictorial descriptions of the target range (see Figure 1) and a mirror view of the participant were shown on the computer screen. Review of the video recordings confirmed that participants moved their arm in a way that corresponded to the target (similar to the gold standard, described next).
The second experiment involved participants with stroke performing the computerized assessment. Each participant performed the computerized assessment one time and were instructed to try to achieve the movement in each task to the best of their ability. Their movements were simultaneously visually assessed by two occupational therapy students who had been trained in scoring this assessment. For the purposes of this article, we call the process of scoring on the basis of immediate visual assessment typical procedure, because it is how assessments are commonly scored in clinical practice. The participants’ movements were also videotaped. The video recordings were later reviewed by the two occupational therapy students to collaboratively determine the score. For the purposes of this article, we call the process of scoring by reviewing a video the gold standard, because scoring from a video offers ample opportunity to review, discuss, and collaboratively determine a score. For all experiments, the Kinect sensor was placed on a table approximately 1.8 m in front of the participant.
Analysis.
We evaluated the accuracy of the computerized scores as follows. For the first experiment, we compared the scores assigned by the computer program with the target scores. The percentage accuracy was computed as the percentage of trials for which the computer program gave the same score as the target score. In addition, we obtained Cohen’s κ coefficients for all tasks combined and for each task individually. Cohen’s κ coefficient represents interrater agreement for categorical items (Hallgren, 2012); a κ of 1 indicates perfect agreement; .81–.99, almost perfect agreement; .61–.80, substantial agreement; .41–.60, moderate agreement; .21–.40, fair agreement; and .01–.20, slight agreement. A κ of 0 indicates agreement by chance, and negative values indicate less-than-chance agreement (Viera & Garrett, 2005). In this study, Cohen’s κ coefficient represented the extent of agreement with the gold standard or the scores that were collaboratively determined on the basis of the video recordings.
For the second experiment, we compared the computerized scores with the gold standard (collaborative visual assessment using video recordings) using Cohen’s κ. Likewise, we compared the scores resulting from typical practice (based on the immediate visual assessment) with the gold standard. We then compared the κ coefficients for the computerized assessment and typical practice to evaluate whether the Kinect-based computerized assessment scored stroke survivors as accurately as occurred in typical practice.
Results
The results of the first experiment are shown in Figure 2A. Scores assigned by the computer program matched the target scores 85% of the time (the three large circles along the diagonal) for all five tasks combined. For the other almost 15%, the computer program gave scores different than the target scores. For example, for 3% of the trials, the computer program scored the movement as II, when the target score was I (see Figure 2A). The percentage matching for Tasks 1–5 was 97%, 93%, 93%, 60%, and 83%, respectively. Cohen’s κ coefficient for all tasks combined was .78.
The results of the second experiment are shown in Figures 2B and 2C for the computerized assessment and typical practice, respectively. The computer program gave the same scores as the gold standard 77% of the time (the three large circles along the diagonal in Figure 2B) for all five tasks combined. The percentages matching for Tasks 1–5 were 100%, 86%, 71%, 43%, and 86%, respectively. Typical practice (immediate visual assessment) matched the gold standard in scoring 82% of the time (the three large circles along the diagonal in Figure 2C) for all five tasks combined. The percentages matching for Tasks 1–5 were 83%, 75%, 83%, 92%, and 75%, respectively. Cohen’s κ coefficient for all tasks combined was .66 for the computerized assessment and .72 for typical practice (averaged across the two human assessors). These coefficient values are within the range for substantial agreement.
Cohen’s κ coefficients for individual tasks are shown in Figure 3. For the first experiment with healthy participants, Cohen’s κ coefficients for the computerized assessment indicated substantial to almost perfect agreement for all tasks except Task 4 (Figure 3A). This trend for a low level of agreement for Task 4 was seen again for the computerized assessment with participants with stroke (Figure 3B, gray bars). For Task 3, almost perfect agreement was obtained when the computerized assessment scored healthy participants (Figure 3A), whereas moderate agreement was obtained when it scored participants with stroke (Figure 3B, gray bar). Typical practice showed substantial to almost perfect agreement for all tasks (Figure 3B, white bars).
Cohen’s κ coefficients for individual tasks for (A) the first experiment with healthy participants and (B) the second experiment with participants with stroke for the computerized assessment (gray bars) and typical practice (white bars).
Discussion
For all tasks combined, substantial agreement with the gold standard was achieved for the computerized assessment for both healthy participants (κ = .78) and participants with stroke (κ = .66). That level of agreement was similar to that for typical practice, in which evaluators scored stroke participants’ movements immediately as they saw the movements (κ = .72). These κ values are comparable with those reported in the literature (Bae et al., 2003). Thus, the data suggest that the computerized assessment using the Kinect can categorize arm gross motor impairment as accurately as can typical practice.
The level of agreement differed by task for the computerized assessment, whereas the level of agreement was similar for all tasks for typical practice; specifically, the computerized assessment worked well for the shoulder abduction, shoulder external rotation, and hand-to-mouth tasks (Tasks 1, 2, and 5). The substantial to perfect agreement for Tasks 1 and 5 may be related to the Kinect’s relatively small error in estimating the shoulder abduction angle (mean error of 7° reported in Seo, Fathi, et al., 2016). Substantial to almost perfect agreement was obtained for Task 2, although a previous study reported a large mean error of 36° for shoulder rotation angle determined by the Kinect (Seo, Fathi, et al., 2016a). In this study, the elbow remained flexed during the shoulder external rotation task, whereas the previous study involved a pointing movement using a straight arm with complete elbow extension. Thus, in this study, error associated with estimating the plane of the arm (composed of the wrist, elbow, and shoulder points) and shoulder rotation angle was minimal.
The hand-to-head task (Task 3) was associated with almost perfect and moderate agreement for healthy participants and participants with stroke, respectively. Healthy participants typically swung their hand to the side in the frontal plane before putting their hand behind the head, which provided a clear view of the hand approaching the head from the side. However, participants with stroke kept their hand in front of their chest until they moved their hand to the back of the head, which gave little clearance between the hand and head for the Kinect to detect. This difference in movement pattern between healthy participants and participants with stroke may be the reason for the difference in level of agreement on the computerized assessment between the two participant groups for Task 3.
For the hand-to-back task (Task 4), only fair and slight agreement was obtained for healthy participants and participants with stroke, respectively. It appears that the hand and wrist positions were not well detected when they were behind and obstructed by the participant’s body. The discrepancies with the computerized assessment for Tasks 3 and 4 accounted for 75% of all discrepancies (see Figure 2B). A possible solution for this technical issue includes replacing the general skeletal tracking algorithm in the Kinect’s OpenNI SDK with a custom algorithm to account for stroke-specific and task-specific movement patterns.
Implications for Occupational Therapy Practice
The results of this study have the following implications for occupational therapy practice:
A computerized assessment using the Kinect sensor has potential for use in tele-evaluation, reducing the burden of patient travel and increasing accessibility. The presence of a human rater is not required, thus reducing therapist workload.
The computerized assessment may be integrated into Kinect-based rehabilitation systems (Bao et al., 2013; Brokaw et al., 2015; Chang et al., 2011; Lange et al., 2011; Lee, 2013; Levac et al., 2015; Seo, Arun Kumar, et al., 2016) to complete telerehabilitation. Patients may participate in therapeutic activity and be evaluated periodically, in the comfort of their own home, while therapists review patients’ adherence to the home program and computerized assessment scores to monitor progress over time and modify the therapy program accordingly, without the need for clinic space.
Limitations
To be comparable to typical practice, use of Kinect-based assessment in practice may be limited to scoring movements that have at least substantial agreement with the gold standard. As mentioned earlier, the Mallet classification is not typically used for patients with stroke, and future work may be needed to address this limitation. A drawback of the current Kinect system is that it monitors only body segments, not other objects. Thus, items in other standard clinical assessments that require object grasping and manipulation cannot be assessed using the Kinect (Kim et al., 2016; Olesh et al., 2014) unless additional sensors are used on the patient’s body or objects (Otten et al., 2015), which hampers feasibility.
The number of participants tested in this study was small. Our future research includes a larger study to assess the generalizability of the findings, integration of the computerized assessment into a telerehabilitation system, and evaluation of the system’s usability in a complete telerehabilitation system.
Conclusion
Computerized assessment using the Kinect sensor was able to place upper limb movements in the gross motor categories as accurately as typical practice (immediate visual assessment). Scoring agreement with the gold standard was substantial to perfect except when the hand and wrist were hidden behind the body. We obtained these findings with stroke patients with a wide range of upper limb impairment levels and with healthy adults aiming to achieve prescribed scores. The results suggest that the computerized gross motor categorization using the Kinect sensor has promise in facilitating tele-evaluation and being integrated into telerehabilitation programs to enhance patients’ access to health care.
Footnotes
Acknowledgment
This research was supported by the National Institutes of Health (Grants 5R24HD065688-04 and P20GM109040). Part of this work was presented at the 37th Annual Meeting of the American Society of Biomechanics, Omaha, Nebraska, and the Medical University of South Carolina 2015 Research Day student poster presentation. The authors declare no conflict of interest.