Abstract
The study findings highlight the value and support the development and use of complex telehealth interventions, which have the potential to improve remote access to occupational therapy for clients living with chronic neurological conditions.
Complex telehealth interventions for neurological recovery have the potential to facilitate remote delivery of occupational therapy services and improve access for people living with complex, chronic neurological conditions (Coscia et al., 2019; Laver & Osborne, 2022). Although there is no clear delineation between “simple” and “complex” interventions, characteristics of complexity include (1) interacting components, (2) specialized patient or provider behaviors, (3) a heterogeneous target audience, (4) multiple outcomes, and (5) adaptability of the intervention (Craig et al., 2008). When these characteristics are applied to neurological rehabilitation, interventions such as brain and muscle biofeedback, serious gaming, and functional electrical stimulation could be described as complex. Interventions become increasingly complex when delivered at home, through virtual therapy sessions, using multiple technologies, and with remote data monitoring or any combination of these. The potential of complex telehealth interventions is underrealized, in part because they are particularly challenging to develop and implement (Stephenson et al., 2022).
The body of literature evaluating complex telehealth interventions for neurological conditions, including stroke, is growing, although many are small trials with relatively weak evidence (English et al., 2022). Even systematic reviews are inconclusive about the efficacy of telerehabilitation, in part because of the variety of interventions and outcome variables (Laver et al., 2020). Although efficacy outcomes are vital to intervention research, early examination of implementation outcomes is also important. Even highly efficacious complex interventions may not be routinely adopted if there is a poor fit among the user, intervention, and delivery context. Therefore, concurrent assessment of efficacy and implementation outcomes has been proposed to support the successful embedding of interventions into service delivery (Curran et al., 2012; English et al., 2022). The evaluation of fit between intervention and delivery context before implementation is referred to as pre-implementation (Kerkhoff et al., 2022).
Facilitators of and barriers to implementing stroke rehabilitation interventions have been well described in the literature, and the field is moving toward developing targeted implementation strategies that promote the sustainable uptake of innovative treatments into occupational therapy practice (Juckett et al., 2020; Stephenson et al., 2022). An important step is evaluating factors known to influence the uptake of complex telehealth interventions (Skidmore et al., 2014). For instance, Broens et al. (2007) identified five determinants of telemedicine success: technology, acceptance, financing, organization, and policy and legislation. The current pre-implementation study examines the first two of these among stroke survivors. The technology determinant includes support, training, usability, and quality. Acceptance (or acceptability) includes the attitudes and perceptions of stakeholders toward the telehealth technology, including its content, complexity, comfort, and other dimensions (Broens et al., 2007; Proctor et al., 2011). The remaining determinants are not examined in this article because they require methods that leverage different stakeholder perspectives (e.g., cost analysis, organizational assessment, policy analysis).
In this study, we evaluated the usability and acceptability of an electromyography (EMG) biofeedback telehealth intervention for poststroke upper extremity motor recovery (Tele-REINVENT; Marin-Pardo et al., 2022). This study builds on previous pilot work examining the acceptability of Tele-REINVENT through thematic analysis of interviews with stroke survivors (N = 4; Donnelly et al., 2023) by using a mixed- methods approach, including validated measures of usability and acceptability, with a larger sample of users. The purpose of this pre-implementation work is to support (1) identification of the fit between EMG biofeedback and telerehabilitation for stroke survivors, (2) optimization of EMG biofeedback interventions, and, more broadly, (3) other efforts to develop complex telerehabilitation interventions for routine clinical practice.
Method
The data reported here were collected between November 2021 and November 2022 as part of a pilot and feasibility trial of Tele-REINVENT in which participants used the system at home for 6 wk through a combination of therapist-guided Zoom sessions and independent sessions. Here, we briefly describe the trial procedures to contextualize the usability and acceptability findings. See the Appendix for additional protocol details and Marin-Pardo et al. (2022) for the technical specifications and preliminary efficacy findings. All study procedures were conducted observing the ethical standards of the University of Southern California Institutional Review Board and the revised (2013) Helsinki Declaration.
Participants
We sought to recruit stroke survivors in the chronic phase of recovery (>6 mo since onset) with moderate to severe upper extremity hemiparesis, which we operationalized as limited wrist extension and a minimum level of extensor EMG activity (i.e., hold 30% of a prerecorded maximum voluntary contraction for 10 4-s trials). English- and Spanish-speaking individuals were eligible. Significant vision loss, receptive aphasia, hand contractures, cognitive impairment (Montreal Cognitive Assessment score < 20), secondary neurological disease, current use of antispasticity medication, and current therapy treatment targeting arm function were exclusion criteria.
We identified prospective participants via a stroke research database, flyers, and word of mouth. Rolling recruitment was completed from October 2021 to October 2022. Prospective participants completed a phone screening for criteria that could be assessed verbally, followed by an in-person screening to test muscle activity and cognition.
Protocol
Participants returned to the laboratory within 1 wk of the screening for baseline clinical and physiological testing. Additionally, we oriented them to Tele- REINVENT (Figure 1) and sent it home with them to complete 15 1-hr sessions over 3 wk. They returned to the laboratory for midpoint testing and were invited to use Tele-REINVENT at home for 3 more weeks if they completed 80% or more of the at-home sessions and demonstrated improvement on at least one measure (Action Research Arm Test [Lyle, 1981], Fugl-Meyer Assessment Upper Extremity [Fugl-Meyer et al., 1975], or EMG task). Finally, participants returned to the laboratory for posttesting.
Tele-REINVENT: (A) The Tele-REINVENT interface: laptop computer, EMG sensors, disposable electrodes, and positioning aids. (B) A Tele-REINVENT user playing skeeball. The sensors are connected to the laptop via a USB cable. (C) Tele-REINVENT game selection screen. (D) Instruction screen for Planet Jump, explaining the objective and which movements correspond with game actions. (E) Skeeball: Wrist extension rolls a ball toward a target. (F) Planet Jump: Wrist extension jumps the astronaut over obstacles, and wrist flexion prevents collisions. (G) Blinko: Wrist flexion and extension move a disk across a vertical game board until it falls and lands on a point value.
Outcome Measures
Demographic data were collected during the first visit. During the posttesting visit, participants completed validated questionnaires that measured perceived usability and acceptability and a semistructured interview.
Quantitative Outcomes
Usability
Usability is dependent on interactions among users, the product, tasks, and contexts. Perceived usability was assessed quantitatively using the Post-Study System Usability Questionnaire, Version 3 (PSSUQ; Sauro & Lewis, 2016). The PSSUQ is a reliable 16-item questionnaire that covers five usability characteristics (quick completion of work, ease of learning, high-quality documentation and online information, functional adequacy, and rapid acquisition of productivity) and is usable with small samples. Each item is scored on a 7-point Likert scale (1 = strongly agree; 7 = strongly disagree). Lower scores indicate greater satisfaction. The PSSUQ produces an overall score and three subscale scores: System Quality (SysQual), Information Quality (InfoQual), and Interface Quality.
Acceptability
Acceptability is the stakeholders’ perception that the components of an intervention are agreeable or satisfactory. Acceptability was assessed quantitatively using the extended Unified Theory of Acceptance and Use of Technology (UTAUT) model (Chao, 2019), which contains 31 items and measures eight constructs that predict technology acceptance (effort expectancy, performance expectancy, perceived enjoyment, satisfaction, trust, mobile self-efficacy, perceived risk, and behavioral intention; see the Appendix for definitions). Each item is scored on a 5-point Likert scale (1 = strongly disagree; 5 = strongly agree). Higher scores indicate greater acceptance.
Qualitative Outcomes
Participants completed a semistructured interview about their experience with Tele-REINVENT (e.g., perceived benefits, preferences, facilitators of and barriers to use). Interviews were conducted in a private room by a member of the research team (Miranda Rennie Donnelly, Octavio Marin-Pardo, or Coralie Phanord) and typically lasted between 20 and 40 min. Questions were intentionally broad and experiential to evoke stories and honest appraisals rather than specifically about usability and acceptability (see Donnelly et al., 2023, for the interview guide). The interviewer did not view the participant’s questionnaire responses before conducting the interview. This allowed participants to share what they perceived to be salient rather than prompting comments on specific features or questionnaire constructs.
Interviews were audio recorded, transcribed verbatim, and checked by a second reviewer. Participant quotes in this article were edited only to add brackets or punctuation for clarity or ellipses for brevity or to remove identifiable information. The “accents” of non-native English speakers (i.e., those who spoke English as a second language) were not edited. Quotes from Spanish-speaking participants were translated.
Data Analysis and Interpretation
All statistical analyses were run in R (Version 3.6.3; R Core Team, 2020). Descriptive analysis was completed for demographic data. An overall PSSUQ score, subscale scores, and mean scores were calculated for each item. An overall UTAUT score, means for each construct, and mean scores for each item were calculated. Multiple linear regression tested whether personal characteristics (i.e., age, education, time since stroke, severity of impairment) contributed to acceptability and usability. Interview transcripts were analyzed using PSSUQ and UTAUT constructs as predefined codes. The complete transcripts were analyzed; however, only segments that addressed the constructs were coded. Initial coding was conducted independently by two reviewers (Miranda Rennie Donnelly and Aisha Abdullah) using a word processor. Codes were then compared line by line, and consensus was reached to establish the final dataset. Concurrent triangulation was used to corroborate the mixed-methods findings; quantitative and qualitative data were collected concurrently, analyzed and reported separately, and integrated during interpretation.
Results
Eleven participants used Tele-REINVENT at home for up to 6 wk. Two of the 11 participants completed 3 wk or less of the protocol but were included in this analysis because their data are relevant to the constructs of interest. Demographic data are summarized in Table 1. Despite efforts to recruit women, all participants were male, which is a challenge noted in the stroke literature (Carcel & Reeves, 2021).
Participant Demographics (N = 11)
Note. FMA = Fugl-Meyer Assessment of the Upper Extremity; ARAT = Action Research Arm Test.
Usability
Quantitative Outcomes
The PSSUQ overall and subscale scores show that Tele-REINVENT is highly rated for usability (Table 2; scale norms are listed for reference). Multiple linear regression was used to test whether personal characteristics significantly predicted usability scores. None of the factors were significantly associated with usability (Table 3).
PSSUQ Scores by Item
Note. Scores closer to 1 indicate greater usability. Items with mean scores ≥3 (indicating areas for improvement) are in bold. The PSSUQ produces an overall score and three subscales. Scale norms are provided for reference. PSSUQ = Post-Study System Usability Questionnaire Version 3.
aFrom Sauro and Lewis (2016).
Summary Statistics for the Relationships Between Personal Characteristics and Perceived Usability and Acceptability
Note. Summary statistics from multiple linear regression to test associations between personal characteristics and usability and acceptability scores. ARAT = Action Research Arm Test (baseline); FMA = Fugl Meyer Assessment of the Upper Extremity (baseline).
We also calculated the mean for each PSSUQ item to identify areas of relative strength and weakness in Tele-REINVENT’s usability. Because a score of 1 indicates greater usability, we defined items with a score of 3 or more as high-priority improvement areas to optimize Tele-REINVENT’s usability. This threshold yielded two InfoQual items (see boldface items in Table 2).
Qualitative Outcomes
System quality
Most participants described a learning process to become productive with the system. They noted that sensor setup and calibration were initially tedious, and the system did not respond well to their attempted movements. The intervention involves various complex tasks (e.g., calibrating EMG sensors), but by the end of the study, most perceived the system as simple, even basic. Some attributed these changes to learning how the system processes movements: Learning the hand movements of what was most effective . . . moving the, either the ball or . . . making the little figures jump. . . . Sometimes I would have overt movements to where instead of keeping the wrist just going at a steady up and down, I would sometimes tense up and turn to the sides. So, when I turned the hand to the sides, I think that’s when I wasn’t getting good reaction with the games.
As participants became comfortable with the intervention, the provider remotely adjusted game pace and sensitivity settings. For example, as 1 participant mastered skeeball, [The researchers] were able to speed it up a little more . . . so that was a good adjustment. . . . The slower speed was definitely good for the first couple weeks because you’re still trying to adjust to the game. But then after, you know, you start to—you attempt to progress at the game and it’s like “Okay, you know, let’s do this a little quicker now.”
Information quality
Substantial evidence showed that the researchers—rather than the system notifications, video tutorials, or manual—were the primary information sources and facilitators of usability. Participants noted that when they encountered issues, the system did not offer clear guidance on the nature of the issue or how to proceed. Early in the trial, the research team was particularly involved in supporting participants as they acclimated to the system, and the information provided during those interactions was perceived as high quality. Only 1 participant reported using the manual, and none mentioned the tutorial videos.
Interface quality
Participants wanted more durable sensor cases with built-in placement cues and identified features they expected the games to have, such as the ability to resume a level, view a previous high score between sessions, or slow the speed of Planet Jump for more processing time.
Acceptability
Quantitative Outcomes
The UTAUT scores indicate that Tele-REINVENT is highly acceptable among stroke survivors (Table 4). Multiple linear regression tested whether personal characteristics significantly predicted acceptability scores. None of the factors were significantly associated with acceptability (see Table 3).
Mean UTAUT Item and Construct Scores
Note. Scores closer to 5 indicate greater acceptability. Items with mean scores < 4 (indicating areas for improvement) are in bold. The UTAUT produces one overall score, and we calculated the mean for each construct. UTAUT = Unified Theory of Acceptance and Use of Technology Model.
aThe M and SD in this row are for the items indented beneath this construct.
We also calculated the mean of each UTAUT item to identify specific barriers to and facilitators of acceptability. Because a score of 5 indicates greater acceptability, we defined items with scores less than 4 as high-priority areas of improvement to optimize Tele-REINVENT’s acceptability. This threshold yielded 7 items for improvement across five subscales (see boldfaced items in Table 4).
Qualitative Outcomes
Effort expectancy
Tele-REINVENT was perceived as easy to use. However, as previously noted, there was a learning curve: “The first time was a little difficult because I had to get used to how to connect everything . . . because it’s . . . a system that I’m not used to,” but “as time went on, it got easier and easier.” Even participants who had limited experience using computers found Tele-REINVENT more effortless to use than they had expected.
Performance expectancy
Participants perceived that Tele-REINVENT supported their rehabilitation goals. Tele-REINVENT encouraged them to focus on controlling the movement and relaxation of their affected arm. Most participants perceived some functional improvement (e.g., grasping handles and knobs more easily) and felt that it elicited greater awareness of their arm.
Perceived enjoyment
Tele-REINVENT was widely enjoyed, particularly when the games were perceived to be an appropriate challenge. Earning a better score or leveling up were particularly motivating and led to enjoyment. Blinko is probability based, and although some participants enjoyed the animations and movement practice, the scoring did not reflect performance, making it less motivating. Technical issues and a limited selection of games somewhat detracted from enjoyment.
Satisfaction
Participants were overall satisfied with trying a novel rehabilitation technology, playing games using their affected arm, interacting with the researchers, and perceived improvements. However, the challenges of using such a system were dissatisfying, such as when sensors were placed incorrectly and “you have to start the program over again.”
Trust
Participants did not discuss trust. We hypothesize that the informed consent process contributed to high levels of trust among participants.
Perceived risk
There were no interview data about privacy risks. However, 1 participant perceived troubleshooting system issues as risky: “You don’t know what might happen if . . . you don’t know what you’re doing. . . . I’d rather just stick to what I know and just take notes.”
Self-efficacy
The self-efficacy findings were twofold. First, participants exuded pride as they described their game achievements. Many were eager to share that they could now use the system mostly independently, even when technical issues arose. It was evident that Tele-REINVENT provided opportunities to skillfully use their affected arm and participate in rehabilitation without hands-on assistance. By contrast, participants sometimes doubted whether they had the skills needed to use the system. For example, some experienced sessions in which they felt they could not control their arm and doubted their own abilities: “I was getting frustrated because no matter how I told the brain to send the signal, it didn’t do it the first few times.” Most participants emphasized that virtual support from the researchers helped build confidence and was a valuable part of the intervention.
Behavioral intention
Most participants said they would continue using Tele-REINVENT if given the opportunity. Alternatively, 1 participant who stopped participating in the study after 3 wk described his intention to not use Tele-REINVENT, noting that it takes “practice, practice, practice. . . . I really didn’t have the chance to practice like I wanted to. . . . I just don’t have time.” He also added that glitches were a deterrent: “To me, it’s just like, you know what? I’ll do this later . . . and of course, technology is technology, but if it was a smoother session, it would encourage more activity.” Although there was interest in continued use of Tele-REINVENT, there are factors that should be addressed to support such use.
Discussion
We evaluated the usability and acceptability of an EMG biofeedback telehealth intervention among stroke survivors to optimize its design, because both usability and acceptability of technology are determinants of successful telemedicine implementation (Broens et al., 2007). We triangulated the mixed- methods findings to identify opportunities to improve Tele-REINVENT and generate broader insights for developing complex telerehabilitation interventions.
Tele-REINVENT scored favorably on technology usability. The PSSUQ is not typically interpreted using normative scores; however, in the absence of PSSUQ data from a similar intervention, the norms (listed in Table 3) can be a reference (Sauro & Lewis, 2016). Notably, PSSUQ scores for Tele-REINVENT exceeded the norms. We identified InfoQual items that were rated less favorably than the other items. The PSSUQ authors report that InfoQual tends to perform worse than the other subscales, in part because creating informative, actionable error messages is challenging (Sauro & Lewis, 2016). Despite their relative weakness, the InfoQual scores for Tele-REINVENT are remarkably positive, even though participants reported challenges addressing errors. The qualitative data suggest that much of the information that facilitated the system’s usability came from interactions with the researchers, not from the system itself. We hypothesize that the disparity between high PSSUQ scores and critiques in the interview data may be due to different framings of the system. Although the software system did not give clear messages to guide recovery from mistakes, participants had easy and quick access to support, which may have led to interpretation of the system as a broader entity encompassing the human and technological components of the intervention. This framing is useful to consider in complex telehealth interventions, in which the patient, provider, technology, context, and tasks each play a role in the intervention.
Similarly, qualitative findings regarding acceptability were nearly inseparable from interactions with the team. The qualitative data suggest that the providers reduced technical barriers that affected effort expectancy, perceived enjoyment, self-efficacy, and satisfaction. For example, guided sessions developed participants’ confidence and skills. This can be seen quantitatively as well; the item measuring confidence using a novel system had a high score, but the item measuring independent use of the system had a lower score. Researcher interactions were a relative strength of the intervention, even beyond the benefits of technical support. However, the level of support provided in this study may not be feasible in clinical environments. Therefore, as implementation efforts progress, repeat usability testing in a realistic clinical environment is warranted.
We also identified relationships spanning constructs on both questionnaires. For example, when participants encountered issues they could not solve (a reflection of SysQual and InfoQual), they perceived the system as more difficult to use (effort expectancy) and sometimes questioned whether the issue was caused by their own lack of skill (self-efficacy). Given that both perceived usability and acceptability are widely accepted as determinants of success of technology implementation (Broens et al., 2007; Chao, 2019), it is not surprising that the usability and acceptability findings are closely linked.
Personal characteristics can influence the uptake of complex telehealth interventions. These associations are important to identify so that interventions can be adapted for diverse groups. For example, one study found that educated and resourced young-old White women (i.e., those ages 65–74 yr) were more likely to use health technology than their peers (Hung et al., 2020), revealing disparities that can be addressed methodically. In our study, the personal characteristics we tested had no significant explanatory value for acceptability or usability. These null results suggest that Tele-REINVENT may be acceptable and usable regardless of age, education, time since stroke, and stroke severity. This is promising because older adults with chronic, severe stroke and less education have more gaps in access to care (Ayala et al., 2018; Teasell et al., 2018).
In addition to identifying relative strengths and weaknesses of Tele-REINVENT, we extracted insights to more broadly support other pre-implementation efforts to develop complex telerehabilitation interventions for use in clinical practice. We list here the five lessons learned, along with examples of how we have applied (or plan to apply) each and this study’s findings to optimize Tele-REINVENT. Include stakeholders early and often in the design and implementation planning of complex telehealth interventions. There has been a rise in literature describing the benefits of cocreation and pragmatic tools for engaging stakeholders (Kerkhoff et al., 2022; Pérez Jolles et al., 2022). Although not discussed here, our group often engages stroke survivors in the design of interventions; for example, they have consulted on wearable sensor design, game development, and alpha testing. Although stroke survivors were participants in this study, we prioritized using knowledge from lived experience to influence enhancement of the system. Identify and enable the distinct value of the provider as both a clinician and a facilitator of technology use. The intervention provider is an important component of complex telehealth intervention delivery, as seen in our study and in others (Caughlin et al., 2020; Neibling et al., 2021). However, skilled therapy providers can quickly become technical support when using technology-based interventions. We identified that the primary way that providers add value to Tele-REINVENT is by personalizing the intervention. For example, participants benefited from feedback about positioning and technique and adjustments to game settings. We strategized ways to enable that value and reduce barriers. For example, in the version of Tele-REINVENT tested in this study, game settings were adjusted in configuration files. We have now developed a clinician interface for all setting adjustments. We also identified that troubleshooting consumes valuable participant–provider contact time, so we also focused on reducing the impact of technical issues, which is discussed more later. Differentiate user error from system error. We observed that uncertainty about the source of a problem can quickly led to frustration, low self-efficacy, and dissatisfaction. However, when participants knew they had triggered an error (e.g., by placing a sensor incorrectly), the problem was manageable, and they could act. Similarly, if they realized that a game was not reading their EMG signal, they could restart the software. Providing clear, actionable error messages equips users with information they need to proceed. On the basis of our findings, we developed simple, jargon-free, and actionable error messages for common issues. Develop implementation strategies to address common barriers. Technical issues are ubiquitous in rehabilitation technology, whether an intervention is elegant or bootstrapped; however, in isolation they do not have a detrimental effect on acceptability (Caughlin et al., 2020; Donnelly et al., 2023; Neibling et al., 2021). In fact, implementation strategies that address inevitable barriers can enhance usability and acceptability. In addition to a brief in-person orientation and virtual sessions, we provided a printed user manual with simple wording and photos. We also embedded tutorial videos into the system. In the interview, only 1 participant mentioned the manual, and no one mentioned the videos. However, they all reported technical challenges. This provided evidence that the implementation strategies we developed were not the right fit for our users—that is, they clearly preferred personal (live) consultations versus written guides or asynchronous videos. We intend to codesign more effective and sustainable implementation strategies with both stroke survivor and clinician stakeholders, given that live consultations and training may not be feasible in real therapy contexts. Embed progress tracking and social features. Gamified experiences can be motivating, and this has been leveraged in many rehabilitation technologies. However, game mechanisms and preferences may differ by target population. For example, of the three games included in this version of Tele-REINVENT, most participants found the levels in Planet Jump and scores in skeeball to be the most motivating. Blinko users emphasized that they were motivated by using their affected arm, not by their score. Our previous preliminary work examining the acceptability of Tele-REINVENT (Donnelly et al., 2023) and the current study suggest the following considerations for future game development: (1) Preserve achievements between sessions (i.e., all-time high scores), (2) use scoring schemes that directly reflect performance, (3) gamify multiple rehabilitation goals, and (4) develop options to compete (e.g., leaderboard) and communicate with other users.
Limitations and Future Directions
This study used inexpensive, user-focused, and replicable methods to assess pre-implementation usability and acceptability. This pragmatic approach allowed quick iterations of intervention design. Despite the benefits, these methods do not capture the full depth and breadth of usability and acceptability, so additional testing in future stages of this intervention is warranted. Additionally, both the small sample size and the exclusion criteria may limit the generalizability of the usability and acceptability findings. We gained valuable perspective from stroke survivors, and future work is underway to assess pre-implementation outcomes among other stakeholders, including therapists, because they are important decision makers and influencers of rehabilitation technology uptake (Chen & Bode, 2011). Finally, future work should examine financing, organizational, and policy factors to support future implementation of REINVENT (Broens et al., 2007). This pre-implementation work will support embedding the system into routine clinical practice in the future.
Implications for Occupational Therapy Practice
This study has the following implications for occupational therapy practice: ▪ Complex telehealth interventions can facilitate remote therapy services and facilitate access for people with chronic neurological conditions. ▪ Occupational therapy practitioners can influence the acceptability and usability of complex telehealth interventions.
Conclusion
This study contributes to an emerging body of literature examining the use of complex telehealth interventions with survivors of neurological injury. Our findings and lessons learned underscore the value of occupational therapists in delivering telehealth interventions and opportunities for telehealth to enable remote access to therapy. Implementation outcomes, such as acceptability and usability, are often reserved for late-stage intervention research; however, evaluating these outcomes pre-implementation can ensure that complex interventions meet the needs and requirements of their intended users and delivery context. This is important because complex telehealth interventions can open doors for survivors of neurological injury who face barriers to accessing occupational therapy and would benefit from technology-enabled therapy at home.