Abstract
Client-centered care, which is well established in occupational therapy, is a key component of health care reform that is intended to promote positive health outcomes, increase client satisfaction, and ensure efficient use of resources (Constand et al., 2014; Mroz et al., 2015). Client-centered care requires interprofessional provider teams to consider the needs and priorities of clients and their family members when creating and implementing a care plan. Team members must establish effective, efficient, and collaborative ways to communicate with families and with each other throughout the process of care (Amtmann et al., 2011; Brewer et al., 2014; Chiarello, 2017). As part of interprofessional care teams that work to improve client participation in occupation, occupational therapists are poised to reinforce client-centered care, beginning with initial care planning (Law et al., 1995; Mroz et al., 2015; Stoffel et al., 2017).
Barriers to client-centered care include time and resource constraints as well as the perceived power differential between providers and clients (Whalley Hammell, 2015). Technology-based assessments may help to minimize these barriers by providing diverse clients with options for how they engage in planning and monitoring participation-focused care for their child (Wang et al., 2016). Some assessments supporting client-centered practice, such as the Canadian Occupational Performance Measure, now provide an option for electronic administration (Donnelly et al., 2017)
The Participation and Environment Measure (PEM; Coster et al., 2012) is a participation-focused assessment that allows caregivers to evaluate their child’s participation (e.g., frequency and involvement in mealtime and whether change is desired), their perceptions of environmental supports and barriers to their child’s participation (e.g., sensory qualities in the home), and current strategies for supporting their child’s participation at home, at daycare or preschool, and in the community. It also can be administered electronically. The Young Children’s PEM (YC–PEM; Khetani, Graham, et al., 2015) is a psychometrically sound measure that yields a range of information relevant when developing a young child’s occupational profile. The YC–PEM does not, however, automatically yield a care plan that caregivers can use to facilitate the collaborative care process and improve their young child’s participation in occupations.
The PEM+ is an electronic health (e-health) application that addresses the need for flexible options to improve family-engaged care planning and promote client participation. It was designed with caregiver and provider input (Khetani, Cliff, et al., 2015, Khetani et al., 2017) to help caregivers use their YC–PEM results to determine activities that are a high priority for change, generate specific and meaningful goals, and identify strategies for goal attainment. The PEM+ was found to be accessible to caregivers when delivered by phone, which prompted the development and testing of a web-based version (Khetani et al., 2017).
The purpose of this study was to examine the usability of the PEM+—specifically, its technical effectiveness, ease of learning, ease of use, and user satisfaction. Usability is the degree to which products are effective, efficient, and satisfying for people to use (Jokela et al., 2003; Zhou et al., 2017). Usability testing is a crucial step when developing user-centered web applications so that design and layout issues that affect user navigation can be resolved before the tool’s feasibility and preliminary effects are tested (O’Malley et al., 2014). Study results informed revisions to the PEM+ prototype before further testing.
Method
Participants
The PEM+ is designed for use by caregivers of children 0–5 yr old whose participation may differ by age group (e.g., 0–3 yr and 4–5 yr; Dunst et al., 2002; Law et al., 2012). In accordance with recommended usability testing guidelines (Nielsen, 2012), we established a target recruitment of 5 or more caregivers of children 0–3 yr old for this first usability study.
Participants were 6 caregivers recruited through convenience sampling from two programs, one in Chicago and one in Denver. Chicago participants were recruited from Blue Bird Therapeutic Day School, an early childhood educational setting that includes intensive group and individual therapies. Denver participants were recruited from Rocky Mountain Human Services, the largest early intervention program in the Denver metropolitan area. Caregivers were eligible if they (1) were at least 18 yr old; (2) read, wrote, and spoke English; (3) had broadband internet access; and (4) had a child between 0 and 3 yr old who was accessing services for developmental delay. Ethics approval was obtained before recruitment and data collection (June–August 2017).
Intervention
Caregivers first completed the YC–PEM home and community sections online to evaluate their child’s current participation and whether they desired it to change. When caregivers desired change, they were asked to identify their current strategies for enabling the child’s participation in activities of that type. Once caregivers had evaluated their child’s participation, they evaluated environmental supports and barriers to participation in a setting. After completing the YC–PEM, caregivers were directed to complete one iteration of PEM+ online. The current PEM+ has five steps:
Rank, sort, or otherwise specify a priority activity;
Develop a goal for the child, including the type and amount of desired change in priority activity, using the YC–PEM and their own words;
Consider the relevance of existing strategies for goal attainment in the priority activity;
Consider environmental strategies for goal attainment; and
Review and confirm care plan content (Table 1).
PEM+ Approach for Remote Care Planning
Note. PEM = Participation and Environment Measure.
Measure
To test PEM+’s technical effectiveness, we used web analytics to record PEM+ completion rate and time. In addition, caregivers were prompted to complete a set of eight user tasks that corresponded with key technical features of the application (e.g., log out and back in) during PEM+ completion. They were then asked to rate each task on a 7-point scale ranging from 1 (very difficult) to 7 (very easy). To evaluate ease of use, ease of learning, and user satisfaction, caregivers completed the Usefulness, Satisfaction, and Ease of Use (USE) questionnaire (Lund, 2001) after completing one iteration of the PEM+. Caregivers rated 22 questions (e.g., “I quickly became skillful with it,” “It meets my needs”) on a modified 6-point scale ranging from 1 (strongly disagree) to 6 (strongly agree).
Data Collection
Participants were approached by study staff in person or via phone (N = 13). Eligible participants who consented to enroll completed an electronic demographic questionnaire to gather information about the caregiver and child (e.g., age, gender, race/ethnicity, household income and size, reason for service use). Caregivers then completed the YC–PEM home and community sections to access an online YC–PEM report with a web link to begin PEM+ (N = 6). After PEM+ completion, participants were routed into Research Electronic Data Capture (REDCap), a secure web application for capturing and managing data (Harris et al., 2009). Users completed the USE questionnaire in REDCap and were mailed $20.00 gift cards for their participation.
Data Analysis
Demographic and usability data collected via REDCap and an administrative dashboard were exported for analysis. Demographic characteristics and YC–PEM summary scores were reported for each user. To address the main study question, we derived usability estimates for the total study sample, according to technical effectiveness (PEM+ range, mean percentage completion rate, mean completion time, mean task difficulty rating) and USE summary scores (mean and range) for ease of use, ease of learning, and user satisfaction.
Results
Most caregivers were female, of Hispanic or Latino ethnicity, between 30 and 39 yr old, and residing with multiple children. Race, education, and income levels varied (Table 2).
Participant Characteristics
Note. Pseudonyms were used to protect caregiver confidentiality. Desire change is the percentage of activities to which parents responded “Yes, desire change” when completing the Young Children’s Participation and Environment Measure. ASD = autism spectrum disorder; NA = not available.
All caregivers completed the PEM+ in its entirety. Caregivers completed one PEM+ iteration in 13.6 min on average (range = 3.3–22.0). Users identified priority activities in both the home and community settings. Caregivers who focused PEM+ on home participation identified “getting rest” as their priority activity. Caregivers who focused PEM+ on community participation selected a broader range of activities, including “shopping/errands,” “religious/spiritual gatherings,” and “community attractions.” The PEM+ sort feature was primarily used (67%) to identify a priority activity. Technical effectiveness of PEM+ features was rated an average of 6.1 (range = 5.7–6.3 out of 7.0). Tasks with the lowest ratings were accessing activity examples, knowing when work was saved, and accessing the parent-to-parent strategy exchange. Also, caregivers varied in how much assistance they needed to complete the PEM+ (e.g., logging in and out, reminders to continue writing their plan). Mean ease of use, ease of learning, and user satisfaction were rated 4.4, 5.4, and 4.2 (of 6.0), respectively (Table 3).
PEM+ Usability
Note. Technical effectiveness was rated on a 7-point scale ranging from 1 (very difficult) to 7 (very easy). USE Questionnaire items were rated on a 6-point scale ranging from 1 (strongly disagree) to 6 (strongly agree). PEM+ = Participation and Environment Measure Plus; Q&A = question and answer; USE = Usefulness, Satisfaction, and Ease of Use.
Discussion
Client-centered care is reinforced by offering clients accessible options for designing and monitoring care, thus encouraging intentional therapeutic interactions between providers and clients (Taylor, 2008). Technology-based tools with evidence of usability may reduce provider burden by improving the ease of gathering information about client need and related contextual factors (Wang et al., 2016) as well as by expanding provider reach for care processes such as care planning, a common task that typically requires significant investment of provider and client resources (Bailey, 2017).
In this study, all caregivers could complete the PEM+ care planning tool online on the first attempt and did so in less than half the time, on average, than it took to complete phone-based interviews using PEM+ (Khetani et al., 2017) or other interview-based assessments that are standard in usual care, such as the Routines Based Interview (McWilliam, 2009). PEM+ harnesses technology so that caregivers can have greater flexibility to clarify their care priorities, define goals, and identify and share strategies for goal attainment with service providers to facilitate the collaborative care planning process. This utility holds promise for reducing provider burden, in turn increasing the opportunity for more effective care planning. In addition, technology may help providers address known disparities in client access and use of services such as rehabilitation (Pew Research Center, 2018).
Similar to prior study phases, the web-based mode of administration was accessible to users for care planning across multiple settings (e.g., home, community) and types of activities (e.g., getting rest, community attractions). Caregivers who focused on improving their child’s home participation consistently prioritized improving their child’s participation in getting rest, a potentially relevant intervention target for families and a relatively new focus of occupation (American Occupational Therapy Association, 2014). Although caregivers used all three options to identify a high-priority activity in the first step (see Table 1), sorting continued to be a preferred option among the majority of caregivers who also had a higher rate of desire for change. This trend warrants future study with larger samples to examine the effect of caregivers’ rate of desire for change (i.e., high vs. low percentage of desire for change) on PEM+ completion trends.
Several trends in PEM+ survey feedback have informed optimizations to the electronic prototype before further testing. First, we found variability in how difficult it was for users to access examples of activities within each YC–PEM activity category. This finding prompted the development and integration of activity illustrations into the PEM+ prototype to visually prompt and focus users on the types of activities that commonly fall within a broader YC–PEM activity category of focus.
Second, we found variability in users knowing when their work was saved. This prompted clarification in wording to guide users once an iteration was completed, as well as the development of a PDF storage function so research staff could provide users with their reports on request.
Third, users varied with respect to their ability to access a parent-to-parent strategy exchange feature during the third and fourth steps of PEM+ (see Table 1) when identifying ways to support goal attainment. This point of user feedback has, in turn, prompted two considerations. First, we changed the structure of PEM+ to automatically show parents the strategy exchange in the fourth step. Second, accessible educational materials could help introduce caregivers to the task of identifying and appraising child and environmentally focused strategies for goal attainment before they access a strategy exchange.
Evidence regarding the impact of environment on young children’s participation is growing (Di Marino et al., 2017), but environment is a multidimensional concept and a relatively new focus of participation-focused intervention (Anaby et al., 2015; Chiarello, 2017). Although users complete an environmental assessment before beginning PEM+ (Khetani, Graham, et al., 2015), they may need support to apply this type of assessment information when developing a care plan. Last, users varied in how much assistance they needed to get through PEM+. This finding prompted the optimization of automated text and email messages based on user interactions with PEM+.
Limitations and Future Directions
Several of our study limitations informed optimizations to the PEM+ prototype or study design before beginning larger scale testing that is underway. Future testing will aim to recruit male and female caregivers across a broader age range. PEM+ is designed for caregivers of children ages 0–5 yr, but we were only successful in enrolling caregivers of children aged 0–3 yr. We have gained recruitment site clearance and ethics approval to expand inclusion criteria to caregivers of children 0–5 yr old. Second, caregivers from one of two recruitment sites experienced intermittent technical glitches, resulting in disruptions that may have resulted in lower ease of use and satisfaction ratings. A simplified electronic route was developed so that users can complete YC–PEM and PEM+ using a single web link. Last, a number of survey options for usability testing need further evaluation (Zhou et al., 2017). We adjusted the USE questionnaire to offer users a 7-point response scale. Because the USE has no established cutoff score, we will use Likert item responses and open-ended feedback.
Implications for Occupational Therapy Practice
This study has the following implications for occupational therapy practice:
E-health tools with evidence of usability, such as PEM+, can decrease response burden and promote family-engaged and participation-focused care processes.
The YC–PEM offers electronic administration so that caregivers can flexibly contribute their expertise when planning their child’s care.
The PEM+ is designed to extend the use of the YC–PEM by helping caregivers to specify and communicate their priority concerns and ideas for goal attainment.
Study results suggest that clients can navigate this e-health tool. Further optimization and testing are warranted.
Conclusion
This single-arm usability trial established the PEM+ as a promising tool for use by caregivers to support engagement in participation-focused care plan development. The results informed optimizations before further testing of the PEM+.
Footnotes
Acknowledgments
Jessica M. Jarvis and Andrea Gurga share first authorship of this article. This work was supported by the American Occupational Therapy Foundation (AOTFIRG15KHETANI) and the University of Illinois at Chicago. The use of Research Electronic Data Capture was supported by the National Center for Advancing Translational Sciences, National Institutes of Health (UL1TR002003). This study was registered at 
(NCT 03820011).
We thank our colleagues at Rocky Mountain Human Services and Blue Bird Therapeutic Day School for their feedback on PEM+ and their support of caregiver recruitment as part of their current workflow. We thank students enrolled in a graduate elective focused on pediatric processes of care for their critical review of a draft of this article. We dedicate this work to Gary Bedell, Wendy Coster, and Mary Law, whose pioneering interdisciplinary work and mentorship have shaped opportunities for our rewarding research collaboration.