Usability and Reliability of the Pediatric Evaluation of Disability Inventory–Patient Reported Outcome (PEDI-PRO) Software Interface for Youth With Developmental Disabilities (DD)

PURPOSE: Assessment developers recognize that technology can enhance the accessibility of educational tests for youth with disabilities (Beddow, 2012), but this vision has yet to be fully realized for self-reports (patient reported outcome measures (PROMs)), in occupational therapy practice. Most PROMs, which are predominately paper based, present many barriers to completion for transition-age youth with developmental disabilities (DD). Paper based PROMs can be cumbersome or impossible to modify to reduce the visual-perceptual, motor, and cognitive demands required for completion. The PEDI-PRO is a PROM designed for transition age youth with DD ages 14-22; it measures perceived performance of Daily Activities, Social/Cognitive, and Mobility functional skills. Previous research established that youth with DD interpreted the initial PEDI-PRO items in the intended manner (Kramer & Schwartz, 2017). However, additional research is needed to evaluate the usability and accessibility of these items when administered via an accessible software interface. Research Aim 1 (A1) Evaluate the usability of the PEDI-PRO software in clinical contexts. Research Aim 2 (A2) Establish the reliability and acceptability of the PEDI-PRO user interface with transition age youth with DD.

DESIGN: A1: We used survey methodology to evaluate clinical usability. We recruited clinicians across the US through professional networks and a database of former research participants. A total of 14 clinicians (OT =12, PT = 2) administered the PEDI-PRO in diverse practice contexts (School-based= 9, outpatient= 4, other setting= 1) to 39 youth with DD ages 14-22.

A2: Test-retest design with a convenience sample of youth with DD recruited through schools and agencies in a large metropolitan area, and database of former research participants. 54 youth completed the PEDI-PRO: M age= 19.7 yrs, sd = 1.7 yrs; 60% male; 64% Caucasian; 64% intellectual disability.

METHOD: A1: Clinicians administered the PEDI-PRO to three youth, and then completed the System Usability Survey (SUS) (Bangor et al., 2008). SUS scores range from 0-100, with higher scores indicating greater usability; the accepted industry standard for usability systems is a mean score of >68.

A2: Research staff administered the PEDI-PRO software two times, 1-2 weeks apart. Items were counterbalanced across testing occasion to avoid order effects. Youth also completed a questionnaire about the features of the PEDI-PRO. We used a Rasch rating scale model to obtain scores from the original Likert response scale, then used ICC consistency model (3,1) (Trevethan, 2017) to investigate the reliability of scores across testing occasions; ICCs > .75 indicate fair reliability.

RESULTS: A1: SUS rating M = 84 (sd= 11.68), exceeding industry standards.

A2: Results reported by PEDI-PRO domain. Daily Activities: ICC= .83. Social/Cognitive: ICC= .85. Mobility: ICC = .81. Almost all youth reported they “liked” the accessibility features, including interface images, button sounds, item read-aloud audio, and rating category choices with images: Range = 84%-98%; M = 88.8% (sd = 5.1%).

CONCLUSION: Clinicians and youth with DD found the interface easy to use and accessible. Our reliability results provide initial evidence that the PEDI-PRO user interface produces relatively stable scores during a time period in which change in functional performance is not expected to occur.

IMPACT: PROMs that incorporate technology, such as the PEDI-PRO, can provide access to PROMs for youth with DD and a range of functional needs. Access to PROMs can help address the current underrepresentation of this population in the evaluation of rehabilitation and healthcare outcomes and support client-centered practice.

References

Beddow, P. A. (2012). Accessibility theory for enhancing the validity of test results for students with special needs. International Journal of Disability, Development and Education, 59(1), 97-111. https://doi.org/10.1080/1034912X.2012.654966

Kramer, J. & Schwartz*, A. (2017). Refining the Pediatric Evaluation of Disability Inventory–Patient-Reported Outcome (PEDI-PRO) item candidates: Interpretation of a self-reported outcome measure of functional performance by young people with neurodevelopmental disabilities. Developmental Medicine and Child Neurology, 59(10), 1083-1088. doi: 10.1111/dmcn.13482.

Bangor, A., Kortum, P. T., & Miller, J. T. (2008). An empirical evaluation of the system usability scale. Intl. Journal of Human–Computer Interaction, 24(6), 574-594. https://doi.org/10.1080/10447310802205776

Trevethan, R. (2017). Intraclass correlation coefficients: clearing the air, extending some cautions, and making some requests. Health Services and Outcomes Research Methodology, 17(2), 127-143. https://doi.org/10.1007/s10742-016-0156-6