Abstract
Health care and educational legislation and policy require that clinicians demonstrate, using measurement and report of outcomes, accountability for services rendered. Clinical algorithms have been developed and are used by various health care professionals to assist with hypothesis generation and systematic clinical reasoning; however, they do not explicitly guide measurement of outcomes as part of the reasoning process. Schaaf and colleagues developed the Data-Driven Decision Making (DDDM) process to address the greater need for outcome measurement, systematically support decision making, target intervention more precisely, and measure and document outcomes. This article describes the application of the DDDM process with a child with ASD who received occupational therapy using Ayres Sensory Integration®.
Keywords
Contemporary health care legislation requires clinicians to document and report on patient outcomes to demonstrate accountability for services rendered (Burke & Gitlin, 2012; King, Wright, & Russell, 2011), but the move to evidence-based, outcome-oriented practice has been slow and inconsistent (King et al., 2011). Commonly reported barriers include time, lack of knowledge and skill, and reluctance to use measures that are not perceived to meet client need. Clinicians have indicated concern that a move to data-based practices will result in “‘recipe-oriented’ practice and may feel there are contradictions between using standardized measures and providing client- or family-centered care” (King et al., 2011, p. 2664).
To address clinicians’ reluctance, clinical algorithms have been developed and are used by various health care professionals to assist with hypothesis generation and systematic clinical reasoning (McEwen, 2009). For example, in physical therapy the Hypothesis-Oriented Algorithm for Clinicians and the Hypothesis-Oriented Pediatric Focused Algorithm are used to guide novice clinicians and students in a systematic, stepwise approach to patient care (Kenyon, 2013). These algorithms provide a systematic approach to guide clinical reasoning and are clinically useful. One limitation, however, is that they do not explicitly guide measurement of outcomes as part of the reasoning process.
Ideally, outcome data should support decision making and allow evaluation of intervention effectiveness; this information, in turn, helps target intervention more precisely, which then provides a higher standard of care (Frolek Clark, 2010). Moreover, outcome measurement in clinical practice can enhance third-party payment and provide a foundation for controlled trials of intervention efficacy, effectiveness studies, and translation and dissemination of interventions to foster sustainability (Burke & Gitlin, 2012). The need for evidence and data for occupational therapy practice for children with autism spectrum disorder (ASD) has reached a particularly critical level given shrinking funds and “attempts to restrict access to and payment for occupational therapy practice utilizing sensory integration” (Clark, 2012, para. 1).
To address the need for greater outcome measurement in occupational therapy, Schaaf and colleagues (Schaaf, 2015; Schaaf & Mailloux, 2015) developed the Data-Driven Decision Making (DDDM) process. One unique feature of DDDM is that it emphasizes data use as standard practice in occupational therapy. DDDM includes use of a systematic process for clinical hypothesis generation and testing; use of assessment data to develop and tailor client-centered, replicable intervention; and measurement and reporting on proximal and distal outcomes (see Figure 1). Proximal outcomes are the identified factors that affect participation, and distal outcomes are the skills, abilities, and behaviors that are expected to change in response to the intervention (Melnyk & Morrison-Beedy, 2012). Using DDDM, clinicians can “create evidence through practice by utilizing data to guide and measure practice” (Schaaf, 2015, p. 5). Such evidence may help to ensure that therapy practices can survive and thrive in today’s health care reform environment.
Data-Driven Decision Making process (Schaaf & Mailloux, 2015).
In this article, we describe the application of the DDDM process with a child with ASD who received occupational therapy using Ayres Sensory Integration® (ASI) to address deficits in functional skills and participation in daily activities. This case involves a participant from a larger study, a randomized controlled trial of occupational therapy using sensory integration (Schaaf et al., 2014) and is used in this article to illustrate the value of using a systematic approach to guide clinical reasoning and a methodology for focusing on occupation-related outcomes. Although this article operationalizes the use of the DDDM process with a child with ASD using ASI, the process can easily be adapted to other practice areas.
Procedures
As illustrated in Figure 1, DDDM is a sequential process that systematically guides the clinician’s reasoning and the therapeutic process. Each step in the application of DDDM is described in the paragraphs that follow. As noted, the case of a child diagnosed with ASD is used to demonstrate the process.
Step 1: Identifying the Child’s Strengths and Participation Challenges
In the first step in the DDDM process, the clinician identifies participation challenges that are affecting the person’s ability to engage in desired life activities and describes the current level of function for each participation challenge that is identified. Participation challenges are derived from the history, strengths and concerns, and observations of and discussions with the client and key stakeholders.
M was a boy age 4.8 yr, non-Hispanic and White, with a diagnosis of ASD confirmed by psychologists using the Autism Diagnostic Observation Schedule Module 2 (Lord, Rutter, DiLavore, & Risi, 2002) and the Autism Diagnostic Interview–Revised (Rutter, Le Couteur, & Lord, 2003). M’s full-scale IQ using the Stanford-Binet 5 (Roid, 2003) was in the low average range (score of 91), with a nonverbal IQ of 96 and a verbal IQ of 87.
When queried about M’s challenges participating in everyday activities, M’s mother expressed concerns about sleep, play skills, dressing skills, and socialization. She was also concerned about M’s high activity level and poor safety awareness. She described his behavior as disorganized: constantly “on the go,” distractible, and impulsive. M’s mother noted that she had to watch him carefully at all times at home and on the playground because he ran around, played roughly with others, and was not always aware of safety. Moreover, she noted that M woke up very early and was disruptive, running around the house and waking up his siblings. She had tried setting out toys for M the previous evening so that he would play with them upon waking rather than disrupt the family, but this approach had not been successful.
M’s mother also reported that M did not continue with a play activity for more than 2–3 min, moving quickly from one toy to another. This lack of focus annoyed his siblings and caused them to refuse to play with him. In addition, she reported that getting dressed in the morning was another area of difficulty because M was easily distracted and required an excessive amount of cuing and supervision. Finally, M’s mother also stated that M could not attend community activities, such as synagogue with his father, because he was unable to quietly sit still and attend to the service for more than 5 min.
In sum, these behaviors affected the family’s quality of life and M’s independence. Participation challenges were thus noted in the following areas: morning awakening routine, independence in dressing, play with peers and siblings, safety during play, and participation in synagogue.
Step 2: Conducting the Assessment
In Step 2 in the DDDM process, the clinician conducts assessments to identify factors that may be affecting participation challenges. Clinicians choose assessment tools that are in keeping with the theory base or frame of reference identified on the basis of the client’s presenting needs. Given M’s history of over- and underreactivity to sensation and poor sensory perception and discrimination, ASI (Ayres, 2005) was chosen to guide assessment. This choice was based on the literature showing that many people with ASD have difficulty processing and integrating sensory information (American Psychiatric Association, 2013; Baranek, David, Poe, Stone, & Watson, 2006; Mailloux & Smith Roley, 2010) and that this difficulty may be associated with behavioral difficulties such as those demonstrated by M (Hilton, Graver, & LaVesser, 2007; Schaaf, Toth-Cohen, Johnson, Outten, & Benevides, 2011). Accordingly, ASI provided the theoretical framework for assessment and intervention.
The Sensory Integration and Praxis Test (SIPT; Ayres, 1989) and the Sensory Profile (Dunn, 1999) were administered to obtain specific data about M’s challenges in processing and integrating sensation and the impact they may have had on his identified participation challenges. In addition, the Pediatric Evaluation of Disabilities Inventory (PEDI; Haley, Coster, Ludlow, Haltiwanger, & Andrellos, 1992) was administered to further characterize M’s functional skills and to measure change in them.
The SIPT consists of 17 tests that measure a child’s ability to discriminate, integrate, and use visual, tactile, proprioceptive, and vestibular sensations, as well as motor and praxis abilities such as balance, bilateral coordination, imitation, sequencing of actions, laterality, and crossing midline. Raw scores are converted to Z scores for each test and the total test score. Scores falling below −1.0 indicate lower than age-expected performance in that domain. Interrater reliability coefficients reported in the SIPT manual for the major scores are very high, ranging from .94 to .99. Test–retest reliability coefficients vary among the 17 tests, with most reliabilities reported to be in the acceptable range (Ayres, 1989).
The Sensory Profile is a parent questionnaire that measures children’s responses to sensory events in daily life and provides data about how patterns in sensory development might be contributing to or creating barriers to performance in daily life (Dunn, 1999). The Sensory Profile yields numerical scores for each area (e.g., touch processing), which are rated as typical, probable difference, or definite difference (DD) on the basis of normative data. The Sensory Profile has good construct validity and strong internal reliability (>.80) and discriminant validity (>95%) in identifying children with and without sensory issues (Dunn, 1999).
The PEDI is a comprehensive clinical assessment instrument that samples key functional capabilities and performance in children ages 6 mo to 7.5 yr. The PEDI measures both capability and performance of functional activities in three content domains: (1) self-care, (2) mobility, and (3) social function. A normative standard score of 50 represents the mean, and the standard deviation is 10. The PEDI is reported to have excellent internal consistency reliability, ranging from .95 to .99 within the six scales, and moderate validity, ranging from .71 to .73 (Haley et al., 1992). Assessment data are summarized in Figure 2.
M’s assessment data.
Step 3: Generating a Hypothesis
In Step 3, the clinician links the assessment findings to the participation challenges to create a hypothesis summary statement. Analysis and synthesis of assessment data facilitate creation of a hypothesis. Schaaf (2015) indicated the importance of generating hypotheses “that are theoretically driven and that use assessment data to identify the factors affecting participation and identifying and measuring outcomes that are both proximal and distal to participation goals” (p. 3) in the DDDM process.
M’s assessment data are displayed in Figure 2 and revealed decreased body awareness or proprioception (low score on kinesthesia), decreased tactile perception and discrimination (low scores on graphesthesia, localization of tactile stimuli, and manual form perception), decreased vestibular processing (low score on postrotary nystagmus), and decreased ability to plan and carry out novel actions (praxis, as shown by low scores on postural praxis, sequencing praxis, design copying, and motor accuracy). His Sensory Profile factor scores indicated a DD in hyporeactivity in auditory, tactile, proprioceptive, and vestibular processing; difficulty regulating responses to sensation (also known as poor sensory modulation) as shown by DDs on several modulation subscores; and extreme behavioral and emotional reactions during activities as indicated by DDs in scores on modulation of sensory input affecting emotional responses and activity-level subscales.
These findings, in combination with parent report and the therapist’s clinical observations, suggest that M’s difficulty with participation in sleep, play, self-care skills, and community activities may have been related to sensory–motor limitations. Thus, specific hypotheses were generated to articulate the sensory–motor factors that may have been affecting M’s participation. First, M’s behaviors of running around during play, excessive roughhousing and touching of objects and people, impulsivity, and difficulty sitting still, which create safety concerns during play, were hypothesized to stem from sensory hyporeactivity that led to seeking behaviors. Next, decreased somatosensory perception and praxis were hypothesized to be affecting his ability to generate ideas for play or to expand play schemes; last, decreased praxis was also hypothesized to be affecting independence in dressing. The low SIPT scores on postural and sequencing praxis support these latter hypotheses.
Step 4: Developing and Scaling Goals
Five occupation-based goals were identified and scaled using Goal Attainment Scaling (GAS) methodology (Kiresuk, Smith, & Cardillo, 1994). Each goal was scaled according to procedures outlined by Kiresuk et al. (1994), with a rating of 0 as the expected outcome for the 10-wk, 30-session intervention period. These goals were based on the participation challenges noted earlier and were subsequently verified as important areas by M’s mother. Goals for M were as follows:
Awakening quietly: On awakening in the morning, M will play for 10–14 min with the toy set out for him the previous night rather than run around the house or disrupt siblings.
Dressing: After initial verbal instruction, M will independently don three articles of clothing with fewer than three redirections.
Play: M will participate in at least one representational play activity with three to four social exchanges relative to the activity with his siblings for 10–14 min given two prompts.
Safety: M will play safely on a playground given five to six redirections during a 20-min play session.
Participation in synagogue: M will remain seated in synagogue for 10–14 min with a fidget toy.
An example of goal scaling is shown in Figure 3. Goals focused on improving sensory processing as a basis for enhanced ability to participate in everyday activities. In keeping with the theory of ASI and the practice guidelines for occupational therapy (AOTA, 2014), goals were based on assessment findings and focused on occupation-based outcomes. Using DDDM, each goal was stated, the hypothesized underlying factors were listed, and the current level of function was described.
Sample goal and Goal Attainment Scaling.
Step 5: Identifying Outcome Measures
In Step 5, the clinician identifies the proximal and distal outcomes to monitor progress. Proximal outcomes in M’s case were changes in sensory perception, sensory reactivity, and praxis as measured by specific SIPT tests, and distal outcomes were the changes in the goals as measured by GAS score. Each SIPT test yields a standard score that is based on the child’s performance in comparison with age-matched norms and represented by standard deviations; a score of zero represents the mean score for a particular age group.
Step 6: Setting the Stage for the Intervention
In Step 6, the clinician develops and plans an evidence-based, replicable intervention. Use of evidence-based interventions is an important aspect of DDDM. For M, the chosen intervention was occupational therapy using ASI (OT–ASI). OT–ASI follows a manualized protocol that is based on the principles described by Parham et al. (2011). The manual was tested in a feasibility study (Schaaf, Benevides, Kelly, & Mailloux-Maggio, 2012) and a randomized controlled trial (Schaaf et al., 2014), providing evidence for the intervention. Accordingly, each session focused on providing sensory-rich experiences embedded in play that facilitated adaptive responses designed to address the underlying sensory–motor factors hypothesized to be affecting goal attainment, as outlined earlier (Ayres, 1979). Each treatment session was planned to progress from active sensory–motor play focusing on improving sensory perception and decreasing sensory hyperreactivity to active sensory–motor activities that facilitated opportunities for praxis.
The intervention occurred 3 times per week for 10 wk (30 sessions in total) in an outpatient hospital setting in a large gym area equipped with the necessary equipment and other environmental needs. The intervention followed a manualized protocol of occupational therapy using ASI for autism (Schaaf et al., 2014; Schaaf & Mailloux, 2015).
Step 7: Conducting the Intervention
To enhance tactile and proprioceptive perception (somatosensory perception) and praxis, the intervention provides playful opportunities to move the body on, under, and through a variety of tactilely rich play surfaces on equipment and mats. The client completes such activities as “climbing the mountain” (a large foam wedge) and “diving into the lake” (a large ball pit). The session also incorporates active tactile exploration using beads, beans, rice, bath sponges, Koosh balls, and shaving cream. Pushing, pulling, hanging, and weight-bearing activities provide opportunities for active experiences in which muscles must work against resistance, muscle tension, and stretch.
To help regulate sensory over- and underreactivity, active, individually tailored sensory–motor activities embedded in play were integrated into the treatment session with the child’s active participation. They provided opportunities for
High levels of proprioception (e.g., wheelbarrow walk; pulling, climbing, carrying heavy items)
Pressure touch (e.g., hug, massage, using a weighted blanket, playing in spandex, moving under large pillows)
Oral–motor activities (e.g., resistive chewing, blow toys, drinking thick liquid, using a straw to blow items across table)
Antigravity control positions during play
Opportunities to rearrange, remove, and replace equipment or materials, based on the child’s response
Changes in intensity, duration, frequency, or rhythm of sensory experiences, based on the child’s response.
In keeping with ASI, the therapist and child developed an active, trusting relationship within the context of the intervention, and the therapist constantly monitored activity demands, adjusting them as needed to ensure the just-right challenge and the child’s success.
Step 8: Measuring Outcomes and Monitoring Progress
In Step 8, the clinician measures, displays, and analyzes data. Outcomes are measured at regular intervals and displayed for analysis and interpretation. The primary distal outcome marker in this case was GAS. Following GAS methodology, the expected outcome for each goal is a rating of zero, and at postintervention a T score is calculated on the basis of level of goal attainment. To measure level of attainment, M’s mother was asked to rate each goal postintervention. Of note, M’s mother was not privy to the goals during the treatment period to maintain some degree of objectivity. The independent evaluator reviewed the ratings with M’s mother to assure her that the goals captured M’s mother’s intent. Moreover, this process ensured that the goals and, subsequently, the intervention were tailored to M’s specific needs on the basis of his unique client and contextual factors.
Once all goals were rated, they were summed, and a T score was calculated using the formula described by Kiresuk et al. (1994). A summed score of 50 would indicate that goals were met at the expected level. M’s postintervention GAS T score was 68, indicating that he met or exceeded expected outcomes on his goals. Specifically, M’s mother rated him as +2 on Goals 1, 2, and 5 and 0 (expected outcome) on Goals 3 and 4. During the posttest GAS interview, M’s mother described great improvement in self-dressing, independent and sibling play, safety, and attention during community outings. She noted, “M bathes himself except for turning on the water. He can bathe in his bathroom while I bathe the baby in another room, with minimal supervision. I don’t have to worry about him not being safe.” Using the prompt of dressing, M’s mother reported, “This is an area of success. M can get dressed by himself and chooses his own shirt and pants. He comes down dressed and even cleans up the bathroom.” Regarding play, his mother reported, “He doesn’t wake the family any more by roaring through the house. The other day, he also was quietly playing with toys in his bed until everyone else woke up. . . . Things of interest he can now attend [to] for a long time, like 30 minutes.”
To further evaluate any changes in functional skills, the PEDI was administered postintervention. M improved in every domain of the PEDI (see Figure 4). His most dramatic improvements were in the areas of mobility functional skills (from a score of 32.2 to 60.1), self-care caregiver assistance (from a score of 36.1 to 60.1), and mobility caregiver assistance (from a score of 41.9 to 59.2). He also improved in self-care functional skills (from a score of 33.5 to 36.7), social function functional skills (from a score of 20.9 to 36.7), and social function caregiver assistance (from a score of 18.3 to 31.4).
Pre- and postintervention scores on the Pediatric Evaluation of Disability Inventory.
Discussion
Health care and educational legislation and policy require that clinicians be accountable by measuring and reporting outcomes (King et al., 2011). This article provides an example of a systematic process to guide occupational therapy practice with a focus on outcome measurement. DDDM’s structured format facilitated the clinician’s use of the systematic process and provided a method to describe the reasoning process, including a data-based rationale for treatment decisions. The process allowed the clinician to engage in a client-centered practice and to individually tailor the intervention to target the specific underlying sensory–motor factors that were identified through the collection and analysis of assessment data. Moreover, the DDDM process allowed the clinician to systematically and clearly document outcomes showing that the occupational therapy intervention, in this case OT–ASI, helped this child to meet his goals. Although further application and testing of DDDM are needed, this case provides a model for its use in ASI that can be applied and tested in other areas of occupational therapy practice to guide implementation of interventions and outcome measurement. Practices such as these can increase evidence in occupational therapy and help determine which types of occupational therapy interventions are useful for specific client characteristics. This process of tailoring treatment more closely to the client’s individual profile of strengths and needs is in keeping with contemporary health care practices (National Institutes of Health, 2015).
DDDM also provides a useful strategy for incorporating outcome measurement into daily practice. First, hypothesis generation allows the therapist to use assessment data to determine an appropriate intervention approach. Second, the identification and measurement of outcomes at the proximal and distal levels allow therapists to monitor intervention effectiveness and also to validate reasoning. When both proximal and distal outcomes show concurrent improvements, the hypothesis not only provides support for the occupational therapy intervention but also creates a bridge from practice to research. When outcomes do not show change, the hypothesis can be revised, the approach altered, or the theoretical principles guiding assessment and practice changed.
King et al. (2011) found that parents indicate that outcome measurement and display of outcome data are helpful for them in terms of keeping abreast of how their child is progressing. Display of outcomes can thus be a useful strategy for communication with parents and other stakeholders. Regarding the use of GAS as an outcome measure, this process was useful in terms of individualizing goals in a way that was quantifiable and measurable. Although the creation and scaling of the goals can initially be time consuming, it proved useful for monitoring and measuring progress for this child and in the larger study. GAS provides a mechanism for identifying individual outcomes in a quantifiable way and then a way to measure against the child’s own initial baseline and expected outcomes. It will be important to determine the feasibility of using GAS in settings with high productivity requirements.
Implications for Occupational Therapy Practice
The results of this study have the following implications for occupational therapy practice:
The DDDM process is a tool to assist clinicians with systematic decision making and outcome measurement.
Clinicians may need to reframe their thinking related to outcome measurement and incorporate it into their everyday practice.
Use of outcome measurement as a standard of practice will allow clinicians to create evidence through everyday practice.
Conclusion
Occupational therapy practitioners are charged with documenting and reporting on patient outcomes to demonstrate effectiveness of intervention strategies in keeping with current health care legislation. Evidence for occupational therapy interventions for children with ASD, particularly the intervention used in this example, OT–ASI, is sorely needed (Clark, 2012). To produce evidence, clinicians must be systematic in their approach, conduct standardized assessments that provide data to guide intervention, and measure and report on outcomes. The DDDM process used in this case example provides a model whereby clinicians can integrate outcome measurement into their daily practice to produce evidence through practice.
Footnotes
Acknowledgment
Special thanks to M and his mother for their time and participation in the study. Thanks also to Teal Benevides, Regina Freeman, and Donna Kelly for their contributions to the larger study and members of the Sensory Integration Research Collaborative (Drs. Janice Burke, Ellen Cohn, Jane Koomar, Shelly Lane, Teresa May-Benson, Diane Parham, Sarah Schoen, and Susanne Smith Roley) for contributions to the intervention manual and the Ayres Sensory Integration Fidelity Measure© used in this case, as well as wisdom and time. This case is part of a grant funded by the Autism Speaks Foundation (No. 3797). This article was completed in partial fulfillment of an occupational therapy doctorate at Thomas Jefferson University.