Effect of Assistive Technology in a Public School Setting

Research Design

We used a repeated-measures pretest–posttest quasi-experimental design. The repeated-measures design allowed students to serve as their own controls, which was necessitated by high participant heterogeneity (a wide variety of disabilities, ages, and types of devices used), thus eliminating between-subjects variability in the baseline (pretest) and follow-up (posttest) conditions (Carey & Boden, 2003; Portney & Watkins, 2000; Ritchie, 2001; Smith, 2000). We used a repeated-measures design instead of a randomized controlled trial because the small number of new students served by the AT team prevented analysis of independent groups necessary for the latter methodology. An additional problem with a randomized controlled design would be the necessity of withholding intervention to create a control group. Withholding or delaying intervention is unethical in this situation. Another positive aspect of using a repeated-measures design over independent samples is the greater power in statistical procedures despite the small number of participants (Portney & Watkins, 2000).

Participants

We determined the minimum number of participants for the study through consultation with the coordinator of the AT team. She predicted, based on past referral rates, that there would be a minimum of 10 newly referred students or existing students with new devices within the first semester of the 2005–2006 school year (G. Williams, personal communication, May 2005). The study exceeded the minimum number, with 13 participants.

The main inclusion criterion consisted of all students newly referred to the AT team (or ongoing AT students who had new AT) for the 2005–2006 school year. Additional inclusion criteria were the student’s age (between 3 and 21) and special education status (the student had to have a current IEP). The study limited participation to those students whose case managers could complete the follow-up before the end of the 2005–2006 school year. Limiting data collection to 1 school year was an attempt to limit extraneous variables that could arise from data collection over 2 or more school years (i.e., new teachers, grade expectations, or school buildings).

We excluded students who met the inclusion criteria but whose parents and IEP team case managers declined consent. We also excluded students who would otherwise have met the inclusion criteria but whose parents did not read English or Spanish fluently because consent forms were available only in those two languages. Only 1 student had parents who spoke a language other than English or Spanish (Mongolian) and was thus excluded.

Outcome Measures

We used the Student Performance Profile (SPP; Assistive Technology Outcomes Measurement Systems, 2004; Fennema-Jansen, 2004) as the study measure. The SPP, developed by the Ohio Department of Education and the Rehabilitation Research Design and Disability (R2D2) Center, began as an online, study-specific instrument for the Ohio Assistive Technology Infusion Project. The Ohio Assistive Technology Infusion Project used the SPP to collect data on approximately 4,000 students. Teachers in the Ohio study recorded their students’ experiences with the AT provided by the project. Evidence for content and discriminant validity has been presented through dissertation studies and national conferences (Fennema-Jansen, 2004, 2005; Fennema-Jansen, Smith, Edyburn, & Binion, 2005). In this study, we used a modified print version of the instrument written in conjunction with and approved by the developers of the original SPP (Assistive Technology Outcomes Measurement Systems, 2004). Readers are directed to the R2D2 Center Web site (www.r2d2.uwm.edu/atoms/archive/sppi3.html) to see the version of the SPP used in this study.

Our version of the SPP included pretest and posttest forms. The pretest form included three sections: Section I, Student Information; Section II, Areas of Need; and Section III, Relevant (to AT) IEP Goals or Objectives and Current Ability Level. Case managers could identify and rate up to three AT-relevant IEP goals or objectives. The case managers rated each identified IEP goal or objective on a scale ranging from 0% ability to 100% ability in 10% increments (e.g., a student who has mastered the goal or objective would receive a 100% ability-level rating). The posttest form included three sections relevant to this article: Section I, Student Information; Section II, Current Ability Level of (previously identified) Relevant IEP Goals or Objectives; and Section III, Contribution of Interventions.

We used the SPP because it can be customized to a wide variety of students, it is specific to the use of AT in a public school setting, and it bases performance on IEP goal or objective ability level. We also chose the SPP because the developers had experience with its use with a heterogeneous group of AT users in public education. The case managers rated each student’s ability level on IEP goals or objectives on the SPP pretest form and on the SPP posttest form. The measure of performance was the difference in averaged ability level (of up to three goals or objectives) on the pretest and posttest forms.

Rating student performance with the SPP (ability level on relevant IEP goals and objectives) does not distinguish between student performance with and without AT. The SPP measures only the change in IEP ability level after the introduction of AT. The SPP posttest form attempts to distinguish the impact of AT on the student’s ability to achieve IEP goals and objectives with an additional section, Section III, Contribution of Interventions. Section III includes 10 rating scales representing 10 interventions. The scales range from 0 (no contribution) to 10 (substantial contribution) to improved IEP ability level. The interventions are broken into four categories: student strategies, teacher strategies, special services, and AT.

Procedures

Institutional review board (IRB) approval was obtained through Nova Southeastern University. The IRB approval preceded the approval from the school system in which the study took place. We also sought additional approval from the school principals at each school with participants. Data collection began after the completion of these three levels of approval.

The single independent variable in the study consisted of the AT devices and services provided by the AT team to the participating students (nominal data). We chose AT team member services (including AT device provision) as a single intervention because this service was the consistent variable across all participants. The services provided by the AT team were at minimum an interdisciplinary team review of the AT referral, discussion about the referred student’s case and assignment of the most appropriate AT team member to the case, an initial visit by the AT team member to the referring case manager (usually the student’s special education teacher), an observation of the student performing the activities for which the AT was being considered, and collection of student baseline performance data (e.g., words per minute produced in student’s usual written communication method).

If the AT team member recommended AT, the team member provided these services: AT device or devices, initial training of the teacher and student in the use of the AT device, and initiation of a follow-up regarding the student’s AT use at least twice a year and provision of support for the use of and maintenance of the AT as needed. The AT team member’s level of service was higher for students with complex AT (i.e., speech-generating devices or dictation software).

The two dependent variables in this study consisted of student performance and the relative contribution of the AT intervention provided by the AT team compared with that of nine other commonly used interventions to improvement in student performance. Student performance was defined by student ability level on AT-relevant IEP goals or objectives as rated by the student’s case manager (0% = no ability, 100% = a fully met IEP objective or goal). Student performance was rated using the SPP. Rating scale scores are usually viewed as ordinal data; however, the SPP has the assumption of an interval scale because, as noted earlier, the ratings are in equal, numerical intervals or increments (0% to 100%, with even increments of 10%).

The level of measurement for the second dependent variable, contribution of the AT intervention to the improvement in student IEP objective or ability, is ordinal because the level of contribution is based on an SPP rating scale (ranging from 0 [no contribution] to 10 [substantial contribution]). The case managers were instructed that multiple interventions could be rated the same (i.e., each intervention did not need to be ranked relative to the others) and that the numbers associated with the ratings did not need to add to a specific number (e.g., 100).

We recruited every participant who met the inclusion criteria. The case manager of 1 student declined participation, and 2 students’ parents declined to give consent. The case managers of all participants completed the posttest evaluation. The data collection process started when the AT team provided services to a newly referred student or provided new AT devices to an existing student (and after all parties signed informed consent forms). The first author (Watson) met with the case manager (the IEP team member primarily responsible for the student’s IEP; most were special education teachers) and other appropriate IEP team members for the student at the case manager’s school. The first interview included a verbal introduction to the study but withheld the study hypotheses.

At the first data collection meeting, the first author asked the case manager to identify up to three pertinent areas of student need addressed by the AT and included these on the SPP pretest form. The SPP pretest form was used to guide the case managers to identify one to three IEP goals or objectives most relevant to the AT. Although the school system used goals only for students taking standardized achievement tests, it used goals and objectives for students with alternative testing, thus goals or objectives could be used on the SPP.

The first author reviewed the instructions for the instrument with each case manager and answered any questions that arose. The case managers were instructed to rate the student’s ability level on each AT-relevant IEP goal or objective on the SPP pretest’s Section III (Relevant IEP Goals or Objectives and Current Ability Level), using the rating scale described earlier. Case managers rated their student’s ability level on each goal or objective on the basis of the student’s performance immediately before AT was introduced. To avoid the potential for bias, case managers completed the SPP pretest form while Watson was in the building but not in the room.

Approximately 4 months after AT provision, the same case manager and any other IEP team members who completed the SPP pretest form completed the SPP posttest form. Watson delivered all follow-up instruments to the case managers and reviewed the instructions with them. The SPP posttest form included the goals or objectives (entered via a word processor) identified on the SPP pretest form for each student, but it did not include the pretest rating levels. Pretest ratings were excluded from the posttest to minimize case managers’ scoring bias when rating posttest Section III (Relevant IEP Goals or Objectives and Current Ability Level.)

The case manager completed the SPP posttest form regarding the student’s ability level on IEP goals and objectives, as indicated earlier, and the relative contribution of 10 intervention strategies for each identified IEP objective (Fennema-Jansen et al., 2004). The SPP posttest form asked the case manager to identify any possible confounding variables that may have influenced the student’s progress negatively or positively before or after the introduction of AT. The case managers did not identify any untoward events during the time of any student’s AT use.

Data Analysis

We used SPSS software (Version 15.0; SPSS, Inc., Chicago) and Microsoft Excel 2004 for Macintosh (Version 11.3.7; Microsoft Corporation, Redmond, WA) to analyze the data. The study included descriptive statistics for student participant characteristics and for the AT provided to students. The descriptive analysis also provided frequency distributions of data used for inferential statistical tests. We used the paired t test to compare the mean (of IEP ability level) of each participant before AT was introduced with the mean of each participant after AT was introduced. We used a significance level of p < .01 (one-tailed) on all analyses using the t test. The SPP scores used for the t-test analyses were the mean percentage ability level on the chosen, relevant IEP goals or objectives for each student.

The first author analyzed the case manager–reported contribution level of AT provided by the AT team versus other interventions using a one-way analysis of variance (ANOVA). An analysis of the mean of each intervention revealed that no single intervention alone or the averaged mean contribution of all nine non-AT interventions was rated as highly as the AT intervention. We used an ANOVA to compare the two highest rated interventions (of the nine non-AT interventions) with the mean AT rating to determine whether significant differences existed.

Description of the Sample (Participant Characteristics)

None of the participants were lost at follow-up, and all continued to use their AT (0% abandonment). The participating students ranged in grade level from preschool to eighth grade. Table 1 summarizes the participating students’ grade levels. The participating students represented a variety of special education disability classifications (see Table 2).

Assistive Technology Provided

The AT team provided two broad categories of AT, speech-generating devices (for oral communication) and AT for written communication. The participants received 32 AT devices altogether. Written communication devices consisted of hardware or software. Table 3 summarizes the devices and software provided by the AT team.

Student Performance Progress

SPP posttest Section II measured student performance as the students’ ability level (ranging from 0% to 100%, in 10% increments) on AT-relevant IEP goals and objectives. The case managers identified 27 IEP goals or objectives relevant to AT; the average number of IEP goals and objectives per student was 2.08. (If a student’s case manager identified multiple IEP goals and objectives for the SPP, performance was measured as the average of the change in ability level for all identified IEP goals or objectives.) Most of the students (11 of 13) showed improvement in ability level on identified IEP goals or objectives at posttest (after receiving AT) compared with ability level at pretest (before receiving AT). The remaining 2 students’ averaged level of ability stayed the same. The mean percentage of ability for the 13 participants was 36% before the AT intervention and 67% after the AT intervention, for a mean improvement of 31%. The participating students’ overall performance on SPP posttest Section II (percentage of ability on IEP objectives and goals) improved significantly (t[12] = 5.54, p = .00, one-tailed, d = 1.40). See Table 4 and Figure 1 for a summary of SPP Section II pretest and posttest scores.

OPEN IN VIEWER

Figure 1.

Frequency distribution of the scores of the averaged Student Performance Profile (SPP) on the pretest and posttest forms, in percentages, of current ability level of individualized education program goals and objectives (N = 13).

Relative Contribution of AT Provided by the AT Team Compared With Other Interventions

Case managers completed SPP posttest Section III after rating each student’s ability level on every identified IEP goal and objective on the SPP posttest form. The participants received multiple concurrent forms of interventions in addition to AT, thus inquiring about the level of contribution of AT versus other interventions helped to clarify the reason for the improvement in IEP ability level. The SPP posttest Section III rating scale ranged from 0 (no contribution) to 10 (substantial contribution). Our hypothesis was that AT would contribute to IEP improvement at least the same as or more than the nine other interventions.

The two non-AT interventions with the highest means were “adaptations of specific curricular tasks (e.g., worksheet modifications, alternate test taking)” and “related and support services (e.g., occupational therapy, physical therapy, speeach–language therapy, Title I, Tutoring),” which were Items 3 and 7 on SPP posttest Section III, respectively. For Items 3 and 7, means were 6.00 and 5.37, respectively (standard deviations [SDs] = 1.88 and 2.36, respectively) compared with a mean of 7.74 (SD = 1.99) for Item 10 (the contribution of AT provided by the AT team). Each intervention had 27 ratings of contribution to IEP goal ability level (based on the 27 relevant IEP goals and objectives identified by the case managers). The data from the three data sets mentioned previously (Items 3, 7, and 10) meet the assumptions of the ANOVA for homogeneity of variance (SDs = 1.88, 2.36, and 1.99, respectively) and normal distribution. See Figures 2 and 3 for frequency distributions of the ratings of Items 3, 7, and 10 of SPP posttest Section III.

OPEN IN VIEWER

Figure 2.

Frequency of each rating (contribution) level, by Student Performance Profile Posttest Section III, Item 7 (related and support services) and Item 3 (adaptations of specific curricular tasks; N = 27 questions).

OPEN IN VIEWER

Figure 3.

Frequency of each rating (contribution) level, by Student Performance Profile Posttest Section III, Item 10 (assistive technology [AT] intervention).

A one-way ANOVA with a Tukey’s post hoc test between these three intervention strategies (Items 3, 7, and 10) revealed a significant difference between AT provided by the AT team (Item 10) and each of the other two interventions (Items 3 and 7) but no significant difference between Items 3 and 7 (F[2] = 9.35, p = .00). This analysis may provide evidence of AT’s positive contribution compared with that of other intervention strategies in promoting attainment of IEP goals and objectives (see Table 5).

Limitations

This study’s limitations include the small sample obtained by consecutive sampling (a form of convenience sampling), the use of instrumentation (the SPP) with relatively little psychometric information, a sample pool limited to one school district, and the use of the t test on ordinal data.

The SPP captured data based on rating scales and therefore met the classification of ordinal data (Portney & Watkins, 2000). We used the t test and the ANOVA based on ordinal data with a distribution unknown a priori, factors that violate the assumptions for the t test and ANOVA of interval- or ratio-based data with a normal distribution (Bridge & Sawilowsky, 1999; Ottenbacher, 1983; Portney & Watkins, 2000; Song et al., 2006).

The t test and the ANOVA, however, typically retain robustness, or avoidance of Type I errors, when the assumptions are not met and thus may account for the frequent use of these statistics in many medical and social science studies. The concern considered more likely by statisticians is the ability of the t test to retain power, or the ability to avoid a Type II error, when used with noninterval and non-normally distributed data. The lack of power is a concern especially in studies that have small (<25) samples (Bridge & Sawilowsky, 1999; Song et al., 2006). We found significant results using the t test despite the violation of the t test’s assumptions and the small sample size. The ANOVA in this study, although based on ordinal data, met the assumptions of equal variance and normal distributions with a sample size >25.

Future Research

The implications and recommendations for future research include SPP instrument development, further research into intervention with specific AT devices, and further definition of the constructs of performance as they relate to AT intervention. Future research on AT outcomes in a public school setting should consider multiple data collection phases over a longer period (>4 months). Some devices used with various disability categories may require >4 months to show changes. The first author (Watson) collected data over only 1 school year and interviewed only case managers who requested AT as an intervention. Data collection over multiple school years with case managers who inherit the AT with their new students may view AT’s effect differently than case managers who request AT.

Clinical Implications

The first clinical implication of this study is that AT provided by a multidisciplinary team may be helpful in promoting improved performance (attaining IEP goals and objectives) in a public school setting among heterogeneous groups of students who have difficulty meeting these IEP goals and objectives with other interventions. However, a practitioner or administrator who uses the results of this study as evidence for AT’s effectiveness in supporting student performance should consider the service delivery model a critical component of its effectiveness. The participants in this study received AT from a multidisciplinary AT team and from an IEP team that made a commitment to seeking and implementing AT.

The second implication for practice is the experience gained with the SPP, an AT outcome instrument in its developmental stage. The SPP appears to have potential as an effective means of collecting AT outcomes data in the public school setting. The SPP has the following characteristics: ease of administration, specificity to relevant measurement constructs (IEP goals and objectives), sensitivity to change in performance over time, and ease of scoring. An instrument with these characteristics may help practitioners pursue data collection regarding the effect of AT on student performance.

A third implication of this study for practice is the impetus it may provide for collection of outcome data in this arena of practice. Many articles have attested to the importance of measuring AT outcomes, but the literature has provided little guidance on this process (DeRuyter, 1997; Fuhrer, 2001; Gelderblom & de Witte, 2002; Jutai et al., 2005; Minkel, 1996; RESNA, 1998a, 1998b, 1998c; Smith, 1996). Some authors have directed practitioners in methods to determine which students are appropriate for AT intervention; however, these same authors do not extend the discussion to the monitoring of outcomes (Cook & Hussey, 2002; Edyburn, 2001; Lenker & Paquet, 2003; Quality Indicators for Assistive Technology Consortium, 2005; Zabala, 2001). Guidelines for collection of AT outcomes data may have implications for individual school district practice. Individual school districts that use this study’s recommendations for data collection and analyses of AT outcomes may improve service delivery and provide a basis for justification of their service. Service delivery could improve if AT providers find deficits in their service and then work to improve them. Data that demonstrate the effectiveness of AT intervention on student performance may help secure continued support from school administrators.