Abstract
Although evidence-based practices for autism spectrum disorders exist, they are often not effectively incorporated into school-based programs, indicating a need for enhanced training strategies for educators. This study examined the effects of immediate video feedback during coaching for teachers and paraprofessionals learning Classroom Pivotal Response Teaching (CPRT). Special education teachers, along with their classroom paraprofessionals, were randomly assigned to a coaching as usual (CAU) or a coaching with video enhancement (VE) condition. Both groups received both verbal and written feedback regarding strengths and weaknesses of their CPRT implementation. Additionally, the VE condition received video feedback during their coaching sessions. Overall, teachers demonstrated higher fidelity of implementation than paraprofessionals, t(44) = −2.73, p < .01, but no significant group differences were identified between VE and CAU conditions. Univariate analysis of variance models were conducted to examine the relationship between participant satisfaction regarding overall quality of the training and highest percentage of CPRT components passed, F(2, 37) = 3.93, p = .03. Results indicate use of the iPad may impact training outcomes and participant satisfaction with training procedures and add to the very limited literature on how technology may be used to enhance in-service training for teachers.
Special education enrollment for autism spectrum disorders (ASD) has increased dramatically, quadrupling nationwide from 2000 to 2011 (Scull & Winkler, 2011
Training In-Service Teachers and Paraprofessionals
Several effective methods for training in-service teachers have been identified, including providing direct instruction, opportunities to practice skills while receiving feedback, and ongoing coaching and supervision (National Advisory Mental Health Council, 2001; Odom, 2009; Reid, Parsons, & Green, 1989; Scheuermann, Webber, Boutot, & Goodwin, 2003; Suhrheinrich, 2011). Underscoring its critical role in training, coaching has been linked with teachers’ increased use of new strategies across areas of education. Rudd, Lambert, Satterwhite, and Smith (2009) investigated the effect of coaching on teacher use of math-mediated language in preschool classrooms, and concluded that while professional development alone resulted in a 56% increase, when preschool teachers received side-by-side coaching, their usage of math-mediated language increased by an additional 39%. In the area of literacy education, a large-scale project found that with coaching, teachers’ practices became more consistent with national standards (Stephens et al., 2011). Research focused on training special educators also supports the importance of coaching. One study involving special educators found only 15% of participants met mastery criteria for use of the intervention after attending a workshop, but 40% of participants demonstrated mastery after only two individual coaching sessions (Suhrheinrich, 2011).
In order to meet the growing needs of students receiving special education services, paraprofessionals provide educational services and support in school programs (Giangreco, 2003). Although paraprofessionals play such a key role in the education of students with disabilities, few studies have focused on the association between paraprofessional implementation and student outcomes (Young, Simpson, Myles, & Kamps, 1997). Schools often hire untrained individuals to fulfill needs and may not provide the necessary training and support for them to adequately educate students with disabilities (Downing, Ryndak, & Clark, 2000). In a series of paraprofessional interviews by Downing, Ryndak, and Clark (2000), most paraprofessionals reported that they entered their jobs with little to no training, and learned about the students, modifying curriculum, and behavior management while on the job. Downing et al.’s survey participants also reported that, although helpful, the limited in-service trainings they attended were not always practical or applicable to the settings in which they worked. Brown, Gatmaitan, and Harjusola-Webb (2013) suggest that performance feedback is a necessary component to ensure generalization of in-service training content to settings in which paraprofessionals are truly working with students. Ongoing coaching and performance feedback, therefore, have been identified as important factors in improving teacher and paraprofessional use of new strategies.
Video-based coaching
There is a growing body of literature on the use of technology in training practices, including coaching. Technology has the potential to facilitate implementation of new strategies (Kelley & Brickman, 2009). Video-based coaching involves the adult learner watching a film of themselves and receiving feedback on their performance. It has been utilized in a variety of settings to enhance training outcomes. For example, review of surgery videos resulted in improvement in and sustainment of resuscitation behaviors in trauma room physicians (Scherer, Chang, Meredith, & Battistella, 2003) as well as development of videoscopic suturing skills (Jamshidi, LaMasters, Eisenberg, Duh, & Curet, 2009) and technical and decision-making aspects of surgery (Hu et al., 2012). Furthermore, Stokes, Luiselli, Reed, and Fleming (2010) compared the effects of (verbal) descriptive feedback alone and (verbal) descriptive feedback in combination with video feedback on high school football players’ pass-blocking skills. In the descriptive feedback alone condition, football players were not able to improve their pass-blocking skills. Once shown a video of themselves and then given descriptive feedback, their pass-blocking skills improved. There is also preliminary evidence to suggest the effectiveness of using video-based coaching for teachers in training. Morgan, Menlove, Salzberg, and Hudson (1994) studied the effects of peer coaching on direct instruction skills in preservice teachers. Coaches reviewed videos of the teacher participants with the goal of facilitating teacher self-evaluation and providing feedback to improve teacher instructional behaviors. After receiving video-based coaching, teachers were able to increase the percentage of effective instructional trials, as well as their frequency of specific praise, which was correlated with increases in student responding. Moreover, these teacher skills were generalized to a different subject matter.
The use of video to instruct teachers has also been well documented for over 25 years (Tripp & Rich, 2012). One example of an activity commonly used for teachers called microteaching requires teachers to record a video of themselves teaching and then later analyze the video with peers. A microteaching video offers the teacher an opportunity to reflect on their lesson and receive constructive feedback. Data support this video-based review as a useful strategy for teachers to both improve the ability to evaluate teaching and make changes to teaching practice (Tripp & Rich, 2012). Teachers involved in microteaching also reported aspects of the video review that helped them to change their own behavior, including a focus on key behavior, the opportunity to see themselves teaching (new perspective), and see their own progress (Tripp & Rich, 2012).
Despite teachers’ report of satisfaction and ability to change their teaching practice using video review for professional development, review with peers may be a barrier to use for some providers. For example, special educators and paraprofessionals may not have relevant peers at their school site. The fact that a peer review session inherently provides delayed feedback should not be ignored. One increasingly available tool that may be useful for providing performance feedback to teachers is the iPad or other tablet technology. Utilizing video recording applications on the iPad, supervisors and/or trainers have a portable, user-friendly method of collecting video samples of teachers’ lessons and can review key moments of the lessons with the teachers during a postlesson feedback session. Therefore, further electronic methods for feedback should be investigated.
The temporal relationship between the teaching and feedback has also been considered. Barton, Fuller, and Schnitz (2015) studied the effects of providing e-mail feedback as a method for coaching early childhood teachers. While they found that the e-mail feedback helped to increase target behaviors in the teachers, they questioned whether providing teachers with more immediate feedback would result in better maintenance of teacher skills. In a review of the literature on providing performance feedback to teachers, Scheeler, Ruhl, and McAfee (2004) concluded that immediacy of feedback was an important piece of long-term change in teacher behavior.
In summary, video-based coaching has a strong evidence base for increasing desired behavior across a range of skills. However, the benefits of using immediate video feedback in school-based training have not been investigated. Additionally, there is no existing literature on training paraprofessionals, together with their lead teachers, using immediate video feedback. The primary objective of the current project was to investigate the effects of using video and iPad technologies, in addition to the standard coaching protocol, on teacher and paraprofessional fidelity of implementation (FI).
Method
To facilitate the current project objectives, participants were recruited from a larger effectiveness trial of Classroom Pivotal Response Teaching (CPRT). Pivotal Response Training (PRT) is a naturalistic intervention based on the principles of behavior analysis and an established EBP for ASD (Wong et al., 2014). PRT has been systematically adapted to better fit a classroom context, resulting in a training manual (Stahmer et al., 2011) and procedures (Stahmer, Suhrheinrich, & Rieth, 2016). CPRT is being evaluated in a randomized waitlist-control trial that includes training and coaching teachers and paraprofessionals in a large urban county in Southern California.
The coaching team was comprised of four members of the research staff with a master’s (n = 2) or doctoral (n = 2) degree in psychology or a related field and clinical experience working in educational settings with children with ASD. Upon hire, each coach was trained to assess FI of CPRT from a video sample. After reliably scoring two consecutive videos with 80% interrater reliability against a key, coaches then practiced providing coaching feedback with the same videos. To prevent observer drift, coaches met monthly to score a new video and discuss feedback to provide the teacher in the video. Coaches were also taught how to film CPRT sessions and tag key moments using the Stage Pro app on an iPad.
Participants (n = 47) included 28 special education teachers and 19 paraprofessional educators from elementary school classrooms in a large urban county in Southern California. Participants were part of a larger trial examining the effectiveness of CPRT (Suhrheinrich, Stahmer, & Rieth, 2016). The participants in the current study represent one training cohort of the larger effectiveness trial; all training and data collection took place during one academic year (see Table 1 for participant demographics and Table 2 for participants’ self-reported job satisfaction and job-related stress).
Participant Demographics.
Job Satisfaction and Stress Levels.
Teachers and the paraprofessionals working in their classrooms were randomized to the coaching as usual (CAU; n = 15 teachers, 9 paraprofessionals) or the video enhancement (VE; n = 13, 10 paraprofessionals) condition. (Please find a full description of procedures for CAU and VE conditions below.) As was standard procedure for the larger effectiveness trial previously mentioned, all teacher participants completed 12 hours of didactic training in CPRT strategies. Training was conducted in small groups and included lecture, discussion, review of video examples, lesson planning, and other application activities and occurred once per week for 2 hours for a total of 6 weeks. Training was scheduled during teachers’ standard work hours on student minimum attendance days. Because paraprofessionals did not typically work during these nonstudent times, they were given the option to participate in the 12 hours of group training or independently view narrated presentations of the same content. Only paraprofessionals who completed all 12 hours of didactic training (via either live group or independent study) were offered the opportunity to receive coaching.
After the first 6 hours of training were complete, in-classroom coaching began for all participants. Coaching appointments were scheduled during the regular school day during a teacher-selected activity. Before training began, teachers selected their activities and identified specific student goals to target during those activities. This was done to provide a predictable activity in which teachers could apply the principles of CPRT to the classroom setting. During coaching, the coach observed the participant working with one or more students, and collected in vivo FI data. To do so, coaches rated participant implementation of CPRT strategies (Table 3) on a scale from 1 (participant did not use the component throughout the session or less than 30% of given opportunities) to 5 (participant implemented the component throughout the session or 100% of opportunities). When participants achieved scores of 4 or above across all components, coaching frequency reduced and they were offered the opportunity to receive coaching in a different activity, to promote generalization of skills.
CPRT Components.
Note. CPRT = Classroom Pivotal Response Teaching.
During coaching sessions in the CAU condition, the participant completed a written form to guide self-reflection of their implementation of the CPRT strategies. Utilizing open-ended questions and statements, the CPRT coach then reviewed the written form with the participant, highlighted areas of strength (components receiving an implementation score of 4 or 5) and weakness (components receiving an implementation score of 1, 2, or 3) based on the coach’s own assessment of FI, and discussed ideas to improve implementation of CPRT components. All feedback was written down, and a copy was given to the participant for his or her records. For participants in the VE condition, in addition to the verbal and written feedback provided to the participants in the CAU condition, the CPRT coach recorded the teaching activity using an iPad and the Stage Pro application. Stage Pro, developed by Belkin International, is an interactive whiteboard and document camera application that lets the user draw or insert images over live video recording (Belkin International Inc., 2015). This application was selected because it is accessible and simple to use. Coaches were instructed to use the touch screen to mark when specific CPRT components were used during instruction. At the end of the session, the CPRT coach and the participant reviewed the video with the embedded feedback notes to highlight successful and challenging areas of implementation.
Measures
Demographics
Participant demographics, including personal information, professional experience and job-related stress and satisfaction, were collected prior to participation in the training. Participant completion of demographic measures was 94%.
Fidelity of implementation
FI of CPRT was evaluated during each coaching session. Three FI calculations are used as outcome measures.
Mastery of CPRT components
Mastery criteria for CPRT are correct implementation of each component 80% of the session (a score of 4 or above on the FI coding form). Mastery of CPRT indicates the participant met these criteria at least one time.
Highest percent of components passed
The percentage of CPRT components correctly used at least 80% of the session (a score of 4 or above on the FI coding form) was calculated for each coaching session. The highest percentage passed across all sessions was used for analysis.
Coaching sessions to mastery
Mastery of CPRT components was evaluated for each participant at each coaching session. The number of coaching sessions that occurred prior to mastery was calculated and used for analysis. Participants who failed to meet mastery criteria throughout all coaching sessions were not included in this calculation.
Satisfaction
Teacher and paraprofessional satisfaction with CPRT and participation in the training procedures was evaluated upon completion of all coaching sessions at the end of the school year.
Results
Teachers completed a mean of 7.68 (range = 2−14) coaching sessions and paraprofessionals completed a mean of 5.32 (range = 1−11) coaching sessions. Analysis indicated teachers demonstrated higher percentages of components passed as compared to paraprofessionals, t(44) = −2.73, p < .01. Further, there was a significant difference in teachers’ demonstrated mastery of CPRT components as compared to paraprofessionals, t(44)= −2.38, p = .02.
Outcomes were also evaluated by coaching condition (CAU, VE). The total number of coaching sessions did not differ by the coaching condition for both teachers (p = .22) and paraprofessionals (p = .47), indicating no difference in the amount of coaching received. There was no significant difference across conditions in mastery of CPRT components (p = .15). However, there was a trend toward a group difference in the highest percentage of CPRT components correctly used during a coaching session, with participants in the VE condition demonstrating higher percentages of CPRT components passed, t(44) = −1.89, p = .07. Additional analysis revealed no significant relationship between participants’ job-related stress and percentage of CPRT components passed, F(3, 38) = .35, p = .79 (see Table 4 for average mastery and sessions to mastery outcomes across participant groups and conditions).
Average Mastery and Sessions to Mastery.
Note. VE = video enhancement; CAU = coaching as usual.
aAverage sessions to mastery level were calculated using data from only the participants who reached mastery level.
Satisfaction With Training Procedures
Both teacher and paraprofessional participants reported high satisfaction with CPRT and the training they received, with 88% of teachers and 92% of paraprofessionals reporting they were very satisfied or satisfied with the quality of the training they received. Additionally, 96% of teachers and 100% of paraprofessionals reported that they would recommend the training and/or intervention to other teachers. When looking at the relationship between condition and satisfaction, there was a trend toward significant group differences with participants in the VE condition reporting more satisfaction with the study training than participants in the CAU condition, t(27) = 1.94, p = .06. Univariate analysis of variance models were conducted to examine the relationship between participant satisfaction regarding overall quality of the training and each participant’s highest percentage of CPRT components passed, F(2, 37) = 3.93, p = .03, with teachers demonstrating higher levels of fidelity reporting higher satisfaction with training.
Discussion
As technological supports for special education teachers become more available and affordable, it is important to explore possible professional development benefits. The research supporting video modeling and performance feedback separately as effective methods for increasing performance suggests the two may be combined for added impact to trainees. The current study evaluated the potential added benefit of video-enhanced coaching using an iPad and the Stage Pro application.
The results indicate use of the iPad to provide immediate video feedback to trainees may impact training outcomes and participant satisfaction with training procedures. Results indicate participating teachers demonstrate higher levels of FI of CPRT as compared to paraprofessionals, which aligns with expectations that trainees with more education and previous training will learn a new related skill more quickly. When evaluating the effect of immediate video feedback, FI of CPRT was not significantly different between conditions. However, paraprofessional use of CPRT may be more impacted by VE coaching than teacher use of the intervention, as indicated by greater group differences between conditions. These preliminary outcomes should be interpreted with caution due to several limitations.
This project had several limitations that may influence interpretation of the outcome data. First, the participant sample was small and was not sufficiently variable. Teachers and paraprofessionals participating in one training cohort as part of a larger effectiveness trial were included in this project. Participants represented number of individual school districts and a range of educational levels, ages, and years of experience working with children with ASD. However, all teachers and paraprofessionals were recruited from one large urban county in Southern California, and 96% of participants were female. It would be beneficial to replicate the study with a larger and more diverse participant sample.
An additional consideration when interpreting the outcomes involves the impact of the training protocol as part of the larger effectiveness trial. To facilitate teacher learning and application of training activities, all teachers selected target activities for the initial CPRT use. Coaching sessions were conducted during these activities. Therefore, these outcomes reflect participants’ FI of CPRT during specific activities and planned observation sessions.
Future research building on these findings is recommended. One area for continued exploration involves how VE coaching may differentially affect providers with varied training and credentials. For example, the current project identified a trend toward differential benefit of VE coaching for paraprofessionals. Future research could evaluate training outcomes with a larger and more diverse sample of educational providers added to further develop this literature. Paraprofessional educators are critical in provision of special education services and often provide a majority of direct service hours for students with disabilities (Giangreco et al., 2002). Moreover, they often receive limited training opportunities. Development of new technologies to train and support paraprofessionals toward the goal of improved intervention accuracy would greatly benefit students. Additionally, future research might explore the effect of delayed video-based feedback, wherein participants are recorded (or self-record) working with students, a coach reviews the video, and a virtual or in-person meeting is scheduled to review the video. This method may support more feasible adoption of VE coaching, allowing coaches and trainees to connect in multiple ways. In conclusion, although outcomes of this project are modest, they add to the very limited literature on how technology may be used to enhance in-service training for teachers.
Footnotes
Declaration of Conflicting Interests
The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This research was generously funded by the U.S. Department of Education Grant R324A130349 and Autism Science Foundation Research Enhancement Grant.