Effects of SRSD With Social Skills Prompts on the Writing Skills and Problem Behaviors of Students With Emotional and Behavioral Disorders

Participants

Three Black male students from two specialized behavior support (SBS) classrooms for students with EBD participated in the study. Students met the following inclusion criteria (a) were in grades 3–5; (b) had an individualized education program (IEP) with at least one social skills/behavioral goal; (c) could write a complete sentence independently; and (d) read on a first grade level or higher. We used Curriculum-Based Measurement (CBM; Shinn, 1989) for reading and writing to determine students’ eligibility for criteria (c) and (d) above. We administered oral reading fluency measures and used grade level standards to determine students’ reading level and used total words written (TWW) to determine students’ writing abilities. For eligibility screening (i.e., prebaseline), we used reading probes from EasyCBM (Anderson et al., 2014) and writing prompts from Harris et al. (2008).

Rashad was a 10-year-old fourth-grade student receiving special education services under the other health impairment (OHI) eligibility. He had a diagnosis of attention-deficit hyperactivity disorder (ADHD) and oppositional defiance disorder (ODD) and had a behavior intervention plan (BIP) addressing compliance to directions, accepting responsibility, and completing assigned tasks promptly. Student behavior documentation indicated that he was easily frustrated with tasks and would often express this through storming off and engaging in destructive behaviors (e.g., flipping chairs and desks over, ripping up papers, pushing and shoving others). The average prebaseline CBM reading score for Rashad was 165 words per minute, which placed him above third-grade reading level.

Cory was 10 years old in the fifth grade, who received special education services under the OHI eligibility. He had a medical diagnosis of ADHD and ODD and took medications for his ADHD diagnosis. Cory’s IEP goals addressed asking for assistance and/or getting teacher’s attention appropriately, using appropriate language when frustrated, and avoiding aggressive physical contact during a conflict. Cory’s teacher stated that Cory engaged in inappropriate attention-seeking behaviors such as pouting, refusing to complete assignments, and verbally attacking other students who received attention from the teacher. The average prebaseline CBM reading score for Cory was 145 words per minute, placing him above third-grade reading level.

Kasim was a 10-year-old fourth-grade student receiving special education services under the Emotional Disturbance eligibility with diagnoses of ADHD, ADD, and Disruptive Mood Dysregulation Disorder. Kasim’s IEP goals addressed coping skills in frustrating situations, avoiding physical and verbal aggression, taking responsibility for his actions, complying with adults’ requests, and managing his time well during small group and independent activities. He had a BIP addressing these issues. Kasim’s teacher stated that he often engaged in bullying behavior toward peers and instigating other students by calling them names. His physical aggression escalated when he perceived himself to be in trouble and when he was asked to go to an isolated spot to “cool down.” The average prebaseline CBM reading score for Kasim was 161 words per minute, which placed him above third-grade reading level.

Setting and Interventionist

The study took place in a Title 1 K–5 school, serving 717 students, in a suburban area in a southeastern state. The ethnicity breakdown within the school was 53% Black, 31% Hispanic, 8% Asian, and 6% White; 98% of students received free or reduced-price lunch and 10% of students received special education services. The school had an established school-wide positive behavioral interventions and supports system, with schoolwide contingencies to reinforce behavioral expectations. Teachers of both SBS classrooms used the Boys Town Educational Model (BTEM; Fluke et al., 2013) for behavior management. For literacy instruction, the teachers followed the Balanced Literacy (Lynch, 2018) framework, a prescribed district-wide reading program. Classroom observations occurred during the writing block of instruction in both classrooms. The first author served as the primary data collector and interventionist, and all SRSD intervention sessions and assessments took place outside the students’ classroom in a designated room within the school.

Dependent Variables and Measurement

There were four dependent variables, including (a) number of genre elements included in students’ written responses, (b) length of written responses, (c) quality of written responses, and (d) problem behavior. The score of genre elements served as the primary dependent variable.

We used scripted instructions during writing assessment sessions to ensure uniformity; participants were instructed to take a position regarding a behavior or action from the writing prompt. They also were informed that they would not receive help during the planning and writing of the essay, but they would receive help with spelling if they requested for it. During baseline assessment, participants were instructed to try their best to respond to the writing prompts. During post-intervention assessment, participants were reminded to use strategies learned and to create a graphic organizer to assist them in planning for the writing. The first author read all written prompts to each participant and repeated a prompt at participant’s request. Instructions to participants for video prompts were the same as those for written prompts, except for the “read aloud” portion. Although not timed, participants wrote for an average of 2–3 min per prompt during baseline and wrote for at least 15 min with some participants taking up to 45 min to complete their responses during post-intervention and maintenance conditions.

Interrater Agreement

Two doctoral students served as interraters. For the genre elements score, we calculated the interrater agreement (IRA) using the item-by-item comparison method by dividing the number of agreed items by total number of items and multiplying by 100. For length of written responses, we calculated IRA using the gross method (i.e., smaller number by the larger number and multiplied by 100) by comparing the TWW data obtained from two scorers. For quality of written responses, the first author randomly selected a total of six writing samples (three from baseline and three from post-intervention) per participant for the two raters to score. IRA was calculated using the percent agreement method. To collect IRA data for the problem behavior, two raters observed each participant at the same time using split headphones to listen to the same audio prompt produced by the SIT app on the iPhone. We calculated IRA for problem behavior using the interval-by-interval comparison by dividing the number of agreements by the total number of agreements plus disagreements and multiplying by 100.

For participants’ genre elements and length of written responses, an interrater scored an average of 74% (56–100%) of participants’ written responses across conditions with the overall mean IRA being 99.7% (99–100%). For writing quality, the overall mean IRA was 99.4% (98.8–100%). We collected IRA data for participants’ problem behavior for an average of 42.8% (25–75%) across conditions and participants and the overall mean IRA was 98.1% (94–100%).

Materials

Procedural Fidelity

Experimental Design and Procedures

We used a single-case, multiple probe across participants design (Horner & Baer, 1978) and collected data across four experimental conditions (i.e., baseline, post-intervention, maintenance, and choice). We used participants’ scores of genre elements (i.e., primary dependent variable) to make condition change decisions. There was a minimum of five baseline data points with three consecutive baseline sessions showing stability immediately prior to a participant receiving SRSD instruction. During SRSD instruction, writing data were collected, but for progress monitoring purposes only. There was a minimum of five post-SRSD data points. To ensure unbiased access to written prompts and to foster engagement throughout the study, participants were allowed to randomly pull three prompt cards from an envelope, read all the prompts, and select one they preferred during each assessment. Generalization was embedded within each condition and involved the use of video prompts instead of written prompts.

Social Validity

We collected social validity data from participants and their teachers before and after the intervention to measure acceptability of SRSD and to gauge changing perceptions regarding the intervention and student behavior. The teachers completed an adapted version of the Intervention Rating Profile-15 (IRP-15; Witt & Elliott, 1985) pre- and post-intervention questionnaires. Both questionnaires consisted of 15 items rated on a 6-point Likert-type scale (i.e., strongly disagree, disagree, slightly disagree, slightly agree, agree, strongly agree). These items assessed teachers’ perceptions regarding the writing performance of the participants, and the effects and benefits of the SRSD intervention for improving students’ writing skills and/or social behaviors. The participants’ pre-intervention questionnaire consisted of seven items and the post-intervention questionnaire consisted of 12 items, all on 6-point Likert-type scale (i.e., strongly disagree, disagree, slightly disagree, slightly agree, agree, strongly agree) with an option for comments. Items included on participants’ questionnaires assessed their attitudes toward writing, perspectives regarding their behaviors, and perspectives regarding the impact of writing about behavior on their own social behaviors and writing skills.

Genre Elements in Written Responses

All participants’ performance in genre elements based on the rubric scores showed low and stable level during baseline with a mean score of 1.9 (range 1–2) for Rashad, 3 (range 2–4) for Cory, and 4 (range 3–6) for Kasim. During post-intervention, all participants showed substantial improvement in level by 68% with a mean score of 18 (range 17–20) for Rashad, 20 (range 16–21) for Cory, and 22 (range 16–23) for Kasim and with an overall high level of stability (except for an outlier for Cory and an outlier for Kasim). During the maintenance condition, Rashad and Cory both showed slight reduction in rubric score from that during post-intervention but remained markedly higher than baseline scores. Rashad’s choice was a video prompt with a score of 16. Both Rashad (during choice) and Cory (during maintenance) also completed one in-class session of teacher-administered writing assessment, and their genre element scores were slightly below their other scores during the same condition. Visual analysis of baseline and post-intervention data shows there were clear, but delayed, effects of SRSD on all three participants’ inclusion of genre elements (see Figure 1). Generalization data from video prompts were consistent with the data pattern from the written prompts across conditions.

OPEN IN VIEWER

Figure 1.

Participants’ Genre Elements Rubric Score Across Experimental Conditions.

Length of Written Responses

All participants substantially increased TWW from baseline to post-intervention, and this trend continued to the maintenance condition for Rashad and Cory (see Figure 2). Visual analysis of baseline data revealed stable and consistently low scores in terms of TWW across participants with a mean of 7 (range 5–11) for Rashad, 16 (range 13–21) for Cory, and 18 (11–31) for Kasim. Post-intervention data showed an immediate increase in TWW with stability and no overlapping data points between baseline and post-intervention. During maintenance, Rashad showed a slight decreased level in both TWW compared to his performance during post-intervention and Cory maintained his TWW from the post-intervention condition. The in-class assessment data for Rashad (during choice) and Cory (during maintenance) showed similar or slightly above other data points in the same condition. Generalization data for TWW from video prompts were mostly consistent with the data pattern from the written prompts across conditions for all participants.

OPEN IN VIEWER

Figure 2.

Participants’ Total Words Written (TWW) Across Experimental Conditions.

Quality of Written Responses

Results from the holistic scoring revealed moderate improvements in the quality of written products post-intervention across participants. Two scorers separately scored six randomly selected written responses per participant (three from baseline and three from post-intervention). The mean score for baseline responses was 1.1 (unsatisfactory) while the mean score for post-intervention responses was 2.3 (uneven). Specifically, baseline mean scores for Rashad, Cory, and Kasim were 1, 1, and 1.2 while their post-intervention mean scores were 2, 2.4, and 2.5. A closer examination of individual responses offered some interesting insights regarding each participant’s perception of behavior as it relates to consequences. Overall, all participants were more elaborate and thoughtful in their responses post-intervention. For example, Kasim seemed to understand the consequences of certain behaviors but still thought the student(s) in the specific prompts were justified in acting the way they did. On the contrary, Rashad’s responses indicated that the main reason for not engaging in the target behaviors was to avoid being in trouble. In some if Cory’s responses, while he provided alternative behaviors, he would also include statements acknowledging the challenges in engaging in the (desirable alternative behavior(s)).

Problem Behavior

Baseline data depicted a wide range (low to high) of problem behavior occurrences across participants with a steep increasing trend for Rashad (mean 71.4%, range: 56.2–91.6%), a decreasing trend for Cory (mean 40.4%, range 21.0–72.5%), and an overall increasing trend for Kasim (mean 12.5%, range 0–37.5%). During post-intervention, all participants showed a decrease in the percentage of levels of problem behaviors with a mean of 28% (range 18.6–39.2%) for Rashad, 8.3% (range 0–17.5%) for Cory, and 5.7% (range 0–11.3%) for Kasim. During the maintenance condition, both Rashad (mean 16.7%, range 8.8–23.8%) and Cory (mean 0.4%, range 0–1.3%) maintained low levels of problem behavior. Although there were reductions in participants’ problem behavior from baseline to post-intervention, a functional relation did not exit due to Cory’s and Kasim’s baseline data patterns (see Figure 3).

OPEN IN VIEWER

Figure 3.

Participants’ Percentage of Intervals of Problem Behaviors Across Experimental Conditions.

Social Validity

Both teachers had an optimistic view of the potential effectiveness and appropriateness of SRSD for addressing students’ writing needs pre-intervention (with ratings of 5 [agreed] or 6 [strongly agreed]). They affirmed this view post-intervention with both stating that (a) the intervention was effective in remediating writing skills and had the potential to remediate behavioral issues, (b) SRSD could be feasibly used in their classrooms, and (c) they were willing to incorporate social skills prompt/training within their ELA/writing lesson plans.

Cory and Kasim agreed or strongly agreed that SRSD intervention helped improve their writing skills and that writing about behavior improved their classroom behavior. Rashad agreed that SRSD helped him improve his writing skills but did not improve his social behaviors. None of the students elaborated on their responses.

Limitations and Directions for Future Research

There are several limitations in the study that warrant future investigations. First, the study was conducted toward the end of school year, resulting in (a) only one participant being exposed to all experimental conditions and (b) limited data collection on the participants’ problem behavior due to scheduling conflicts and testing. Future research is needed to investigate the full effects of SRSD with social skills prompts on both writing skills and problem behavior of students with EBD across all experimental conditions. Second, the first author served as the interventionist and primary data collector for participants’ problem behaviors. Over the course of the study, all three participants developed positive relationships with the first author. It is possible that some of the behavioral improvements noted post-intervention could be due to these relationships as well the first author’s presence in the classroom during observations. Future studies should explore data collection instruments that capture student behavior using naïve observers through unobtrusive observations. Third, most of the post-intervention and maintenance assessments were completed during the participants’ end-of-year school assessments. Having to participate in two assessments (one for school and one for the study) within the same day might have affected participants’ performance. Similarly, the location and times (morning vs. afternoon) for post-intervention writing assessments were inconsistent, which might have contributed to lower scores for some participants (e.g., Rashad and Cory were more focused and productive in the morning). It is essential for future research to control for possible confounding variables such as scheduling, locations, and time of assessment. Fourth, although the use of video prompts provided participants with an alternative to written prompts, which aligns with UDL’s principles of representation and engagement, we did not collect data evaluating the impact of video prompts versus written prompts on participants’ engagement or motivation. Furthermore, only one participant (Rashad) entered the choice phase for one session. Despite that he chose the video over written prompt, there was no data on the duration of engagement in writing or level of motivation. Future studies should investigate fully the impact of video versus written prompts through a comparative design and collecting social validity data specifically reflecting participants’ preferences on the use of video prompts. Finally, although common in SRSD studies, data collection during post-intervention prevented us from claiming immediate effects that support a clear functional relation. Specifically, the post-intervention data reflected the effects of SRSD instruction after 12–18 SRSD sessions; therefore, visual analysis of the data can only indicate delayed effects without an immediate, detectable change. Researchers in future studies may consider the addition of intermittent probes during SRSD instruction.

Implication for Practice

Findings of this study have important implications for practice. First, teachers working with students with EBD are often faced with the challenges of meeting both academic and behavioral needs of their students. SRSD instruction using social skills prompts has the potential to provide a solution for part of the dilemma by helping teachers simultaneously provide writing instruction and teach social skills. Both teachers stated anecdotally that they could see the value of such intervention in their daily instruction with students with EBD. Second, teaching students with EBD opinion writing using social skills prompts provides an opportunity for them to express their feelings and thoughts regarding actions and/or behaviors in which they tend to engage. This approach has the potential to offer insights on the students’ actual thinking processes regarding their classroom behavior. For example, from Kasim’s responses, it was clear that he understood the school rules well and had a good grasp of inappropriate behavior and consequences. Yet, he exhibited more problem behaviors than any other participants outside of the observations. Teachers could use this knowledge to investigate the cause and/or function of Kasim’s problem behavior, which was not due to a lack of understanding of the rules or consequences. On the contrary, judging from his responses, Rashad did not have a clear understanding of rules and consequences. For most of his responses, Rashad gave “getting in trouble” as the reason why the behavior/action depicted in the prompt was inappropriate. He then would give examples of getting in trouble (e.g., getting referral and being suspended). Based on Rashad’s responses, he would likely benefit from explicit instruction of socially acceptable behavior/actions with examples and non-examples, alongside reasons beyond getting in trouble. For students like Kasim and Rashad, using SRSD opinion writing with social skills prompts has the potential to help practitioners understand the root cause of persistent problem behaviors; this in turn could inform their decisions regarding interventions.