Awareness Is Not Enough: A Double-Blind Randomized Controlled Trial of the Effects of Providing Discipline Disproportionality Data Reports to School Administrators

Abstract

One commonly used strategy used in attempts to decrease racial disproportionality in school discipline across the country is sharing data with school administrators that discipline disparities are a problem in their schools with the assumption that it will increase attention to equity and improve outcomes. The purpose of this study was to assess the effects of providing monthly disciplinary equity reports to school administrators in 35 schools on levels of (a) disciplinary equity report viewing, (b) disciplinary equity, and (c) inclusion of disciplinary equity into school improvement plan goals. We used a small, double-blind randomized controlled trial in which half of the schools were randomly assigned to receive either monthly disciplinary equity reports or monthly general discipline reports. Results showed that schools receiving the equity reports had significantly increased rates of viewing equity reports but no meaningful change in disciplinary equity or equity goal setting.

Keywords

classroom management data decision-making disparities experimental design experimental research longitudinal studies multiculturalism performance feedback principals race racial equity school discipline

The disproportionate use of office discipline referrals (ODRs) for Black students is a longstanding concern in education (U.S. Government Accountability Office, 2018). Although there is evidence for effectiveness of some systems-level interventions, such as positive behavioral interventions and supports (McIntosh et al., 2018), implementing these interventions with fidelity requires administrator commitment and substantial resources; thus, district-wide implementation can be challenging. Accordingly, there is a tendency to seek efficient interventions that can be implemented quickly.

To motivate school administrators to take action, districts or state teams in multiple states have adopted the intervention of sending school administrators regular (e.g., monthly) reports showing the extent to which their schools have racial disparities in exclusionary discipline (Santiago-Rosario, 2019). Although minimal in terms of effort, sending equity reports has intuitive appeal as a way to draw attention to and signal the importance of addressing the problem (Kluger & DeNisi, 1996). Furthermore, there is evidence that school teams that use data more often for decision-making also have lower racial discipline disproportionality (Tobin & Vincent, 2011). Even so, at the school level, simply making equity reports available to school administrators on demand does not necessarily increase their viewing of discipline data (McIntosh et al., 2014). Moreover, sharing reports showing racial inequities could backfire, increasing resistance to equity efforts (Hetey & Eberhardt, 2018; Kluger & DeNisi, 1996).

Purpose of the Study

The study’s purpose was to provide a rigorous experimental test of providing disciplinary equity data reports to school administrators. We assessed the effects of providing monthly disciplinary equity reports compared to a control of monthly discipline summary reports to school administrators on (a) viewing disciplinary equity reports, (b) equity in school discipline, and (c) identifying disciplinary equity as a school improvement goal.

Method

Settings and Participants

A total of 35 public K–12 schools and their school administrators within three school districts in the U.S. Pacific Northwest participated in the study. The 26 elementary schools, five middle schools, and four high schools used the School Wide Information System (SWIS; May et al., 2018) to record and review discipline data. The average Black-White ODR risk ratio was 2.2 (range = 0–6.3), meaning that on average, Black students were more than twice as likely to receive one or more ODRs than White students.

Measures

Viewing of school discipline reports

Through SWIS, we extracted counts of discipline data reports (i.e., sets of graphs and tables) generated by school users per month. We standardized the data by using the metric of reports generated per 20 school days, roughly a month of school.

Equity in school discipline

We assessed equity in school discipline using the Black-White ODR rate difference, calculated by subtracting the ODRs per White student per day per month from the ODRs per Black student per day per month. This metric of disciplinary equity accounted for enrollment of each student group and varying school days in each month. A rate difference of 0 indicates perfect equity in school discipline, and positive values indicate higher ODR rates for Black students. The rate difference was selected because unlike risk ratios, it has a symmetrical distribution based on a standard scale (change in rates) and thus is more appropriate for computation of difference scores without transformation and can be calculated when one group receives 0 ODRs (Girvan et al., 2019).

School improvement plan goal setting

We obtained publicly available, annual school improvement plans for each of the participating schools from 2016–2017 and 2017–2018 that described their formal academic and behavior goals. We coded the plans for whether they included disciplinary equity as a goal (intercoder agreement = 88%).

Intervention

The intervention was a monthly email to school administrators containing one of the continuously available SWIS school discipline data reports (see http://www.pbisapps.org): either the (a) School Summary Report (control condition), which includes figures and tables showing general patterns of ODRs, or (b) School Equity Report (intervention condition), which includes figures and tables showing the extent of racial discipline disproportionality. No other communication or support was provided.

Procedure

Once recruited, schools were matched into pairs within districts based on their disciplinary equity (i.g., the Black-White ODR risk ratio) and randomly assigned to condition in January of Year 2. School administrators were emailed the reports on the first day of each month for the remaining 5 months of the school year (one school was lost from the sample due to not receiving its report in time). Figure 1 shows the participant flow diagram. The school and district administrators, lead author (principal investigator), second author (methodologist), and fifth and sixth authors were all blinded to condition until all analytic decisions were completed. Only the third and fourth authors, who generated and sent the reports, were aware of condition during the study, and they did not participate in analytic decisions.

Figure 1.

Study participant flow diagram.

Data Analysis

Report viewing and disciplinary equity

Analyses assessed the effects of receiving the emailed equity reports on (a) rates of viewing disciplinary equity reports and (b) actual discipline disparities. For both dependent variables, we subtracted Year 1 data for each month from Year 2 data for the same month to produce year-on-year monthly change over time. Thus, the analyses directly addressed the hypothesis that the schools that received equity reports compared to general reports would experience an immediate increase in their viewing of reports and disciplinary equity above levels from the same time in the previous year. The Supplemental Appendix available on the journal website provides a detailed description of data analysis.

School improvement plan goal setting

We evaluated each school’s school improvement plan for 2016–2017 and 2017–2018 to assess the extent to which the intervention condition group had an increased proportion of schools including goals to increase racial equity in school discipline. We conducted a chi-square test of proportions across groups to assess statistical significance.

Results

Report Viewing and Disciplinary Equity

Figure 2 presents the year-on-year mean differences for equity report viewing and disciplinary equity by condition, with 95% confidence intervals before and after intervention. Results showed a statistically significant increase in equity report views (p = .023, Hedges’s g = 0.65) after intervention for the intervention schools but no decrease in Black-White ODR differences, whereas control schools saw a decrease (p = .155, Hedges’s g = 0.09). The Supplemental Appendix available on the journal website provides detailed results.

Figure 2.

Mean year-on-year differences in equity report views per 20 school days (top) and Black-White ODR rate differences per day (bottom).

School Improvement Plan Goal Setting

Prior to intervention, no schools in either condition included school improvement plan goals referring to increasing racial equity in school discipline. After intervention, no control schools and one intervention school reported a school improvement plan goal of increasing disciplinary equity. A χ² analysis of posttest proportions showed that the one-school increase in equity goal setting was not statistically significant (χ² = 1.09, p > .05).

Discussion

Given the need to address disciplinary equity, district and state teams are increasingly engaging in the practice of sending school administrators reports of their disciplinary equity. Our study examined the effects of directly sending monthly disciplinary equity reports to school principals on (a) accessing data, (b) disciplinary equity, and (c) equity goal setting. The double-blind randomized controlled trial design with an attention control provided an opportunity to assess the effects of this common intervention with less susceptibility to reactivity or social desirability bias.

Regarding data report viewing, although they had already been available to school administrators, sending equity reports directly to them significantly increased how much they accessed and viewed the reports. Although the increase in equity report viewing was encouraging, the provision of reports did not lead to any meaningful changes in disciplinary equity or school improvement plan goal setting. As such, we view this intervention as ineffective in improving equity on its own. This study adds to previous research in performance feedback and equity data that simply sharing data showing inequities could have iatrogenic effects, especially if recipients perceive that they do not have sufficient control of the context to achieve equitable disciplinary outcomes (Bastable et al., in press; Hetey & Eberhardt, 2018; Kluger & DeNisi, 1996).

Limitations

Although tightly controlled and rigorous, this study has limitations worth considering. First, due to software limitations, we could not assess which users were accessing reports. As a result, it is unclear whether the increase in viewing came from school administrators or other school or district personnel. Second, the study was small and may have been underpowered to detect changes in ODRs.

Implications

Our findings indicate that sending equity reports directly to school administrators increases attention to data, although such effects do not appear to change equity in school discipline or goal setting. This study suggests that when provided evidence of disproportionality, school administrators are unlikely to take important actions, like setting equity goals in their school improvement plans. Yet it is noteworthy that providing reports increased views, which has been an elusive outcome (McIntosh et al., 2014). More research is needed to determine how this common intervention might be supplemented to increase goal setting or disciplinary equity.

Supplemental Material

McIntosh_Online_Supplement – Supplemental material for Awareness Is Not Enough: A Double-Blind Randomized Controlled Trial of the Effects of Providing Discipline Disproportionality Data Reports to School Administrators

Supplemental material, McIntosh_Online_Supplement for Awareness Is Not Enough: A Double-Blind Randomized Controlled Trial of the Effects of Providing Discipline Disproportionality Data Reports to School Administrators by Kent McIntosh, Keith Smolkowski, Cody M. Gion, Lauren Witherspoon, Eoin Bastable and Erik J. Girvan in Educational Researcher

Footnotes

Notes

ORCID iDs

Kent McIntosh

Eoin Bastable

Authors

KENT MCINTOSH, PhD, is Philip H. Knight Chair of Special Education at the University of Oregon, 1235 University of Oregon, Eugene, OR 97403-1235; kentm@uoregon.edu . His research focuses on racial equity in school discipline, positive behavior support, and implementation science.

KEITH SMOLKOWSKI, PhD, is senior research scientist at Oregon Research Institute, 1776 Millrace Dr., Eugene, OR 97403; keiths@ori.org . His research focuses on academic, social-emotional, and behavioral skills and disparities in education and prevention science.

CODY M. GION, PhD, is a special education program director at Gresham-Barlow School District, 1331 NW Eastman Parkway, Gresham, OR 97030; gion@gresham.k12.or.us . His research focuses on racial equity in school discipline and positive behavior support.

LAUREN WITHERSPOON, EdD, is a senior human resources generalist at VRM Mortgage Services, 4100 International Pkwy, Carrollton, TX 75007; lwither6@gmail.com. Her research focuses on diversity and inclusion, leadership, and equity in the workplace.

EOIN BASTABLE, PhD, is a postdoctoral research associate at Educational and Community Supports at the University of Oregon, 1235 University of Oregon, Eugene, OR 97403-1235; ebastabl@uoregon.edu . His research focuses on enhancing equity in school discipline, school coaching practices, and implementation science.

ERIK J. GIRVAN, JD, PhD, is an associate professor at the University of Oregon School of Law, 1221 University of Oregon, Eugene, OR 97403-1221; girvan@uoregon.edu . His research focuses on understanding and addressing causes of racial disparities in the legal system and related contexts, including school discipline.

References

Bastable

McIntosh

Falcon

S. F.

Meng

(in press). Exploring educators’ commitment to racial equity in school discipline practice: A qualitative study of critical incidents. Journal of Educational and Psychological Consultation.

Girvan

E. J.

McIntosh

Smolkowski

(2019). Tail, tusk, and trunk: What different metrics reveal about racial disproportionality in school discipline. Educational Psychologist, 54(1), 40–59. https://doi.org/10.1080/00461520.2018.1537125

Hetey

R. C.

Eberhardt

J. L.

(2018). The numbers don’t speak for themselves: Racial disparities and the persistence of inequality in the criminal justice system. Current Directions in Psychological Science, 27(3), 183–187. https://doi.org/10.1177/0963721418763931

Kluger

A. N.

DeNisi

(1996). The effects of feedback interventions on performance: A historical review, a meta-analysis, and a preliminary feedback intervention theory. Psychological Bulletin, 119(2), 254–284.

May

S. L.

Ard

W. I.

Todd

A. W.

Horner

R. H.

Glasgow

Sugai

Sprague

J. R.

(2018). School-Wide Information System (Version 6.0) [Computer software]. Educational and Community Supports, University of Oregon. http://www.pbisapps.org

McIntosh

Eliason

B. M.

Horner

R. H.

May

S. L

. (2014). Have schools increased their use of the SWIS School Ethnicity Report? [PBIS evaluation brief]. OSEP TA Center on Positive Behavioral Interventions and Supports.

McIntosh

Gion

Bastable

(2018). Do schools implementing SWPBIS have decreased racial disproportionality in school discipline? [PBIS evaluation brief]. OSEP TA Center on Positive Behavioral Interventions and Supports.

Santiago-Rosario

M. R.

(2019). State and district team provision of disciplinary equity data to school administrators [Unpublished report]. University of Oregon.

Tobin

T. J.

Vincent

C. G.

(2011). Strategies for preventing disproportionate exclusions of African American students. Preventing School Failure, 55, 192–201. doi:10.1080/1045988X.2010.532520

10.

U.S. Government Accountability Office. (2018). Discipline disparities for black students, boys, and students with disabilities. Report to congressional requesters, U.S. House (Report GAO-18-258).

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.52 MB