Using a Text-as-Data Approach to Understand Reform Processes: A Deep Exploration of School Improvement Strategies

Policy Background of School Improvement for Underperforming Schools

This study explores reform strategies that un-derperforming schools planned and implemented to improve student achievement and reduce absenteeism, focusing on school reform efforts during the era of No Child Left Behind waiv-ers and SIGs from 2010 to 2016. We focus on Washington State because this state largely adopted federal policy requirements and used three widely used policy instruments to turn around its underperforming schools: accountability and monitoring, funding/grants, and technical assistance to schools provided by improvement coaches (Hurlburt, Le Floch, Therriault, & Cole, 2011; Hurlburt, Therriault, & Le Floch, 2012).

During this period, Washington State implemented a multitiered identification and support system to remedy schools’ underperformance. The state used three school improvement designations: focus schools, priority schools, and SIG schools. Focus schools were defined as those in the lowest 10% of subgroup performance based either on the 3-year average for subgroups on state assessments in English language arts and math (combined) or on an adjusted 5-year cohort graduation rate that was less than 60%. The state defined priority schools as those in the lowest 5% based on all students’ performance across several criteria. The majority of schools identified had a 3-year average proficiency level for all students on state assessments in English language arts and math (combined) that was less than 40%, or in the lowest 5% based on the achievement index score, ¹ or had an adjusted 5-year cohort graduation rate for all students that was less than 60%. SIG schools have to also be identified as priority schools. Other factors that could be considered when selecting SIG schools included geographic location, school size, and commitment and capacity to use SIG funds to substantially raise student achievement.

Once a school was identified as a focus, priority, or SIG school, typically that designation remained in place for 3 years. Schools would receive supplementary funding on top of their regular budgets; SIG schools were primarily funded through federal grants, whereas priority and focus schools were primarily funded through state funds. SIG schools were also required to follow federally prescribed school reform models. Almost all the SIG schools in Washington State adopted either the transformation model or the turnaround model. The transformation model requires replacing the principal, implementing curricular reform, and introducing teacher evaluations (based, in part, on student performance) into personnel decisions (e.g., rewards, promotions, retentions, and firing). The turnaround model includes all of the transformation model requirements, along with replacing at least 50% of the staff. Priority and focus schools received less, although still substantial, funding and assistance. Because they received state funds, they closely followed the state’s guidelines on school turnaround, which are largely consistent with SIG models but have less strict requirements for replacing school personnel and tying educator evaluations to student growth. Overall, several features of these reform efforts—such as their relatively long duration, their systematic and dramatic approach to change, and the substantial influx of resources they prompted—make them a fertile ground for researchers and policymakers to learn useful lessons about school improvement strategies that move the needle for students.

To date, conventional evaluation studies of these school turnaround models have shown mixed effects on student achievement. Studies of SIG programs in California and Massachusetts using either regression discontinuity or difference-in-differences models found that the programs had positive effects on student achievement (Dee, 2012; Papay, 2015; Sun, Penner, & Loeb, 2017). However, a U.S. Department of Education study, using data from 22 states, found largely null impacts on test scores, high school graduation, and college enrollment for the cohort of SIG schools funded in 2010 (Dragoset et al., 2017). In several states that won Race to the Top funding or received No Child Left Behind waivers, research has yielded mixed evidence on the effectiveness of their school turnaround reforms. Heissel and Ladd (2018) found negative effects from the programs in North Carolina, whereas Zimmer, Henry, and Kho (2017) found some positive effects in Tennessee, particularly among Innovation Zone schools that were governed and managed separately by three school districts. Two companion studies in Kentucky and Louisiana showed opposite findings: Over each of 3 years, Louisiana’s focus school reforms had no measurable impact on school performance (Dee & Dizon-Ross, 2017), whereas Kentucky’s focus school reforms led to substantial improvements in both math and reading achievement (Bonilla & Dee, 2017).

Some of the disparities in these results may be explained by sample selection and estimation strategies, as Guthrie and Henry’s (2016) work in North Carolina illustrates. However, a more plausible explanation for the differences in findings across studies is the variation in the design and implementation of school reform interventions across schools, districts, and states (Dragoset et al., 2017). Given the state of the literature, it is apparent that another efficacy study using a “black box” approach would not be sufficient to inform future school improvement efforts. Rather, schools and districts need studies that use novel data and methods to investigate school improvement processes to generate actionable knowledge that can guide policy and practice directly.

Text Data

To develop a more detailed understanding of the mechanisms of change in schools, we analyze data on school reform processes collected through the Comprehensive School Improvement Plan-ning and Implementation Reports (CSIPIRs) by Washington State’s education agency, the Office of Superintendent of Public Instruction (OSPI). CSIPIRs are submitted by schools through a Web-based platform called Indistar. The state has specified seven principles of student and school success to guide a school’s use of the reports: strong leadership; staff evaluation and professional development; expanded time for student learn-ing and teacher collaboration; rigorous, aligned instruction; use of data for school improvement and instruction; safety, discipline, and social, emotional, and physical health; and family and community engagement. When using the Indistar system to build a school improvement plan, schools are required to select at least one indicator (from a bank of indicators provided by Indistar) for each of the seven principles. Selecting an indicator allows a school to see the evidence supporting that indicator, with the aim of providing evidence-based practices for schools.

The variation across schools then derives from the specific reform strategies that individual schools develop themselves, along with their implementation of these strategies. Once an indicator has been selected, a school is asked to describe and rate its current level of practice and to establish goals for what it will look like in practice if this indicator is fully achieved. Next, the school is asked to lay out specific tasks needed to achieve each goal, including designating the individual(s) responsible for the goal, the target completion date, and the frequency of the task. (See Figure 1 for an example.) A school is allowed to plan as many tasks as needed to achieve a goal. The school is also required to update its report periodically to mark the completion date for completed tasks, add comments on implementation, and explain how the school plans to sustain the task. This structured template helps schools to develop detailed information about their school improvement plans and implementation.

OPEN IN VIEWER

Figure 1.

Examples of tasks written in the Comprehensive School Improvement Planning and Implementation Reports (CSIPIRs).

These reports provide a useful source of data on school improvement actions and activities for several reasons. They are not merely planning reports, but rather capture what schools actually implemented, as indicated by the date markers for task assignment and task completion. They are not merely compliance reports, either. In our interviews with personnel at 10 schools, many school leaders reported that because they could access the Indistar online tool anytime and anywhere, this reporting format was more convenient for coordination and communication among school staff than the old-fashioned paper format. Moreover, the Indistar system provides evidence associated with each indicator. School leaders indicated that they had developed more evidence-based planning and implementation with the Indistar system in place than they had before. The Indistar online tool and CSIPIRs were reported to help schools develop shared language and strategies among staff members. In addition, schools had little incentive to present lofty goals that they might be unable to achieve later because the reports were submitted after schools had been identified for improvement and had already received federal or state funds, rather than being written during the grant competition stage. Furthermore, the state does not withhold funds, or hold schools accountable in other ways, for less ambitious plans or fewer tasks completed. The reporting is thus a nonconsequential requirement. Finally, the state provides coaches to identified schools to (a) support the development and implementation of improvement plans and (b) serve as a third-party monitoring mechanism. The above contexts provide us confidence in the validity of these data. As discussed later in the article, we also established our own procedures to further assess the quality of the text data using alternative data sources (e.g., interview data).

All identified schools in Washington State were required to submit CSIPIRs as of the 2011–2012 school year. We obtained these reports through a research–practice partnership with OSPI, then extracted information from these reports, from originals in PDF format into Excel spreadsheets using Python. Among the CSIPIR data that we received and cleaned (from 2011–2012 to 2015–2016), 55.2% of all unique tasks proposed were marked as completed with specific completion dates. ² Incomplete tasks were either removed in later years’ reporting or never marked with a completion date. Our subsequent analyses use only these completed tasks because they represent the completion of resource allocation and schools’ committed actions. However, we acknowledge that our analyses using only completed tasks may cause upward bias in the results because schools may have abandoned strategies that they deemed were not yielding positive effects on student outcomes.

For some SIG schools, we used annual reports if they did not submit CSIPIRs, particularly in the early reform years (e.g., 2011–2012). Only SIG schools were required to submit annual reports, which have structured reporting elements similar to those in CSIPIRs but were submitted only at the end of the school year and included a summary of the year’s completed initiatives. SIG annual reports were used by the state school improvement coaches as part of their validation process of CSIPIR data. All tasks mentioned in the annual reports would be counted as completed tasks, per the requirement of the reports. Because schools are asked to submit three CSIPIRs per year, the total number of reports yielded from these schools by the end of the 2015–2016 school year is 2,873 CSIPIRs and 85 SIG annual reports ³ (in the early years, some schools submitted two CSIPIRs per year). We have 25,486 completed tasks from CSIPIRs and 510 tasks from SIG annual reports. Texts with less than seven tokens were removed (about 5.8% of the total sample), however, because they provide little useful information on what schools actually did and are not suitable for the LDA model. After trimming those from the corpus, we have 23,997 unique tasks from CSIPIRs and 502 tasks from SIG reports. The next section describes our approach to deriving quantitative measures of reform strategies from these data.

Text Analysis

Schools conceived of their own reform strategies and used their own words to report the tasks that they undertook and completed. Our goal of text analysis is to identify schools’ fine-grained reform strategies as well as the extent to which they were implemented in each school using this large volume of unstructured textual data. After carefully considering many text-as-data methods, including dictionary methods, clustering, and supervised methods, the topic modeling approach, specifically LDA (see Blei, 2012; Blei, Ng, & Jordan, 2003), stands out as the most appropriate one. Rather than requiring researchers to condition on known constructs or topics beforehand, LDA uses modeling assumptions and properties of texts to generate a set of topics and simultaneously assign tasks to those topics. It is particularly useful when learning the patterns of text data or trying to identify topics that are theoretically meaningful but perhaps understudied or previously unknown. Using LDA, we are able to condense thousands of diverse CSIPIR text entries into a limited number of discrete and sensible categories, or topics, and simultaneously derive the composition of topics for each text entry.

LDA is a generative statistical model that identifies the latent topics and corresponding proportions that compose a document. LDA assumes that each document (a reform task in our setting) is a mixture of topics. For each task, π_ik represents the proportion of task i dedicated to topic k. Each task collects the proportions across topics, as π i = ( π i 1 , π i 2 , … , π i K ) . We used an R package (-stm-) to implement the analysis.

LDA allows us to estimate both topic prevalence (e.g., the proportion of a task discussing each topic) and latent content constructs with observed information about school improvement processes. In contrast to other clustering methods that assign documents to only one topic (or latent construct), LDA analysis aims to discover the latent topics across all tasks and represents a given task as a set of topic weights, rather than assigning each task to a single topic. The topic weights, as indicated by the proportion of texts, can then be aggregated across all task entries for a given school in a given year to produce an overall assessment of task emphasis. The topic proportion indicates the prevalence of reform strategies in schools and reflects a mix of factors, such as the time that schools spent on a reform topic, the importance of the reform topics, or the depth that a school engaged in this reform topic. Our measure is similar to that used in a U.S. Department of Education study of the proportion of practices under SIG reform topic areas (Dragoset et al., 2017) collected via surveys, but with greater accuracy and comprehensiveness. Below, we describe how we processed the raw text data, derived a longitudinal measure of topic prevalence at the school-year level, and used alternative data sources (such as human-coded metrics and interview data) to assess the validity and robustness of the results.

Sample and Structured Administrative Data

We then linked these reform process measures with school contextual and student outcome measures from state administrative data sets to examine (a) which schools and communities adopted which types of reform strategies and (b) how the reform processes explain the variation in the effects of school improvement efforts on student outcomes. We used both student absenteeism and achievement on state standardized tests to measure school improvement outcomes. OSPI collects data on four types of absences: full-day excused, part-day excused, full-day unexcused, and part-day unexcused. A full-day absence is defined as missing more than 50% of instructional time during a day. In our analyses, we combine excused and unexcused absences because such division can be imprecise if students, parents, or schools treat them as fungible. Along with running analyses on the raw numbers of partial and full days students missed, we created a chronic absenteeism measure for students who were absent for 15 or more full school days. ⁴

Achievement on state standardized test scores is standardized within a given grade, year, and test to account for differences in tests across grade levels, subjects, and years. Tests include Smarter Balanced Assessments in math and English language arts in Grades 3 to 8 and 11, Washington State’s Measurements of Student Progress tests, and end-of-course exams in Grades 9 to 12, among others. If a student took more than one math test in a year (e.g., geometry and algebra), we took the average of the standardized scaled scores as the measure for this student.

Table 1 summarizes the number of schools identified as SIG, priority, and focus schools for which we have both text data and student outcome measures in either absenteeism or achievement during school years 2010–2011 through 2015–2016. In 2010–2011, 18 SIG schools were identified; however, because the state did not adopt the Indistar system until 2011–2012, we do not have their CSIPIRs or SIG annual reports. Although in 2011–2012 the state identified 28 SIG schools, we are missing either the reports or the student outcome measures for four of those schools, so there are 24 SIG schools in our analytic sample. Since 2012–2013, more priority and focus schools were included in the analysis over time. In total, our sample includes 318 schools and 623 school-year observations. As shown in the online Supplemental Table S1, the final analytic sample is representative of all identified schools in terms of prereform characteristics and performance.

OPEN IN VIEWER

Table 1

Number of Treatment Schools That Have Test Score Data by Reform Type and Cohort

School year	SIG	Priority	Focus
2015–2016	0	68	116
2014–2015	0	56	131
2013–2014	10	34	79
2012–2013	24	15	66
2011–2012	24	0	0
2010–2011	0	0	0

Note. This table includes the number of schools that have text data and one outcome measure (e.g., either achievement or chronic absenteeism). Our analysis only includes the first designation of a given school. SIG = School Improvement Grant.

Table 2 summarizes the characteristics and performance of these identified schools as well as nonreform schools in the state. Identified schools, on average, serve larger proportions of historically underserved students—including students of color, low-income students, and homeless students—than do nonidentified schools. Students in identified schools are also relatively lower achieving and more likely to be chronically absent.

OPEN IN VIEWER

Table 2

School-Level Characteristics and Performance by Reform Type

	SIG	Priority	Focus	Nonreform
% White	0.27	0.39	0 .39	0.56
	(0.26)	(0.32)	(0.25)	(0.29)
% African American	0.13	0.06	0.05	0.04
	(0.19)	(0.11)	(0.09)	(0.07)
% Hispanic	0.36	0.34	0.42	0.16
	(0.32)	(0.31)	(0.26)	(0.19)
% Asian	0.06	0.03	0.04	0.05
	(0.09)	(0.06)	(0.07)	(0.08)
% Pacific Islander	0.01	0.01	0.01	0.01
	(0.02)	(0.02)	(0.02)	(0.02)
% Native American	0.10	0.11	0.03	0.02
	(0.2)	(0.22)	(0.08)	(0.08)
% Multiracial	0.05	0.06	0.05	0.05
	(0.05)	(0.05)	(0.04)	(0.05)
% Eligible for free or reduced-price lunch	0.78	0.71	0.69	0.41
	(0.19)	(0.26)	(0.17)	(0.27)
% English language learner	0.36	0.29	0.36	0.17
	(0.26)	(0.28)	(0.23)	(0.19)
% Homeless	0.05	0.06	0.05	0.03
	(0.03)	(0.07)	(0.06)	−0.04
% Special education	0.13	0.15	0.14	0.16
	(0.05)	(0.11)	(0.05)	(0.21)
Prior student academic achievement	−0.6	−0.57	−0.33	N/A
	(0.19)	(0.30)	(0.23)
Student academic achievement	−0.42	−0.47	−0.31	−0.05
	(0.25)	(0.37)	(0.29)	(0.44)
Full-day absences	12.66	9.94	9.89	7.18
	(5.85)	(6.12)	(4.67)	(5.32)
Part-day absences	5.83	4.09	4.81	2.89
	(7.86)	(5.37)	(5.85)	(4.59)
% Chronic absenteeism—full day	0.41	0.33	0.33	0.24
	(0.16)	(0.17)	(0.12)	(0.17)
N (school year)	86	200	486	17,404

Note. The data were from 2010 to 2016. This table includes schools’ first reform type identifications. Nonreform schools did not have prior student outcome measures per definition because they did not have a reform start date. The sample sizes reported here only reflect the analytic sample that provides all the demographic information. The sample sizes for the absence measures are 39, 200, 485, and 9,465 for SIG, priority, focus, and nonreform schools, respectively. Standard deviations are reported in parentheses. SIG = School Improvement Grant.

Model Diagnostics Statistics

The LDA approach requires researchers to specify the number of topics. OSPI specified seven principles of school improvement; Bryk, Sebring, Allensworth, Luppescu, and Easton (2010) also identified five essential supports for school improvement. These categories of school improvement efforts are broad (such as “building school leadership teams” or “developing teacher capacity”) and do not discuss specific strategies schools might employ. Aiming to discover new and more specific reform strategies, we began the modeling process by specifying 10 to 30 topics, and then, we used diagnostic statistics to aid our model selection, as illustrated in Figure 2.

OPEN IN VIEWER

Figure 2.

Topic model diagnostic statistics.

The first diagnostic statistic is semantic coherence, or the degree to which words are internally consistent. In Figure 2a, the y axis indicates log probabilities. Large negative values indicate that top words do not co-occur often, whereas values closer to zero indicate that top words tend to co-occur more often. In our case, the 10-topic model has the highest semantic coherence, whereas the 15- and 20-topic models are slightly worse and the 25- and 30-topic models substantially worse.

The second diagnostic statistic we use, exclusivity, summarizes the harmonic mean of the probability of all the top words under a topic and the exclusivity to that topic (Roberts et al., 2014). The bigger the value on the y axis of Figure 2b, the better the model performs in terms of separating one topic from the others. In our case, the 20-topic model is better than the 15-topic model, and both of these are much better than the 10-topic model.

Given that a topic that is both cohesive and exclusive is more likely to be semantically useful (Roberts et al., 2014), the 15- and 20-topic models appear to provide better balance between semantic coherence and exclusivity than the 10-, 25-, or 30-topic model. These statistics are helpful only to the extent that they provide us with general guidance on model selection. The coherence and exclusivity of the overall model do not directly indicate whether each topic of the model represents a conceptually and practically meaningful “theme.” To assess this, we need human coders to further evaluate the content validity of the topics.

Content Validity Check

Content validity denotes the extent to which our topics identify coherent sets of tasks and measure conceptually sound constructs. We used two expert coders on our research team to assess the content validity of our topics. ⁵ Having expert coders who have both practical and theoretical knowledge is critical to assess topics’ content validity. We then developed a rubric to rate topic coherence and the extent to which the coherent topic is practically meaningful and consistent with the literature (included in the online Supplemental Table S3; Mimno, Wallach, Talley, Leenders, & McCallum, 2011). Using a scale of 1 to 4, the two experts coded topics from the 15-, 20-, and 30-topic models. Interrater reliability, as measured by Krippendorff’s alpha, ranged from .81 to .89, depending on the individual topic model. The two coders first independently labeled each topic and rated its coherence by reading a sample of tasks with the highest loadings on a given topic. After going through this process for each model specification, the two coders compared and discussed their ratings. Most of the differences were only a 1-point difference (e.g., one coder rated a topic’s coherence as a 2, whereas the other rated it a 3). If the coders could reach agreement, they adjusted their individual scores to the score they agreed to. If they could not, they preserved their original ratings (for only one topic did the two coders preserve their own original ratings).

The results of the rating process indicated that the 20-topic model was optimal. Besides having a higher average coherence rating than the 15-topic model (μ_{Model_ 20} = 3.18, μ_{Model_15} = 2.9), 75% of the topics in the 20-topic model have a rating of 3 or higher, whereas 73% of ratings in the 15-topic model are rated 3 or higher. More significantly, 45% of topics in the 20-topic model have a rating of 4, compared with only 13% of topics in the 15-topic model. Because an average coherence rating of lower than 3 for a given topic casts doubt on its content validity per definitions of these categories in our rubric descriptions, only topics with an average rating of 3 or higher were used for subsequent analysis. The second column of Table 3 provides the average ratings from the two coders for each topic of the 20-topic model.

OPEN IN VIEWER

Table 3

Descriptives of Reform Topics

Reform topics	M coherence rating	M (SD)
Topic 11. Leadership teams setting goals and reviewing data for school improvement	4	0.095 (0.082)
Topic 3. Engaging parents about student academic and behavioral learning in schools	4	0.084 (0.078)
Topic 12. Teacher instructional improvement via walkthroughs, observations, and feedback	4	0.075 (0.081)
Topic 1. Interventions and supports for promoting positive student behaviors	4	0.070 (0.068)
Topic 4. Planning, providing, and evaluating professional development for instructional improvement	4	0.068 (0.059)
Topic 9. Teacher team (e.g., grade-level team, PLC) activities (e.g., reviewing data, planning, aligning standards, developing interventions)	4	0.061 (0.077)
Topic 5. Monitoring student progress and using data to develop interventions	4	0.057 (0.058)
Topic 18. Collecting, analyzing, and aligning student assessments	4	0.052 (0.051)
Topic 7. Using assessment data to identify students for targeted support	3	0.047 (0.06)
Topic 2. General parent and community outreach	4	0.045 (0.055)
Topic 17. Extending learning time (or opportunities) for students and staff	3	0.044 (0.06)
Topic 20. (Low coherence)	2	0.043 (0.063)
Topic 16. (Low coherence)	2	0.039 (0.048)
Topic 19. Improving special education	3.5	0.038 (0.049)
Topic 15. Setting goals for and recognizing teachers’ and students’ growth	3	0.038 (0.071)
Topic 8. Extending instructional time and aligning curriculum or assessments to standards	3	0.038 (0.074)
Topic 13. (Low coherence)	2	0.037 (0.046)
Topic 14. (Incoherent)	1	0.028 (0.042)
Topic 10. Administering common assessments and disaggregating data to differentiate interventions	3	0.021 (0.039)
Topic 6. (Low coherence)	2	0.021 (0.054)

Note. Topic proportions are at the school-year level; table is sorted by mean cumulative topic proportions. PLC = professional learning communities.

Reform Strategy Prevalence

The last column of Table 3 indicates that topics vary in prevalence, as indicated by the means of average topic proportions at the school-year level. ⁶ The topic with the highest mean proportion of 0.095 signifies that schools on average spent 9.5% of their reform efforts annually on Topic 11 (“building leadership teams to set goals and review data for school improvement”). Other topics with high mean proportions include Topic 1 (“interventions and supports for promoting positive student behaviors”; μ = .70, SD = 0.068), Topic 3 (“engaging parents in student academic and behavioral learning in school”; μ = .084, SD = 0.078), and Topic 12 (“teacher instructional improvement via walkthroughs, observations, and feedback”; μ = .075, SD = 0.081). Topics with low mean proportions include Topic 10 (“administering common assessments and disaggregating data to differentiate interventions”; μ = .021, SD = 0.039) and most of the topics with low coherence ratings. Moreover, we observe large variations for many topics, with standard deviations equal to 2 times as large as the mean. This shows that the topic prevalence or prioritized reform strategies varied across schools and over time. We further explore this variation in the next section.

Variations in Reform Strategies by SIG, Priority, and Focus Schools Over Time

As shown in Figure 3, SIG schools had more changes in the proportions of tasks implemented over time than priority and focus schools did. For example, from Year 1 to Year 3, SIG schools greatly increased the implementation of Topic 8 (“extending instructional time and aligning curriculum or assessments to standards”) and Topic 15 (“setting goals for and recognizing teachers and students’ growth”). In contrast, the topic proportions for priority and focus schools were relatively stable over time. The patterns for priority and focus schools are largely similar to one another. Compared with SIG schools, they seem to implement more tasks on Topic 1 (“interventions and supports to promote student behaviors”), Topic 11 (“leadership teams setting goals and reviewing data for school improvement”), and Topic 12 (“teacher instructional improvement via walkthroughs, observations, and feedback”).

OPEN IN VIEWER

Figure 3.

Topic proportions by reform types and over reform years.

These patterns are understandable, considering the multitiered accountability and support system Washington State implemented. SIG schools followed federal guidelines that particularly emphasized strategies of aligning curriculum with assessments and standards and extending instructional time, as well as strategies of promoting students’ growth and rewarding teacher performance based on student growth. Priority and focus schools were funded through state resources and were encouraged but not required to follow the SIG guidelines. These schools used the Indistar system to align their efforts with the seven state principles of school reform. Therefore, the following activities stand out at priority and focus schools when compared with SIG schools: building a strong leadership team, implementing new evaluation systems for teachers and principals, and developing positive student behaviors for social–emotional learning and a safe school climate. In addition, SIG schools received stronger treatment over time due to increased accountability pressure, whereas priority and focus schools did not experience the same kind of pressure. This consistency between reform strategies and reform type further sheds light on the promise of the text analysis results.

Internal Structure of Reform Strategy Measures

We next examined the internal structure of the topics—in this case, investigating how topics that theoretically should be related are in fact related, or how topics that theoretically should not be related are in fact unrelated. For example, as shown in Table 4, Topic 1 (“interventions and supports for promoting positive student behaviors”) and Topic 12 (“teacher instructional improvement via walkthroughs, observations, and feedback”) have a near-zero correlation (ρ = −.006) because these two reform strategies do not necessarily depend on one another, with one focusing on student behaviors and the other focusing on teacher evaluation. In contrast, Topic 12 and Topic 15 (“setting goals for and recognizing teachers’ and students’ growth”) are significantly and negatively correlated (ρ = −.246, p < .001). Although both Topic 12 and Topic 15 describe strategies targeting teachers—which explains why they are correlated—schools that spent more resources and time on Topic 12 often spent less on Topic 15. Our interviews with school leaders suggest they viewed Topic 12 as a strategy for supporting teachers’ professional growth and Topic 15 as an incentive-driven strategy. Moreover, Topic 12 summarizes a process-oriented reform strategy, whereas Topic 15 summarizes an outcome-oriented reform strategy. In contrast, Topic 15 is significantly and positively correlated with Topic 8 (“extending instructional time and aligning curriculum or assessments to standards”; ρ = .162, p < .001), which makes sense because both of these topics focus on supporting student and teacher learning and tie learning processes together with learning goals and standards.

OPEN IN VIEWER

Table 4

Pairwise Correlations Among Reform Strategy Topics

	Topic 1	Topic 2	Topic 3	Topic 4	Topic 5	Topic 7	Topic 8	Topic 9	Topic 10	Topic 11	Topic 12	Topic 15	Topic 17	Topic 18
Topic 2
Topic 3
Topic 4
Topic 5
Topic 7
Topic 8
Topic 9
Topic 10
Topic 11
Topic 12
Topic 15
Topic 17
Topic 18
Topic 19

Note. The white background indicates positive correlation coefficients whereas the light gray indicates negative correlation coefficients.

Reform topic labels:

Topic 1. Interventions and supports for promoting positive student behaviors

Topic 2. General parent and community outreach

Topic 3. Engaging parents about student academic and behavioral learning in schools

Topic 4. Planning, providing, and evaluating professional development for instructional improvement

Topic 5. Monitoring student progress and using data to develop interventions

Topic 7. Using assessment data to identify students for targeted support

Topic 8. Extending instructional time and aligning curriculum or assessments to standards

Topic 9. Teacher team (e.g., grade-level team, PLC) activities (e.g., reviewing data, planning, aligning standards, developing interventions)

Topic 10. Administering common assessments and disaggregating data to differentiate interventions

Topic 11. Leadership teams setting goals and reviewing data for school improvement

Topic 12. Teacher instructional improvement via walkthroughs, observations, and feedback

Topic 15. Setting goals for and recognizing teachers’ and students’ growth

Topic 17. Extending learning time (or opportunities) for students and staff

Topic 18. Collecting, analyzing, and aligning student assessments

Topic 19. Improving special education.

PLC = professional learning communities.

OPEN IN VIEWER

Table 5

The Associations Between Reform Topics and School Average Student Absences

Reform topics	Full-day absences	Part-day absences	% chronic full-day absences
Topic 1. Interventions and supports for promoting positive student behaviors	1.599	−3.147	0.055
	(3.177)	(3.560)	(0.082)
Topic 2. General parent and community outreach	−7.447	−2.375	−0.217
	(4.423)	(4.784)	(0.120)
Topic 3. Engaging parents about student academic and behavioral learning in schools	−3.820	−4.656	−0.090
	(2.408)	(3.233)	(0.067)
Topic 4. Planning, providing, and evaluating professional development for instructional improvement	−6.200	0.243	−0.153
	(3.361)	(5.518)	(0.096)
Topic 5. Monitoring student progress and using data to develop interventions	9.895	5.699	0.210
	(5.276)	(5.518)	(0.137)
Topic 7. Using assessment data to identify students for targeted support	3.652	1.387	0.168
	(3.971)	(4.683)	(0.120)
Topic 8. Extending instructional time and aligning curriculum/assessments to standards	7.178	0.229	0.164
	(4.585)	(4.402)	(0.114)
Topic 9. Teacher team (e.g., grade-level team, PLC) activities (e.g., reviewing data, planning, aligning standards, interventions)	−10.330***	−16.750***	−0.241***
	(2.737)	(4.364)	(0.068)
Topic 10. Administering common assessments and disaggregating data to differentiate interventions	−16.36**	−14.06*	−0.396**
	(5.867)	(6.961)	(0.144)
Topic 11. Leadership teams setting goals and reviewing data for school improvement	6.686	3.876	0.131
	(5.018)	(5.153)	(0.106)
Topic 12. Teacher instructional improvement via walkthroughs, observations, and feedback	1.701	6.552	0.0736
	(2.955)	(4.111)	(0.075)
Topic 15. Setting goals for and recognizing teachers’ and students’ growth	1.777	0.389	0.0472
	(3.524)	(4.316)	(0.085)
Topic 17. Extending learning time (or opportunities) for students and staff	1.651	10.32	0.0414
	(4.304)	(5.552)	(0.113)
Topic 18. Collecting, analyzing, and aligning student assessments	−2.080	7.939	0.038
	(6.625)	(8.064)	(0.177)
Topic 19. Improving special education	5.024	−1.479	0.137
	(4.834)	(6.434)	(0.123)
N	599	599	599

Note. Each reform topic was added to the model separately. The models control for schools’ prior achievement, the number of tasks schools were implementing in a given year, and school characteristics. The standard errors are clustered at school level. PLC = professional learning communities.

*

p ≤ .05. **p ≤ .01. ***p ≤ .001.

OPEN IN VIEWER

Table 6

The Associations Between Reform Topics and Student Achievement

Reform topics	Math	Reading
Topic 1. Interventions and supports for promoting positive student behaviors	−0.400*	−0.150
	(0.173)	(0.122)
Topic 2. General parent and community outreach	−0.520**	−0.169
	(0.187)	(0.164)
Topic 3. Engaging parents about student academic and behavioral learning in schools	−0.015	0.0008
	(0.137)	(0.116)
Topic 4. Planning, providing, and evaluating professional development for instructional improvement	−0.282	−0.055
	(0.199)	(0.155)
Topic 5. Monitoring student progress and using data to develop interventions	−0.168	0.372
	(0.176)	(0.195)
Topic 7. Using assessment data to identify students for targeted support	−0.358	−0.261
	(0.359)	(0.147)
Topic 8. Extending instructional time and aligning curriculum/assessments to standards	0.290	0.201
	(0.233)	(0.136)
Topic 9. Teacher team (e.g., grade-level team, PLC) activities (e.g., reviewing data, planning, aligning standards, developing interventions)	0.013	−0.357
	(0.121)	(0.232)
Topic 10. Administering common assessments and disaggregating data to differentiate interventions	0.513	0.297
	(0.269)	(0.265)
Topic 11. Leadership teams setting goals and reviewing data for school improvement	−0.096	−0.090
	(0.167)	(0.147)
Topic 12. Teacher instructional improvement via walkthroughs, observations, and feedback	−0.010	−0.218
	(0.158)	(0.135)
Topic 15. Setting goals for and recognizing teachers’ and students’ growth	0.439*	0.230*
	(0.201)	(0.107)
Topic 17. Extending learning time (or opportunities) for students and staff	−0.128	−0.025
	(0.307)	(0.174)
Topic 18. Collecting, analyzing, and analyzing student assessments	0.038	0.183
	(0.269)	(0.247)
Topic 19. Improving special education	0.023	−0.021
	(0.271)	(0.294)
N	596	580

Note. Each reform topic was added to the model separately. The models control for schools’ prior achievement in math and reading, the number of tasks schools were performing in a given year, and school characteristics. The standard errors are clustered at school level. PLC = professional learning communities.

*

p ≤ .05. **p ≤ .01. ***p ≤ .001.

Further Validation of the Reform Strategy Measures Using a Different Data Source

Another form of validity evidence is correlation with another measure of the same or a similar construct gathered from a different data source. To further validate our reform strategy measures, we interviewed a number of principals and other staff members from 10 schools about 3 to 6 months after the schools submitted their reports. If the schools had fabricated the reports, school staff would have had difficulty recalling their content months after submission. The 10 schools varied in student population, educational level, reform type (SIG, priority, and focus), and geographic location, as well as in their student achievement gains up to the year in which they were interviewed. We asked interviewees to freely describe the important initiatives they undertook in the last school year to transform their schools.

About 82% of the 10 most prevalent topics in the schools’ reports were mentioned as top initiatives by school administrators. In 4 of the 10 schools, the principals and staff referenced nine or 10 of the top 10 topics in the reports, and in the other 6 schools, staff mentioned seven or eight of the top 10 topics. In particular, among these 10 interviewed schools, three of them were SIG schools who submitted SIG annual reports 1 to 4 years previous to the interview. Twenty of 23 of the top areas identified from interviews were also found in SIG annual reports, an alignment of 87%. The high alignment between reports and interviews serve as a robustness check for the LDA priority measures, suggesting that the text analysis results are similar to those derived from interview data, with the additional advantage of being feasible to obtain on a much larger scale and with relatively low cost.

The Predictive Validity of Reform Strategy Measures

If the quantitative measures derived from the topic modeling represent meaningful distributions of schools’ reform strategies, these measures should have some power to predict changes in student absenteeism and achievement.