Designing School Systems to Encourage Data Use and Instructional Improvement

Federal and state accountability

Both federal and state accountability policies had strong influences on data use across all four cases, particularly in relation to the dominance of high-stakes, standardized state assessment results (e.g., Jennings, 2012). Described by one teacher as “the ghost that is chasing us,” numerous educators pointed to state assessment scores and other assessments designed to model these tests as key sources of data because of the link to program improvement status and the resulting penalties. In response, teachers and administrators in all four systems focused on the “bubble” or “cusp” students, those whose scores came within points of designated performance categories. These data were also used for student placement; for instance, administrators in District A used state assessment results for classroom assignment and student scheduling, placing students who scored far below basic or below basic in double periods of resource classes, whereas students who scored proficient or advanced were placed in Honors classes.

Internal accountability

Other patterns in data use relate back to internal literacy initiatives within a school system that were prioritized, supported, and monitored. For District A, school and system leaders focused on the common grade assessments that teachers co-developed in their professional learning communities, a district-wide policy that was supported and monitored at the district and school levels. Similarly, CMO D had performance management systems in place that incorporated internal benchmark assessment results with consequences in teachers’ evaluations and compensation systems. As such, one teacher felt that she prioritized these scores in her instruction because it was a measure of her effectiveness, with consequences attached: “No one here has tenure. So it is your data. You have got to show the data—are you being an effective educator? The data will prove it or not.” The performance systems in these schools meant that teachers focused on the benchmark data for their own job performance and security, which drove their focus on it.

Authorizer accountability

With respect to the charter systems, educators in both CMOs voiced the belief that their schools faced a legitimate threat of closure if state assessment scores were poor. One central office administrator at CMO C argued, “the [state assessment results] are our ‘bread and butter,’ the reason we can stay open.” During times of charter monitoring or reauthorization, charter educators felt a pressing need to prioritize state assessment data. One of CMO D’s principals explained:

[State assessment] scores, federal and state assessment rankings are all demands, especially when we are up for renewal. The way charters have been set up, we’ve exchanged our autonomy for increased accountability. So for AYP, especially when you talk about English language learners (ELs) on the [state assessment], it’s absolutely, absolutely a huge thing. We’ve barely made it with ELs in the last couple of years, barely making the threshold, and with it always going higher and higher, you absolutely have to focus with the staff what we are going to do with that group.

CMO leaders also noted that state assessment scores were critical to the network’s potential for expansion under the current authorizer. One leader felt the current charter landscape was “crowded,” and their authorizer, the local school district, may be less likely to grant charters to new schools in the future. High scores on the state assessment would provide evidence to “prove the success” of the model and support the case for future replication.

Market and community accountability

Market and community accountability pressures meant CMOs paid greater attention to college-ready indicators than their peers in the school districts. CMO educators internally used these data to evaluate if they were meeting their schools’ missions for college preparation but externally as well. As schools of choice, the CMOs publicized their college-ready metrics outwards into the educational market to attract parents and families to their organization. Data displays of college-ready metrics established the CMOs’ “reputations” and were included in recruitment materials, the organizational websites, and as part of the parent/student handbooks. As one of CMO D’s central office administrators explained, “to put kids in the seats, we use our graduation rates, SAT scores, college acceptance rates to attract families to our schools. We want to communicate that going to college is part of the ‘[CMO D] brand.’” The college focus helped position these schools in the educational marketplace, and the metrics signaled to parents and families the success of their model.

Finally, one CMO was engaged in a local political debate around the value of charter schools specifically and school choice more generally. The CEO felt that the dialogue around charter schools had recently turned negative, and “the attacks that we’re experiencing have made us even more vigilant about our [state assessment scores] being very, very strong. … They will be used as a weapon to speak badly about us, otherwise.” Here, high stakes assessment scores not only established legitimacy in regards to district schools, but they also distinguished the CMO-related schools as “high quality” when compared to the rest of the field of charter schools.

Structure and decision-making rights

The structure of the two types of organizations shaped the systems’ abilities to mobilize resources for data use. The two school districts were hierarchical, with compartmentalized departments. In contrast, CMOs were best described by respondents as a home office that provided back office support to their decentralized networks of schools. Although the CMOs did have some of the same departments as the districts (e.g., human resources), the CMOs were flatter, with fewer levels on their organizational charts. In all four school systems, the central offices supported schools and provided resources, whereas individual school sites were expected to be responsible for student achievement. However, by design, a greater level of autonomy and decision-making rights were distributed to the school sites in the CMOs, particularly in the areas of budgeting, staffing, and curriculum development. ⁶

Consequently, the governance structure largely shaped the flexibility with which system resources could be mobilized for data use. For the districts, these data-use initiatives were sometimes fractured along the departmental lines. For example, in District A, departments were distinct, overseen by separate supervisors with individual budgets. The assistant superintendent of secondary schools shared that the departmental silos got in the way of sharing resources for data-use initiatives: “I have very, very few resources to begin with. … But do you think that [the assistant superintendent of elementary schools] is willing to share her resources that are more abundant than mine with any of my schools?” When one such initiative was unveiled, the compartmentalized nature of the districts also shaped the success of its implementation. For instance, one department in District A purchased video cameras, so that teachers could record their instruction and share it as data of instructional practice with their fellow teachers. However, this initiative was seen as the “baby” of the one department, and as such, it did not receive widespread support within the district central office, nor was there uptake of the initiative from teachers or principals. In contrast, District A’s professional learning community initiative to support collaboration around common assessments results that was led by the superintendent and involved all departments in the district met with a greater level of commitment. In sum, the compartmentalization of the districts sometimes constrained their mobilization of resources when there was not a concerted effort from the top leadership to connect and unite departments.

Based on their organizational charts and self-reports, the two CMOs had thinner home office structure, with fewer levels of authority between school sites and the top management. On one hand, the network approach (along with decision-making rights distributed out to school leaders) offered the flexibility to adjust resources at the systems level when necessary and potential for adaptation to changing conditions. The two central offices of the CMOs saw their role as developing a “menu of supports” from which the school sites could choose, which were then constructed and revised to meet the needs of individual school communities. As explained the CMO administrator:

Bureaucracy gets in the way. We’ve noticed over and over that mandates don’t get you any farther. Even if you do mandate it, it’s not like it happens, so you might as well do it the hard way which is: “We think this might work. Here’s the best thinking we’ve got to date on this. Let me put this in front you, teacher, principal, coach. What do you think?” And those stakeholders, buying in to that and talking what they think works, and then using all those incredible feedback loops … so that our materials are constantly growing and changing.

Additionally, when it came to making classroom or school-level decisions based on data, the individual charter school sites had greater flexibility to make adjustments. One teacher noted that at the school sites, “our principal has the autonomy and responsibility for the budget, so we can use our resources however we need for our classrooms based on the data.”

On the other hand, the commitment to a thin back office also impacted the level of support and resources to schools. As a principal at one CMO C school reported, “It can be frustrating, because [home office] thinks it is providing all this support, but they’re really not. Some people are stretched so thin, they can’t do anything to really help you.” Additionally, several educators in both CMOs expressed concern that the push to centralize and establish “standards operating procedures” for analyzing and responding to data and “nonnegotiable” practices around instruction, curriculum, and assessments would move the systems toward a more top-down structure.

Size and growth trajectory

Closely related to system structure was the size of the school system and where it stood in its growth trajectory (i.e., age of school system and individual schools). The two school districts were larger in terms of the number of students served, although the overall number of schools in each system was comparable. Additionally, the districts with only slight variation had maintained their size over the past few years, whereas the two CMOs had seen remarkable growth within a short period of time. CMO C, for instance, had grown from one to over 30 schools over the course of 10 years. As such, the two CMOs were newer school systems compared to the two districts that had been established years before.

Size largely impacted how the school systems allocated their resources to support data use in schools. For the districts, their size and complexity compelled them to use a top-down strategy for data-use initiatives when they wanted to reach all schools. One central office administrator shared, “Unless you roll something out with a structure, you can’t replicate it across the system as big as ours.” In contrast, the CMOs believed that their smaller size enabled their flexibility and responsiveness when mobilizing or reallocating resources. Explained a vice principal in one of the CMOs, “We’re like a little tug boat, and we can kind of navigate through the waters. Whereas you have [a larger system], it’s this big hulking Titanic [that] can’t stop as easily.”

The second factor related to size was where the organizations stood in terms of their developmental trajectory. The two districts were at relatively fixed sizes, whereas the two CMOs had spent the past 10 years growing their networks of schools. This contextual factor supported enabled data-use efforts, as the CMOs designed their school systems and structures from the ground up. Reflecting on the initiatives in place to support data use, CMO C’s chief academic officer noted: “They’d be incredibly difficult to do in places that don’t do them now because we all know how difficult change is. But we weren’t overcoming that. We started from nothing, and if you get to build a system from the ground up, you can build it really well.” This “ground-up” design may also have contributed to the close alignment of data-use initiatives. For example, as CMO C rolled out its data management system, they also built in the complementary human capital resources–professional development and a dedicated “Data Leader” at each school site. As younger organizations, CMOs may have been better positioned to invest in technology than the established school districts, particularly when establishing the infrastructure to fit the changing data accounting.

The push for growth in the future, a feature of both CMOs, was also part of the push behind the investment in the sophisticated, automated data management systems and tools for school and system leaders. As the Director of Data and Assessment at CMO D articulated:

Because of the economic crisis, we are being hit hard just like everybody else, and we are also continuing to grow schools. So, it’s becoming more and more challenging to be able to do all the [data] analysis that we want with just two people, and we don’t have the budget to grow as our schools grow. So as a result, there are certain school-level analyses that we do that we’ve always done manually, and we’re going to have to figure out how to automate it.

However, the rapid scale-up in CMOs led to what one CMO D staff member called organizational “growing pains.” Despite efforts to plan, decisions to develop new data-use initiatives sometimes occurred retroactively—that is, in response to the multiple new stresses on the system that followed the expansion of the system. CMO D’s Director for Technology explained that the impetus for their integrated data management system was a recent investment. He explained:

When you have a small, 100 student school, you don’t need a comprehensive system. You can remember everybody’s name and scores longitudinally. But then the amounts of data, they compound quickly. As things get moving, it all becomes a second thought until one day, you hit a wall, and you’re forced to revisit.

Finally, bringing new schools online meant additional demands on the home office staff (e.g., identifying new streams of funding, finding a new facility, hiring new teachers), as well as the need to build capacity at that site around data use (e.g., creating norms of data use with the new school leader/staff, professional development and training). In the meantime, the home offices of both CMOs had not grown in proportion with the new schools, making data-use support to schools even more challenging. As one staff member responsible for supporting schools’ access to the data system at CMO C shared, “there’s a lot of anxiety around the growth plan. We just added four more schools, but I couldn’t add another headcount to my team. I’m not sure how this is going to work.” Although findings indicated that the two CMOs had made “significant” investments in resources to support data use, the continued pressure for growth, coupled with financial constraints, raised questions around the long-term sustainability of the data-use resources.

Financial resources

In all four systems, the statewide budget cuts were identified as a critical constraint for investing in data-use initiatives. This set of financial constraints caused educators to consider where they would find “the biggest bang for the buck” when it came to investing in organizational resources to support data use. For instance, in District A, the superintendent saw the investment in training for professional learning communities to be a front-end cost; once fully implemented, the PLC model would be self-sustaining, providing professional development from within the system at little cost.

For both CMOs, the limited financial resources were further complicated by the self-reported per-pupil funding differential between charters and district schools. Together, these circumstances led to heavy involvement by philanthropic partners to support data-use initiatives. Two large national foundations funded CMO C’s advanced data management system as well as the 10-person support staff. One CMO leader shared, “working with foundations has helped us launch several projects related to using data to inform organizational decisions—programs we definitely couldn’t do without their help.” However, working closely with the foundations had its own restrictions and limitations. For one, CMO leaders reported feeling that some foundations used funding as a “carrot” to encourage the CMOs to engage in a new project in one of the foundation’s areas of interest. Having projects driven by grants also created internal pressure based on the life of the grant. As one member of CMO C’s central office explained, “CMO C is very go, go, go, when the money is there. … The timeline is always tight, the funding is always tight, the resources for people’s time is always tight.” It also meant that programs or projects risked termination once a grant ends. At the time of the study, CMO C’s technology team was exploring how it could monetize data tools and technologies to other schools and districts to become self-funded once its grant ended.

New demands for data collection also occurred based on foundation interest and strings attached to funding. Fulfilling data requests from foundations was “time consuming because each funder wanted to see it in different ways,” reported the head of CMO C’s technology team. This demand for data as part of grant writing and reporting was another reason behind CMO C’s move to collect multiple forms of data in a central, unified data management system.

Degree of regulation and collective bargaining agreement (CBA)

Educators pointed to the presence or absence of a CBA as an important factor that shaped their ability to mobilize their human capital resources to support data use. Neither of the two CMOs had a CBA with a local teachers association, whereas the two districts did. For CMOs, this deregulation gave the charter schools more flexibility to allocate their human capital resources. “When you have the ability to hire and fire at will, as scary as it sounds,” one vice principal at CMO C shared, “it’s the starting point to be able to break the rules a bit, and be more flexible and agile with your staff.” It also gave them the ability to develop a range of positions (e.g., lead teacher, mentor teacher, and induction coach) where students’ performance was an important part of hiring and promotion.

The absence of a CBA also allowed the CMOs to quickly make changes to organizational routines in response to data. One notable example arose between District A and CMO D. In one CMO D school, the ELA department analyzed winter benchmark results and decided that, as their instructional response, teachers would change their prep periods so that they would have more available time to spend in each other’s classes, supporting students one on one. Between administering the exams and adjusting the schedule, the decision process took 2 weeks. In contrast, the president of the District A’s teachers association reported that the union and district had spent over 2 years negotiating the language in the contract around dedicated teacher meeting time to be used for collaboration around common assessment results.

Educators in the school districts stressed the importance of the CBA as a mechanism for protecting the rights of teachers. In both districts, the expectations around new data-use initiatives were high, and many teachers interviewed expressed feeling “overwhelmed.” One District A union representative argued that in some schools, completing data analysis protocols and other paperwork had put a strain on teachers’ time: “It’s not about the classroom and teaching. It’s all the rest to prove that we’re teaching. That’s where it’s gotten bad, all the documentation and the paperwork and the data tracking to prove we’re doing our job correctly.” District A’s union president also noted that although demands on teachers to use data to inform their instruction had increased, professional development and training from the district to effectively do this work had not kept pace. The contract then provided a starting point for the conversation between the teachers union, site leaders, and district office administrators about how to best manage and prioritize the demands on teachers’ time. CMO teachers expressed similar feelings of being overwhelmed, but without a CBA in place, it could be difficult to balance these pressures. Explained one teacher at CMO D, “Work-life balance without a union is a little bit tough, because really no one is telling you that you have the right to go home, even if your data analysis isn’t done.”