Abstract
This qualitative case study examines how board members make sense of federal accountability policies and how their sensemaking shapes their use of assessment data as a policy instrument. Deviating from previous work on practitioner sensemaking, the participants’ interpretations of assessments did not align with their ensuing use of the data. Furthermore, board members’ use of assessment data diverged from both federal and state messaging, illustrating board members’ synthesis and adaptation of external messaging into a locally driven narrative. As the nation has shifted to state accountability systems under Every Student Succeeds Act (ESSA), the findings provide insights to policymakers and practitioners to support local implementation.
Keywords
Introduction
In 2015, President Obama opened a new chapter in federal education policy by signing into law the Every Student Succeeds Act (ESSA), a reauthorization of the 1965 Elementary and Secondary Education Act (ESEA). Then Secretary of Education Arne Duncan explained, Whereas No Child Left Behind prescribed a top-down, one-size fits all approach to struggling schools, this law offers the flexibility to find the best local solutions—while also ensuring that students are making progress. (U.S. Department of Education [DOE], 2015)
ESSA marked a transition in intergovernmental politics by engendering greater control and flexibility for states and local districts. Yet, the law also maintained core elements of No Child Left Behind (NCLB), including the use of student performance data from standardized assessments as a substantive measure of school progress (U.S. DOE, 2016a). State accountability plans incorporate established assessment components with the option to pilot new, innovative elements such as performance tasks and competency-based tests (Evans, 2019). While the innovation in accountability measures is not as extensive as some had hoped (Klein, 2019), state plans encompass variability overall (Education Commission of the States [ECS], 2018). Since Secretary of Education Betsy DeVos approved all 50 state ESSA plans in 2018, State Education Agencies (SEAs) are now working with Local Education Agencies (LEAs) to ensure implementation fidelity of the new accountability measures.
For LEAs, school board understanding of state accountability plans is critical for ESSA implementation. Community-elected school board members do not have the same professional training, certification, or educational expertise as district leaders. Yet school boards oversee policy implementation in their districts (Kenney, 2019; Mountford, 2008) and facilitate local implementation of state accountability plans. Studies by Macartney and Singleton (2018), Trujillo (2013), and Diem et al. (2015) find that board members’ beliefs play significant roles in shaping policy implementation in districts. Few studies, however, examine how boards make sense of policies, particularly when federal and state messaging may not align.
This article addresses the critical need to understand how school board members make sense of and enact assessment policies. The qualitative case study analyzes school boards’ sensemaking of NCLB accountability policies in a region where state and federal messaging about student assessment data diverged. By investigating how boards interpret and implement assessment policies in this context, the study provides insight into how board members may respond to state-level plans and what supports they need to ensure fidelity and coherence of implementation under ESSA. Two research questions frame the study: first, how do school board members make sense of accountability policies that mandate student assessments? Second, given their sensemaking, how do boards subsequently frame the value of these policies for local and state accountability purposes?
Accountability and Assessment Under No Child Left Behind
NCLB, described as a “one-size-fits-all” educational reform, was rolled out with directives that the U.S. Department of Education (DOE) would not allow flexibility with state accountability plans (Shelly, 2008). However, NCLB was designed without consideration for the capacity or will of SEAs to implement these reforms (Sunderman & Kim, 2007; Sunderman & Orfield, 2006). Many states had small, under-resourced SEAs and struggled to implement NCLB with the oversight necessary to ensure local compliance (Sunderman & Kim, 2007). Within a year of implementation, the DOE began to offer flexibility for some provisions of the law (Shelly, 2012); by 2011, the Obama administration authorized state waivers for NCLB. States’ modified accountability plans encompassed variability in how they set standards, intervened for underperforming schools, and set accountability measures (Vergari, 2012).
NCLB’s accountability provisions, which were adapted by SEAs and LEAs, were arguably the most constrictive component (Henig, 2009). Under NCLB, schools were mandated to annually assess students in Math and ELA in grades 3–8 and once in high school (U.S. DOE, 2002). These scores, broken into subpopulations by gender, race, Free and Reduced Price Lunch, English Language Learners, and Special Education, were reported to SEAs. Schools that did not meet annual benchmarks for student proficiency were identified as “failing to meet Adequate Yearly Progress” (AYP), triggering escalating interventions for improvement (U.S. DOE, 2002). The final benchmark mandated 100% of students demonstrate proficiency by 2014, a standard widely perceived to be unrealistic (e.g., Darling-Hammond, 2007; Holcombe, 2014).
Both SEAs and LEAs were required to publicly report student assessment data; this component of NCLB was designed as a policy instrument to both measure educational outcomes and leverage change on the local level (Sunderman & Kim, 2007). By design under NCLB, student assessment data could be used to measure local educational outcomes including school performance, student performance, and educational equity, the latter as measured by differences in achievement by subpopulation. This policy was framed as a matter of educational equity to empower parents and communities, as President Bush (2002) attested: “No longer is it acceptable to hide poor performance. No longer is it acceptable to keep results away from parents.” Collecting annual student assessment data enabled leaders to employ the data as a policy instrument to leverage change for chronically underperforming schools or to maintain support for high-performing schools.
The use of student assessment data was expanded under the Obama Administration’s ESEA waivers, which were issued to allow greater flexibility from the more pernicious components of assessment reporting. In return, states were required to implement revised accountability plans that mandated the use of student assessment data in principal and teacher evaluations, in addition to other substantive changes (U.S. DOE, 2012). Although six states did not receive ESEA waivers (Center on Education Policy [CEP], 2015), the majority did, resulting in widespread use of assessment data to measure schools, principals, and teachers, and as a tool to improve educational equity.
Accountability and Assessment Proposed Under the ESSA
ESSA replaced NCLB in December 2015, when President Obama signed the reauthorization into law. Despite being heralded as a major change in U.S. education policy (U.S. DOE, 2015), ESSA retained central components of NCLB’s accountability provisions (Charnov, 2016). However, the design and implementation of these accountability plans shifted from the federal government to states and local districts (Saultz et al., 2016). The U.S. DOE (2016a) set the parameters for state accountability systems under ESSA, explaining that “The proposed regulations reinforce the statutory requirement that states have robust, multi-measure statewide accountability systems, while giving them the flexibility to choose new statewide indicators that create a more holistic view of student success” (p. 2). States could select from a range of performance indicators to annually assess school quality. Some elements of the accountability plans were mandated, like graduation rates and English Language proficiency, while states were able to choose from other indicators such as school climate, student and teacher engagement, and range of courses (Charnov, 2016). However, the U.S. DOE (2016a) proposed state accountability systems place “substantial weight” on academic indicators (p. 2), such as student assessment data from annual testing.
Although state ESSA plans encompass variability, the federal emphasis on student performance data ensured these measures remain central components of accountability plans. Thus, ESSA plans integrate accountability elements of NCLB with other measures (ECS, 2018). Given the dichotomy of increased state and local flexibility coupled with continued reliance on student assessment data, questions arise as to how district leaders, including school boards, are making sense of the policy changes and how those changes affect the implementation of state plans.
Conceptual Framework
District Leaders’ Policy Sensemaking
As a federal education policy, ESSA requires SEA and LEA coordination for implementation. However, local interpretation and implementation of federal and state educational reforms is not a zero-sum game wherein policy is a top-down input that produces a specific output in practice (Fuhrman & Elmore, 1990; McLaughlin, 1990). Instead, stakeholders integrate multiple dimensions of the policy with their own experiences and environments, which affect how they make sense of and ultimately enact implementation on the local level (Coburn, 2005; Spillane, 1999; Weick, 1995). Researchers studying implementation of standards-based reforms from the 1990s found local stakeholders’ sensemaking of policies generated variable implementation within and across educational systems (Fuhrman & Elmore, 1990; Spillane, 1996). Similar outcomes were observed during NCLB, as the DOE authorized waivers and increased flexibility for state-level accountability plans (Saultz et al., 2016; Shelly, 2008, 2012). Thus, one likely conjecture is that there will also be local variability of ESSA implementation.
In this study, I draw from the cognitive theory of sensemaking (Spillane et al., 2002; Weick, 1995; Weick et al., 2005) as a means to analyze the relationship between school board members’ sensemaking of accountability policies and how they subsequently enact those policies on the local level. Sensemaking is a fluid, recursive process shaped by social dimensions including beliefs, experiences, and the sensible environment (Weick, 1995, p. 17). Although sensemaking occurs through the individual, understandings are collectively constructed through organizations (Weick, 1995). From this theoretical perspective, local policy actors’ individual and collective experiences shape how they understand—or ignore—various policy instrument levers designed to shape implementation (Weick, 1995). As Spillane (2000) explains, policy stimulus is not all that matters . . . From a cognitive perspective, local beliefs, agendas, and situations are important influences on the ideas about reforming practice that implementers construct from policy, not just whether they decide to implement these ideas (p. 146).
In other words, local actors filter policies through their experiences, knowledge, beliefs, environmental context, and so forth as a means to develop cognitive understanding, which in turn shapes how they enact the policies in practice (Coburn, 2005; Weick, 1995). Critically, sensemaking scholarship articulates a tight connection between organizational sensemaking of policies, regardless of whether the sensemaking is accurate with the policy design, and subsequent policy implementation (Gawlik, 2015; Hemmer et al., 2018; Spillane & Anderson, 2014).
For this study, which is part of a larger analysis of locally controlled district governance, I apply the theory of sensemaking as an analytic frame (Ragin & Amoroso, 2011) to investigate the relationship between school board sensemaking of accountability policies and how they enact and leverage the policies on the local level. Research on sensemaking in education largely examines professionals within the system, such as principals, teachers, and central office staff (e.g., Coburn, 2005; Hemmer et al., 2018; Spillane & Anderson, 2014). School board members, as non-professional educational leaders, are largely missing from sensemaking analysis, yet they play a critical role in accountability policy implementation (Diem et al., 2015; Trujillo, 2013). Crucially, school boards do not have the same organizational routines and structures as school districts. Board members are elected from the general public, thereby encompassing a range of professional expertise, networks, and knowledge. The position does not require professional certification or specialized training, and research finds board members may not attend professional development designed to develop collective understanding of legislative and ethical dimensions of their work (Gawlik & Allen, 2019; Hall, 2016; Roberts & Sampson, 2011). Furthermore, board member interactions are limited to official meetings, as board members can violate public meeting laws by creating accidental quorums when gathering outside of formal events. School board members therefore have limited collective sensemaking in comparison to educational professionals because boards do not operate like traditional organizations. The study therefore serves to expand theoretical understanding of the relationship between sensemaking and policy enactment by examining a previous excluded group: school board members.
Hortatory Policy Instruments and Stakeholder Sensemaking
Federal education policies such as ESSA and NCLB require state and local cooperation to ensure fidelity in implementation. Upper levels of government do not have the resources to ensure SEA and LEA compliance with policy implementation (Cohen & Spillane, 1992; Marsh & Wohlstetter, 2013). Policymakers therefore use policy instruments to facilitate efficacious policy implementation on the state and local levels, such as mandates, inducements, building capacity, changing systems, and hortatory messaging (McDonnell, 1994a; McDonnell & Elmore, 1987; Stone, 1997). Hortatory messaging is an instrument that develops stakeholder support for policy ideas through communications such as speeches, memos, and editorials (McDonnell, 1994a; Schneider & Ingram, 1990), akin to public relations for policies. Unlike other policy instruments that use incentives or consequences for compliance, hortatory tools rely on strategic persuasion. Schneider and Ingram (1997) explain, “Symbolic and hortatory tools assume that people are motivated from within and decide whether or not to take policy-related actions on the basis of their beliefs and values” (p. 519). In sensemaking theory, extracted cues from the contextual environments shape perceptions: “Control over which cues will serve as a point of reference are an important source of power” (Weick, 1995, p. 50). Furthermore, the plausibility of such cues is more important than accuracy in organizational sensemaking (Weick, 1995). Hortatory policy instruments, which are intended to frame and persuade stakeholders, are therefore crucial components of sensemaking.
Accountability Measures as Hortatory Policy Instruments
In educational policy enactment, accountability measures have been a major focus for hortatory policy instruments, including how the purpose and value of accountability measures are communicated to SEAs and LEAs, as well as how the outcomes of these measures are messaged to broader communities (McDonnell, 1994a, 2013; Mehta, 2013). McDonnell (1994a) writes, “Assessment is often based on a strong appeal to beliefs and values . . . Assessment has a strong informational component, and, in fact, the basic operating assumption underlying hortatory policy is that people will act on information” (p. 398).
The purpose and use assessment data as hortatory instruments have been communicated by federal, state, and local leaders. The DOE framed assessments as promoting educational equity and accountability using phrasing like “the soft bigotry of low expectations” to signal high-stakes testing could and should change school performance (Darling-Hammond, 2007; McDonnell, 2013). Many states, but not all, embraced similar messaging, framing assessments as a tool to leverage equitable opportunities for students and to improve low-performing schools (Henig, 2009; Manna & Ryan, 2011). In fact, individual states’ interpretation and subsequent messaging of accountability measures varied from that of the federal government (McDermott, 2007; Sunderman & Orfield, 2006; Vergari, 2012). It is therefore likely that local educational leaders received multiple, divergent messages about the purpose of accountability measures.
One major question is how school board members made sense of multiple layers of messaging about accountability measures. Given the difference in professional experiences and training, it is likely that board members’ sensemaking of high-stakes assessments varies from district leaders (McDonnell, 1994a, 1994b). Studies by Trujillo (2013) and Feuerstein and Dietrich (2003) found school board members have greater reliance on student performance data than superintendents and other district leaders. However, it is not clear how board members’ sensemaking is influenced by federal, state, and local messaging. Understanding how boards make sense of hortatory policy instruments is therefore critical to support their implementation of non-local policies.
Study Design and Method
The study is a qualitative, multiple case study (Creswell, 2007) to explore district leaders’ sensemaking and policy adaptation of NCLB accountability provisions. The case study design enables comparison across districts, as well as the integration of multiple data sources (observations, interviews, documents, focus groups) that engender a richer analysis of school board sensemaking. This study is part of a larger project examining how educational leaders interpret and enact district governance in a locally controlled region. I collected data between 2012 and 2015, a time period that captured Vermont’s shift from regional assessments—New England Common Educational Assessment Program (NECAPS)—to national assessments—Smarter Balanced (S-Bac). The case study sites are part of a regional supervisory union (SU) governed by a central school board and superintendent. The SU is comprised of more than 10 school districts, each of which are governed by community school boards (for more, see Hall & McHenry-Sorber, 2017; McHenry-Sorber & Sutherland, 2019). Within the SU, I selected three town-based school districts as case study sites: Ashfield, Conway, and Jackson (see Table 1). The districts were purposefully selected: they share a superintendent and central office, yet each board retains control of local policies and programming, and they each oversee a single school (PK/K—6/8). To maintain confidentiality, towns are identified with pseudonyms, and some identifying details have been obscured.
Characteristics of Case Study School Districts.
Data Collection and Analysis
The primary method of data collection for this case study was participant interviews with local educational leaders. Over the course of the study, I interviewed 17 district leaders, including the superintendent, SU board chair, district principals, and 13 community board members. Although school boards are the primary focus of the study, interviews with other district leaders enabled alternative perspectives of board decisions and dynamics, as well as district context. All district-level interviews used semi-structured protocols (see Appendix), lasted between 30–90 min, and were recorded and transcribed. On the SU level, I conducted three semi-structured interviews with the superintendent and two semi-structured interviews with the SU board chair. All SU interviews lasted between 1 and 2 hr and were recorded and transcribed. On the district level, I conducted two interviews with each board chair and interviewed district board members 1 or 2 times, depending on their availability. As one school board underwent significant turnover over the course of the study, I interviewed three former board members to provide additional perspectives. Two board members were unable to participate in interviews due to scheduling conflicts.
I used ethnographic observations to supplement and inform interview data (Emerson et al., 2011), documenting 30 hr of school board meetings. I recorded observations using ethnographic field notes, which I expanded after I left the field (Emerson et al., 2011). When possible, I cross-referenced meeting field notes with the documented public minutes. To substantiate observation and interview data, I also collected documents including school handbooks, town reports, school board and town meeting minutes, and other official materials. Other data sources, such as newspaper articles, blog posts, and press releases enabled additional analysis of the hortatory context.
To understand how Vermont framed and messaged assessment policies to local districts, I collected publicly available hortatory policy instruments including memos, op-eds, resolutions, letters, and newspaper articles from Vermont’s AOE, Secretary of Education, and State Board of Education (SBOE) that addressed NCLB, assessment, or accountability policies. These documents were gathered online, primarily on official state websites, and include materials from 2001 to 2015. I used documents to validate findings by data triangulation, comparing them with field notes and interviews to confirm evidence of themes across data sources, and to identify disconfirming evidence (Ragin & Amoroso, 2011).
Data analysis in the study was split into two separate processes: district-level data analysis, which included interviews, observations, and documents related to sensemaking of assessment policies, and state-level data analysis, which included documents related to Vermont’s messaging of assessment policies. I integrated district-level data analysis with data collection, which provided opportunities to refine data collection and test emerging hypotheses during fieldwork (Spillane, 1998). I inductively coded initial data using qualitative coding software to identify emergent themes related to assessment (Miles at al., 2014). Once data collection was complete, I used the initial themes to iteratively refine the coding scheme, incorporating both a priori and emergent codes (Creswell, 2007). I then recoded data with a thematic coding scheme, using major categories of educational beliefs by participants and educational practices in schools. These categories were further broken into thematic categories including student outcomes, external assessment reporting, and internal assessment reporting.
The coded data were sorted by district into thematic matrices (Miles et al., 2014), which I arranged by assessment practices, outcomes, and beliefs; interactions and perceptions of the Vermont AOE; and perceptions of local control. I used interview data from principals and the superintendent to confirm emerging themes and to identify alternative patterns (Ragin & Amoroso, 2011). I analyzed the matrices first within districts, then across the collective cases. The matrices illustrated patterns of practices that persisted across the districts. The matrices also enabled me to rule out initial hypotheses, such as school board resistance to mandated assessments, as it became clear most participants supported accountability policies. To ensure trustworthiness, I used member checks to clarify and confirm findings.
State-level data were initially sorted into a conceptually ordered data display by their intended audience—general public or district leaders—to assess if policy messages varied by target audience (Miles et al., 2014). I then drew from prior work by McDermott (2007), McDermott and Jensen (2005), and McDonnell (1994a, 2013) to construct a priori thematic coding. I coded evidence of equity and accountability discourses, as well as state descriptions of federalism. I analyzed data within and across target audiences. Evidence of variable messaging by audience was minimal, but I found evidence of state resistance to federal accountability measures. To verify my policy analysis, I reviewed findings with two participants who had experience with state-level education policy.
State Context: Vermont NCLB Implementation
The research study is situated in Vermont, one of the few states that declined an ESEA waiver. As LEAs were never granted flexibility from NCLB’s requirements, school boards had to consider the original intent of the mandate throughout the study. Data collection encompassed a 3-year period when the state transitioned from NECAP to the Common Core-aligned S-Bac. During this time, school board members routinely discussed accountability policies and tools, providing critical opportunities to observe discourse about the value and purpose of mandated assessment data.
Vermont has been recognized for setting quality standards, prioritizing educational equity, and demonstrating consistently high student outcomes on national and international assessments (Holcombe, 2014; McDermott & Jensen, 2005). In 1997, the state implemented a comprehensive equity-based school funding reform (Vermont State Board of Education, 2001). The law, first known as Act 60, and revised in 2001 to Act 68, introduced state learning standards; a hybrid accountability system integrating local assessments, student portfolios, and national tests; and a progressive school funding system (Vermont State Board of Education, 2001). However, Act 60’s accountability measures were replaced by NCLB.
Since the passage of NCLB, multiple Vermont school districts opposed NCLB, and several joined a lawsuit in opposition that was led by the National Education Association (Vergari, 2012). Vermont’s leaders also opposed many of the requirements, in some cases circumventing the design of the law through their accountability plans (Shelly, 2008, 2012). Of note, Vermont did not implement high-stakes interventions for schools (McDermott, 2003). Instead, the state primarily employed technical supports to intervene for chronically underperforming schools in lieu of school reorganization or closure. Vermont also publicly opposed the punitive classification of schools as failing to meet AYP under NCLB regulations. One notable example occurred in 2014, after all of Vermont’s schools were identified as failing to meet AYP:
Educational leaders in Vermont did not support federal policies to expand the use of accountability data as evaluative measures for educational outcomes (Holcombe, 2014; Vermont State Board of Education, 2016). The state also withdrew its application for an ESEA waiver, which required states to use assessment data as a component of principal and teacher evaluations (Vergari, 2012): We chose not to agree to a waiver for a lot of reasons, including that the research we have read on evaluating teachers based on test scores suggests these methods are unreliable in classes with 15 or fewer students, and this represents about 40-50% of our classes (Secretary Holcombe, 2014, p. 2).
Instead, Vermont’s AOE messaged that assessment data should be used holistically in tandem with local assessments and other measures. Over the course of the study, Vermont’s AOE and SBOE consistently employed this modified rationale for the purpose of mandated assessments in their communications to educational leaders and the general public.
Although Vermont leaders opposed some elements of NCLB, the AOE did agree with NCLB’s stated goals to use standardized assessment data as a policy instrument to ensure educational equity. Vermont repeatedly employed an equity frame when conveying the value of testing to stakeholders. For example, in a 2015 memo, Secretary Holcombe (2015a) wrote, To us, the primary benefit of the scores is that they provide another data point for our conversation about the statewide gaps in achievement that we see for children who live in poverty, children with disabilities, children who are learning English, and children who are affected by historic or structural discrimination or inequities (p. 2).
The AOE consistently communicated that mandated assessment data should be used to measure school outcomes and educational equity and in tandem with other measures.
Findings
In the case study, school board members’ sensemaking of the value of NCLB assessment policies reflects a lack of collective processing typical of educational leaders and therefore encompass significant variation. Yet board members across all three districts had near consensus on the purpose and use of assessment data, which neither aligned with federal or state messaging. Boards’ sensemaking of assessment data as a policy instrument reflected the integration of multiple external messages within their local environment.
In the following section, I first describe the different ways board members make sense of assessment policies, synthesizing their perspectives into three conceptual categories: oppositional progressives, ambivalent rationalists, and neoliberal proponents. I then analyze board members’ sensemaking of the value of assessment data, with a specific focus on hortatory instruments, and how boards subsequently rationalize and enact accountability policies
Board Members’ Individual Sensemaking of Mandated Assessments
Sensemaking is a social, collective process that incorporates individual beliefs, experiences, and values. It is therefore important to begin analysis by examining board members’ individual beliefs about mandated assessments to understand their subsequent collective sensemaking. In this study, board members’ individual sensemaking of assessments incorporates significant variations, which are grouped into three conceptual categories: oppositional progressives, ambivalent rationalists, and neoliberal proponents. The conceptual categories align with broader national perspectives on mandated assessments and the goals of public education.
Oppositional progressives
Four of the school board members believe mandated, standardized assessments undermine child-centered learning, disempower teachers, and erode local control of education. Board members employing this perspective believe in a traditionally progressive approach to education where schools are sites of democratic community participation, and learning is authentic and experiential (for more, see Dewey, 1997); this view also aligns with progressive, social justice scholarship on the implications of NCLB (Apple, 2007; Darling-Hammond, 2007). To oppositional progressive board members, standardized testing is incompatible with an authentic, democratic approach to education. “We do all these things, and they are so rich and very substantive to me compared to bubble tests,” said a school board chair. They express concerns that implementing standardized testing in a school negatively effects curriculum and instruction, and they believe instruction should be measured by meaningful locally designed assessments, not mandated tests. Reflecting Ashfield’s approach to assessment, the board chair said, People love the tests because they just provide this data that is so easy, and if you look at us, well, our points are: our children are doing these comprehensive, parent-teacher evaluations where they talk about the work they’re doing. . . they’re keeping a portfolio of their work.
To oppositional progressives, holistic and substantive evaluations such as portfolios provide the most accurate assessments of student learning over time.
In this study, all oppositional progressive board members lived in Ashfield, a small college town. Ashfield’s school district strongly embraces progressive education: the staff prioritizes student-centered, project-based learning and measuring student growth through portfolios while eschewing standardized assessments. Notably, Ashfield was one of the several Vermont districts that initially refused to participate in NCLB. In an excerpt from a 2001 memo to the state, the board asserted their opposition: We believe that the No Child Left Behind Act is an inherently flawed piece of legislation that fails students and fails schools . . . NCLB is a vehicle to remove control of our children’s education from local communities and school districts and place that control with the Federal Government.
Nonetheless, Ashfield was forced to implement NCLB mandates after Vermont’s Education Commissioner (later renamed Secretary of Education) threatened to revoke the licenses of the superintendent and principal. The LEA then developed a plan for implementation that met the letter of the law while undermining its efficacy. “We said we will offer the tests, but we’re not going to insist on anybody taking the test. And we sent a letter to the parents explaining why we felt that [way],” said one former board member. Over the course of the study, Ashfield’s district leaders continued to enable parents to opt-out of NECAPS and S-Bac assessments, and oppositional progressive board members continued to express resistance to the mandated tests.
Ambivalent rationalists
A second group of board members embody an ambivalent rationalist perspective toward mandated assessments. Board members in this group believe testing is neither inherently good nor bad, but assessments can be used to measure school-based outcomes. This perspective closely aligns with educational rationalism. First emerging during the Industrial Revolution, educational rationalists employed business and scientific methods to increase academic and economic efficiencies in school systems (Tyack, 1974). Reflecting similar perspectives as their predecessors, ambivalent rationalists in the study used terms like “tools” or “instruments” to describe standardized tests. Assessments provide meaningful data for school stakeholders about the overall success of the school. Unlike oppositional progressives, these board members do not believe mandated assessments undermine curriculum and instruction. A board member from Jackson shared, “we take great pride in being an excellent school, [but it’s] not test scores. Test scores are a reflection of the instruction.” These participants are pragmatic about testing, asserting measurable data is important, but not the sole measure for their schools.
Six out of 13 board members in the study maintain an ambivalent rational perspective. All three districts had at least one ambivalent rationalist board member. Unlike oppositional progressives, ambivalent rationalist board members did not express strong opinions about curricular and instructional practices. “I don’t pretend to know how to educate a kid,” one said bluntly. These board members are more likely to connect assessments to their own work, suggesting their professional experiences inform their sensemaking processes. A board member who worked as a nurse explained, I don’t know everything that makes a good education, but . . . you have to have a form of measurement. I know as a nurse, you have to document in certain ways, and it has to be a little bit scientific to show that a patient is getting better.
Board members with business and legal background shared similar sentiments, noting that while tests should not be the only tool, there should be some form of measurable evaluations.
Neoliberal proponents
A third group of school board members embodied a neoliberal proponent perspective toward standardized assessments. These board members were highly supportive of mandated assessments, which they considered to be crucial for measuring the success of schools, principals, and teachers. Initially emerging as an educational perspective with the 1983 national education report A Nation at Risk (Mehta, 2013), neoliberalism prioritizes school privatization, standards-based testing and accountability, and global education to prepare students for a rapidly changing technological world. Many argue the accountability design of NCLB is inherently neoliberal (e.g., Apple, 2007; Hursh, 2007; Mehta, 2013). In this study, the neoliberal proponents’ beliefs align with an economically driven model of education. These board members link mandated accountability policies to the future economic viability of students, using phrases like “global citizens” and “21st century learning” in their discussion of the need for such assessments. Neoliberal proponents also believe that student performance data objectively measure school success or lack thereof. After the SU superintendent encouraged Conway’s school board to “move away from NECAPS tests and look at other gauges” of school progress, one board member responded, “I get frustrated by trying to measure additional factors about students because you start getting so subjective.” This emphasis on test scores as a summative measure of schools differs from oppositional progressives, who did not believe test scores accurately measured their schools, and ambivalent rationalists, who described multiple measures when defining school success.
Neoliberal proponents comprise the smallest number of participants in the study, with only three board members out of 13 fitting within the category. All three are board members in Conway, a tourist community economically driven by locally owned small businesses. These board members did not share professional or political backgrounds, so it is unclear what factors shape their sensemaking. Unlike oppositional progressives, they did not express opinions about educational practices, instead entrusting their administrators to make programmatic and pedagogical decisions.
Collective School Board Sensemaking of the Purpose and Value of Assessment Data
School board members in the study adopted three different perspectives on assessments: oppositional progressives who are critical of standardized assessments, ambivalent rationalists who value assessments as one tool to measure schools, and neoliberal proponents who support assessments as the primary means of evaluating educational outcomes. Previous educational research establishes a relationship between how an individual makes sense of policies with how they enact those policies in practice (e.g., Gawlik, 2015; Spillane et al., 2002; Spillane & Thompson, 1997). Yet in this study, board members’ individual sensemaking of mandated assessments diverged from their collective sensemaking of assessment data as a policy instrument. This finding reveals board members’ sensemaking differs from that of educational professionals, as evidenced by the inconsistencies between sensemaking and implementation.
In the following section, I identify four uses of student assessment data that the US DOE and/or the VT AOE communicated via hortatory policy instruments. These include messaging the use of assessment data to measure school performance, principal performance, teacher performance, and educational equity. I then analyze the collective sensemaking of boards in deciding how to use these assessments. Despite variable individual sensemaking of the value of assessments, school board members collectively decided to use assessment results for similar purposes: measuring educational outcomes and preserving local control. Their use of testing data as a hortatory tool in their communities did not concisely align with either federal or state messaging.
Measuring school performance
The use of student assessment data to measure school performance is a foundational component of NCLB and was explicitly communicated by the DOE. All three school boards asserted that the purpose of standardized test data is to assess their schools. Students should perform well on standardized assessments if the quality of education provided by a school is effective. An ambivalent rationalist board member explained, “if it’s a good school, the bottom line is if you do a good job, then your scores will be reflective of that.”
Oppositional progressives, as the most fundamentally opposed to mandated assessments, discussed the conceptual quandary about using assessment data as a measure of school performance. One shared, If you don’t maintain [test scores], I don’t know. I don’t mean to express lack of faith in the current school, but one of the things I do is look at those standardized test scores and say, “wait a second, we used to nail these things.”
To the participant, declining test scores raised concerns about the performance of the school, not the quality or accuracy of the assessment data, despite earlier resistance to the policy. Other oppositional progressives voiced similar fears. “I’m really disappointed in our test scores this year because I’ve always maintained that if we’re doing a good job teaching, even without teaching to the test, the scores should reflect that,” one leader shared. Collectively, board members shared the perception that assessment data could be used to measure school performance. Even board members who opposed standardized testing still expressed the fundamental belief that when a school provides a “good education,” students will perform well on standardized tests.
Measuring principal and teacher performance
As previously discussed, federal and state messaging about the use of assessment data to measure principal and teacher performance did not align. Under the Obama administration, the DOE expanded the use of student assessment data to measure principals and teachers, both of which Vermont’s AOE publicly opposed. All three school boards identified the principalship as the most important role in ensuring academic performance in their schools. Two of the three school boards, Jackson and Conway, asserted assessment data could be used to measure principal effectiveness. One board member explained, “If we’re not getting the student outcomes and we’ve invested in professional development and we’ve invested in small class sizes, then there would need to be some interventions with [school administrators].” Ambivalent rationalists were more likely to describe multiple measures besides assessment data that could be used to assess principals, including teacher turnover and long-term student performance. Neoliberal proponents went further, calling for greater transparency with assessment data reporting as a means to hold school leaders accountable for student outcomes.
Although two boards decided that assessment data should be used to evaluate school principals, no board members discussed the value of using the data to evaluate teachers. Yet, the superintendent and two of the case study principals described using data as at least one measure to evaluate teacher performance and communicated this purpose to the boards. It is unclear why board members’ collective sensemaking did not integrate this local policy dimension.
Measuring educational equity for students
In the construction of NCLB, equity encapsulates the achievement gap between socioeconomic status, race, special education, English Language Learners, and other special populations of students. Vermont, a state comprised of a predominantly rural, White population (U.S. Census Bureau, 2012), emphasized equity by gender, socioeconomic status, and special learner populations. Both federal and state messaging of NCLB centered on the use of assessment data and subpopulation reporting as a means to assess and ensure educational equity (Holcombe, 2015a, 2015b; McDonnell, 2013). Despite explicit messaging from Vermont’s AOE about equity and assessment data, none of the school boards conceptualized testing as a tool to ensure educational equity.
The messaging by the AOE was reinforced by communication from the SU superintendent, who described his role to communicate and reinforce state policies to the boards. In an interview, the superintendent discussed the significance of an income-based achievement gap in Ashfield. “The demographics of the town have changed. . . You can see from the free and reduced lunch numbers, more kids that are hurting. And I observed that the school really was not serving those kids as well as they could.” At multiple board meetings, he explicitly discussed the importance of using assessment data to monitor and improve income-based inequities. However, there was no evidence that school boards members, collectively or individually, perceived assessment data could be used to evaluate equity of educational outcomes in any form. The only local reference in data collection to assessments and educational equity came from Conway’s principal, who wrote in an annual report that the school “is proud to have completely eliminated any family income achievement gap, meaning students performed equally on the NECAP regardless of family income.” Despite the significance of Conway’s achievement, the success was not referenced by Conway’s board members. Likewise, the boards in Jackson and Ashfield did not discuss the use of assessment scores to evaluate or ensure educational equity.
It is unclear why board members did not discuss equity as a component of accountability measures. However, one hypothesis is Vermont’s educational landscape does not align with the national narrative about educational inequity. Some scholars describe a focus in NCLB equity-based reforms as an attempt to equalize the difference between urban schools with predominantly low-income, minoritized student populations with wealthier, less diverse suburban schools (Darling-Hammond, 2007; Vergari, 2012). Given that national messaging on educational inequity focuses on minoritized and urban students, it is possible the board members did not connect the use of assessment data as a tool for educational equity in their predominantly White, rural districts.
Assessment Data as a Hortatory Policy Instrument
All three case study boards’ sensemaking of assessments included leveraging student data as a hortatory policy instrument to sway local and state opinions about their respective schools. Boards implemented this in two ways: first, they used messaging about their student assessment data as a means to sustain community support for school funding. Second, they communicated to their constituents that the assessment data could be used as a policy instrument to preserve local control. The boards’ sensemaking of the use of assessment data as a hortatory policy instrument aligns with the original intent of the NCLB policy (McDonnell, 2013), yet using the data to engender greater local control of education directly contradicted AOE messaging.
Assessment data to leverage community support
As all three boards asserted that assessment data accurately measured school performance, they also found value in reporting these data to leverage support for annual school budgets. Per Vermont Statute, school boards are required to publicly report student test scores and proposed budgets in advance of annual town meetings. At the town meetings, the school board and select board publicly review programming and budgets for the coming year, followed by community discussion and a vote to approve the budget (Bryan, 2010). The meetings often include active debate and an approved school budget is not guaranteed; over the course of the study, multiple districts rejected their school budgets (Galloway, 2014). Annual town meetings therefore are an important event to develop community support for school district funding.
In each of the case study districts, boards used multiple hortatory tools, including presentations, meetings, and reports, to communicate student assessment scores as a means to generate support for the annual school district budget. During the course of the study, Jackson and Conway schools had very strong student outcomes on NECAPS and/or S-Bac assessments. Conway’s board framed the student assessment data as evidence that long-term school board goals paired with a fiscally conservative budget created successful outcomes for students. In the annual district report, the board noted, We have worked hard to protect the students, taxpayers, and voters of [Conway] by offering a “Educationally Sound, Taxpayer Friendly” program. The results of these efforts have kept [Conway] in the top five elementary schools in the state in standardized test scores, [and] have helped us keep special education costs under control.
Jackson’s board made similar connections between high student outcomes and educational funding. The principal, speaking on behalf of the school board, told the audience at Jackson’s town meeting: “Certainly [these scores] are something in which the community can take pride in, because this shows how your tax dollars are spent.” Both boards explicitly connected taxes and student test scores as a means to build community support of the school district budget.
Ashfield’s school assessment data were significantly lower than the neighboring towns: during 1 year of the study, less than one-third of students demonstrated proficiency on the math and reading assessments. Nonetheless, Ashfield’s board, like Conway and Jackson, also leveraged mandated reporting of student data as a hortatory tool to ensure community support for the school district budget. Their board framed the student performance on mandated assessments as evidence of a progressive educational approach, which closely aligned with the community’s historical resistance to NCLB mandates. At one annual school budget meeting, district leaders incorporated local resistance to mandated assessments as they discussed school funding priorities. This was, in turn, supported by community members. One audience member said, [Ashfield] has consistently funded the elementary school because we are so in favor of what it is has done and where the kids have gone in higher education. It is proof of what an incredible school we’ve had for decades, and those numbers are there that demonstrate [Ashfield’s] students are successful.
In short, board members had community support for programming that countered NCLB mandates; therefore, they were able to use assessment data to demonstrate commitment to progressive, community-centered education.
Assessment data to preserve local control of education
Although scholars assert NCLB reduced local control of education (e.g., Apple, 2007; Schafft, 2015), all three case study boards came to understand that assessment data could be used as a hortatory policy instrument to retain local control. District leaders and their constituents prized self-governance and actively resisted perceived attempts to disempower local educational control (Hall, 2016). During the time of the study, all three boards were concerned about state-level policies, including Act 46, legislation that required reorganization or consolidation of districts. At one meeting, Conway’s board chair emphatically called on his audience to contact their legislators to protect local control of the district. His passionate speech typified board members’ persistent concerns that the state would limit their autonomy over school district governance.
As the boards believed mandated assessment data could measure school performance, they also perceived that student outcomes could be used as a tool to preserve local control. Reflecting collective sensemaking, all boards asserted that test scores not only reflected school success but also the skill of the local board to oversee education in their districts. This sentiment was first expressed by a principal describing the school board: My school board’s extremely activist. So the Department of Education in Montpellier gets annoyed by my school board a lot, because they are very: “Don’t mess with [us]! Don’t try to consolidate us. . . We have high student outcomes. We are good! So don’t mess with us!”
Like this principal, board members explicitly connected mandated accountability measures with preserving self-governance and resisting consolidation. They believed that as long as they could demonstrate students were performing well, districts should could retain local control of education.
In Ashfield, board members asserted that schools need to use measurable student data to limit state oversight. In an interview, the board chair shared, “I’m not a big fan of standardized testing because I don’t think it captures everything. But I do understand there needs to be some account.” Although Ashfield’s board did not agree with mandated testing, they began to move toward adopting more formalized assessment measures as a means to demonstrate their academic strengths. They perceived this as an important step to protect the school district from AOE oversight.
Collectively, the three case study school boards made sense of assessment data as different types of tools: an evaluative tool to assess dimensions of school performance; a hortatory tool that can engender community support for school funding and programming; and a buffering tool that enables retention of local control in the districts. Contrary to prior research on school and district leader sensemaking and policy enactment, board sensemaking of assessment data did not tightly align with board members’ individual sensemaking. Furthermore, how the boards leveraged assessment data as tools deviates from both federal and state messaging about how and why such information should be employed.
Discussion and Implications
The study findings illustrate school board sensemaking of the purpose and enactment of assessment policies. Although board members had different perspectives about the value of mandated assessments, the boards collectively decided to use assessment data in similar ways: to measure educational outcomes, to foster community support of schools, and to preserve local control by buffering districts from state oversight. Their primary use of assessment data neither directly aligns with federal or state messaging, reflecting board members’ synthesis and adaptation of divergent messages into one locally driven narrative.
One significant finding from the study is board members’ sensemaking of assessment data does not correspond to how they collectively use assessments as policy instruments. Previous research finds a tighter relationship between policy sensemaking and policy implementation (e.g., Coburn, 2001; Spillane et al., 2002). However, this study finds school board sensemaking differs from that of educational leaders and practitioners, as board members’ individual sensemaking was less significant than the local environment and messaging in shaping collective sensemaking.
A second major finding is that collective board sensemaking did not fully align with either federal or state policy messaging. The US DOE used hortatory policy instruments to signal that student assessment data could be used to measure schools, principals, and teachers and use individual student data to evaluate educational equity. Vermont’s AOE messaged that testing data could be used as one tool of many to evaluate schools and educational equity, but opposed using testing data to evaluate principals and teachers. School boards synthesized these messages in their sensemaking, as they believed student performance data could be used to evaluate their schools, and for two of the boards, that the data could also be used to evaluate principals. However, none of the boards perceived that testing data could be used to evaluate teachers or educational equity. Aligning with both federal and state-level messaging, boards used student performance data as a hortatory policy instrument to maintain community support for local schools. Boards also perceived that student assessment data could buffer them from state oversight and protect local control, a use that was not explicitly conveyed outside of the districts. Board sensemaking of mandated assessments and student performance data therefore incorporate aspects of federal and state messaging within the local environment. These findings therefore extend research by Spillane (1998, 1999, 2000) that local context shapes district leaders’ modification of policy implementation.
There are several factors that may have influenced how boards made sense of federally mandated assessments policies. NCLB’s assessment-driven design encompassed decades of accountability policymaking (Hursh, 2007; Mehta, 2013). The dominant perception that high-stakes testing can effectively measure schools has become an enduring national narrative established through years of political and public discourse (McDonnell, 2013; Mehta, 2013). Given the long-standing durability of this paradigm, the national narrative likely influenced board sensemaking.
On the state level, Vermont’s educational leaders and agencies explicitly and consistently messaged the value and purpose of assessment data as a local hortatory policy instrument. Nonetheless, the boards did not entirely integrate state-level messaging; in some cases, how they made sense of the value assessment data was contrary to AOE recommendations. The local and state misalignment was likely shaped by Vermont’s atypical enactment of NCLB. Although state leaders opposed NCLB, they also withdrew the ESEA waiver, making Vermont one of the few states that continued to hold districts accountable to NCLB’s 100% proficiency benchmark. The AOE messaged a state-specific perspective on the value and use of assessment data that directly opposed federal messaging. Nonetheless, school districts still had to externally report their assessment data. The disconnect between Vermont’s messaging and policy mandates may have generated “unintentional sensegiving” wherein ambiguous or inconsistent messages from upper levels create greater variance in implementation of educational reforms on the lower levels (Wong, 2019). Vermont’s AOE used hortatory policy instruments to signal opposition to aspects of NCLB assessment reporting, yet their policies still held districts accountable to NCLB, thereby sending inconsistent messages to local boards. It is possible that Vermont’s unintentionally ambiguous sensegiving of assessment reporting subsequently influenced board sensemaking of the policy.
Although board sensemaking neither directly aligned with federal or state messaging, the cross-case alignment suggests local influences on the sensemaking process. All board members in the study shared the same superintendent, who spent the majority of his time on policy-related work with the school boards (Hall, 2016; Hall & McHenry-Sorber, 2017), thereby suggesting he is the primary local influence. However, in my analysis, I found the superintendent’s messaging closely aligned with that of the AOE and did not reflect the ultimate use of assessment adopted by the boards. The superintendent communicated state policy to the local school boards, but he did not appear to be a major influence on their sensemaking of the purpose and use of assessments.
The findings from this study have implications for policymakers and district leaders undertaking the shift to LEA implementation of ESSA state-based accountability plans. The plans encompass multiple components to measure schools, necessitating training and support for LEA implementation. School board sensemaking and subsequent policy implementation are likely to deviate from that of superintendents and principals. Three recommendations may help develop board members’ understanding of new state accountability policies. First, SEAs should include school boards in the refinement of state accountability plans and provide opportunities for board feedback to ensure these local voices are included. Second, SEAs can communicate the details of accountability plans directly to school boards, and the messaging should be timely, accessible, and relevant to the work of boards. Filtering communication through the superintendent or other agencies may dilute or undermine the messaging to boards. Third, states should provide ESSA-related training specifically for boards. Because board members are notoriously avoidant of professional development (Gawlik & Allen, 2019; Hall, 2016), it is critical that any training is explicitly designed for school board members. In this study, participants expressed strong interest in learning more about policies, provided that training was responsive and accessible to board members. Finally, as sensemaking is a recursive process, it is essential for SEA and LEA leaders to continually engage with their boards about educational policies. Board members, and their districts, will ultimately benefit from ongoing opportunities to refine their individual and collective sensemaking about educational policies.
Conclusion
Assessments are a policy tool much like any other, and as such, school board members will use these tools for their own devices. The hortatory qualities of standardized assessments can be powerful; yet it can be hard to control how implementers will make sense of them. As states develop greater flexibility with their accountability systems, it is important to consider the legacy and context of what school board members already understand and believe about testing. School board members play a central role in supporting and sustaining—or countering and undermining—ESSA accountability implementation. Therefore, it is crucial for states to improve communication about the new plans directly to board members, include board members in plan development, and provide greater access to training. Starting over with ESSA assessment plans is not the same as starting with a blank policy slate. The messaging of high-stakes testing may be the most enduring legacy of federal oversight in education.
Footnotes
Appendix
Declaration of Conflicting Interests
The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The author(s) received no financial support for the research, authorship, and/or publication of this article.