Abstract
As accountability systems have increased demands for evidence of student learning, the use of data in education has become more prevalent in many countries. Although school and administrative leaders are recognizing the need to provide support to teachers on how to interpret and respond to data, there is little theoretically sound research on data-driven decision making (DDDM) to guide their efforts. Drawing on sociocultural learning theory, extant empirical literature, and findings from a recent study, this paper develops a framework for understanding how to build teacher capacity to use data, specifically informing what practices administrators might employ, when in the DDDM process to employ these practices, and how these mechanisms may build teacher knowledge and skills. Given the global economic climate, administrators face difficult choices in how to invest scarce resources to support data use and once invested, how to ensure that teachers gain, and sustain, the needed capabilities once the supports are removed. The framework provided herein presents a set of concepts that may be useful in guiding these decisions. Implications for leadership practice, as well as suggestions to guide future research and theory development, are discussed.
Keywords
An increased availability of technology, financial support from policymakers, and greater accountability for student outcomes have all contributed to the increased focus on data use for educational improvement globally (Hamilton et al., 2009; Mandinach, 2012; Schildkamp and Lai, 2013; Wayman and Stringfield, 2006). This international trend has additionally been fueled by international rankings based on student outcome data, such as results from the Programme for International Student Assessment (PISA) (Sanders, 2008). Recent research has examined this trend in schools in Holland (Schildkamp and Kuiper, 2010), New Zealand (Lai and McNaughton, 2013), England (Downey and Kelly, 2013), Ireland (McNamara and O’Hara, 2006), Ghana (Prew and Quaigrain, 2010), and South Africa (Archer et al., 2013), among others. Similarly, in the USA, advocates tout data-driven decision making (DDDM) as the next major strategy to support instructional improvement and student achievement, or, as some have described, “the mantra of the day” (Ikemoto and Marsh, 2007). DDDM refers to “teachers, principals, and administrators systematically collecting and analyzing various types of data…to guide a range of decisions to help improve the success of students and schools” (Marsh et al., 2006: 1).
Research suggests, however, that although educators have access to a broad range of data (e.g. results from formative classroom, common grade, district interim, and state standardized assessments; student work; and observations of teacher instruction), they do not always know how to use data in a way that leads to deep changes in instruction and improved student outcomes (Heritage et al., 2009; Olah et al., 2010). Teachers and administrators may lack adequate skills and knowledge to formulate questions, select indicators, and identify solutions (Marsh et al., 2006; Cosner, 2012; Means et al., 2011; Olah et al., 2010; Supovitz and Klein, 2003). Another set of challenges arises as teachers work to incorporate these new expectations for data use into their practice, making sense of them in relation to their current beliefs and expectations (Young, 2006). While district and school leaders play a critical role in supporting DDDM in schools, they too face challenges in supporting teachers in this work, such as lack of time, expertise, and tools (Anderson et al., 2010; Copland et al., 2009; Cosner, 2011a, 2011b, 2012; Park and Datnow, 2009; Supovitz and Klein, 2003; Young, 2006). For example, Cosner (2011b) found that principals struggled with effectively supporting teachers’ data use, providing overly general guidance or emphasizing the importance of examining future instructional responses to the exclusion of deep examination of past practice.
To date, educational leaders in districts and schools have invested in a wide range of interventions to help improve teachers’ capacity to engage in DDDM—from system-level reform initiatives to more narrowly focused workshops, tools, and technology (for full review which includes international references, see Marsh, 2012). However, most research on these interventions is atheoretical and incomplete, providing little information on what constitutes effective capacity building and under what circumstances it occurs (Marsh, 2012; Coburn and Turner, 2011). Scholars have called for a move away from normative claims, advocacy work, and how-to guides (prevalent in the literature on DDDM) to more analytic and theoretically driven research, noting that stronger theoretical frameworks will enable deeper understanding of the dynamic between educational interventions and on-the-ground responses and actions (Marsh, 2012; Coburn and Turner, 2011; Little, 2012; Moss, 2012; Spillane, 2012; Young and Kim, 2010). It is this call to which our paper responds.
We posit that the field of educational administration and instructional leadership would benefit from a sociocultural learning perspective when considering how to best support teachers as they interpret and use data to inform their instruction. We develop a framework based on extant literature, as well as our research on three promising types of capacity-building interventions (CBIs): literacy coach, data coach, and data team. 1 Not only are these CBIs widely implemented, some research studies suggest they all play an important—but potentially varying—role in building teachers’ skills and knowledge to analyze data and identify instructional responses.
Widespread throughout the USA, literacy coaches have become a central part of federal, state, and district literacy reforms. Defined as specially trained, master teachers who offer on-site and ongoing support for teachers—one-on-one or in groups—so they can improve the literacy skills of their students, literacy coaches often perform multiple roles, of which data support may be only one (Marsh et al., 2008; Coburn and Woulfin, 2012; Rodgers and Rodgers, 2007; Smith, 2007). Studies find that coaches can help teachers become more expert in interpreting data, understanding student thinking, and designing instructional responses (Marsh et al., 2009; Chrismer and DiBara, 2006; Means et al., 2010; Roehrig et al., 2008). According to a US survey, half of districts have made available data analysis experts or coaches to at least some of their schools (Means et al., 2010). Providing specific guidance on interpreting and using data, data coaches may be school-based or shared among a set of schools, and are sometimes associated with the central office or an intermediary organization (Lachat and Smith, 2005; Love et al., 2008). Little research exists on the effects of data coaches on teacher or student outcomes. Finally, data teams are frequently associated with data-driven reform initiatives and often take the form and name of professional learning communities or inquiry groups (Nelson et al., 2008; Vescio et al., 2008). They typically involve collaborative work among peers, guided by a lead teacher or facilitator. One recent study suggests that working in a small group setting promoted more sound data interpretations, with colleagues clarifying and correcting analysis errors (Means et al., 2011). Other studies point to positive effects of data teams on teacher beliefs, understandings, and practice, although these effects are not always universal to all participating groups (Gallimore et al., 2009; McDougall et al., 2007).
In this article, we build the components of our capacity-building framework. We start by describing the normative theory of action underlying DDDM and provide an overview of key concepts from and applications of sociocultural learning theory that inform our understanding of capacity building for data use. After describing our methods, we present research from a year-long comparative case study of six low-income secondary schools where school and district leaders were committed to this work. We highlight the key practices and artifacts commonly employed in CBIs, challenges to their enactment, and conditions that appear to mediate the capacity-building process. Finally, we discuss implications for administrators, outline potential limitations of sociocultural learning theory in understanding this phenomena, and offer suggestions for future research.
Data-use theory of action
The theory of action for data use promoted by data advocates and adapted from the literature suggests that data alone do not ensure use (Ackoff, 1989; Marsh et al., 2006; Mandinach, 2012; Mandinach et al., 2008). Instead, data must be collected, organized, and analyzed to become information and then combined with stakeholder understanding and expertise to become actionable knowledge (center box in Figure 1). A teacher is then expected to apply this knowledge to instructional practice. Depending on how this process plays out, the same raw data may point to very different solutions and actions depending on the situation and judgment of data users. Once a teacher has acted and outcomes have resulted, these results and new data can be collected to assess the effectiveness of actions, leading to a continuous cycle of collection, organization, and synthesis of data in support of instruction and improvement.
Capacity building for data-driven decision making.
Applied to classrooms, this model recognizes that the DDDM process is not necessarily as linear or continuous as the diagram may first appear. In the process of data analysis, a teacher may realize she needs additional data before identifying an appropriate instructional response. Also, not all data-use activities will complete the full cycle; a teacher may filter test scores into a graphic display but do nothing more with the information.
This conception of DDDM implies a set of “data literacy” competencies that may be needed to engage in meaningful data use and move from data, to information, to knowledge, to action (Knapp et al., 2006; Means et al., 2011). These competencies may include the ability to examine multiple measures, synthesize data, and draw inferences. Extant literature suggests not all teachers possess these skills. One study found that while most teachers were capable of finding information on a graph, they had difficulty comprehending complex data displays and showed a limited understanding of key statistical concepts of test validity, score reliability, and measurement error, leading to invalid inferences (Means et al., 2011). Content and instructional knowledge also play a critical role in a teacher’s ability to connect the gap between identified problems and the appropriate instructional response (Goertz et al., 2009: 241).
As illustrated in Figure 1, one can imagine multiple opportunities to assist a teacher in this process and to build stronger data literacy and instructional knowledge. A CBI may support teachers in accessing or collecting data (1); organizing, filtering, and analyzing it into information (2); combining information with expertise and understanding to become actionable knowledge (3); knowing how to respond and adjusting their instruction (4); and evaluating the effectiveness of the response or outcomes that result (5) (Marsh, 2012).
In the next section, we consider the utility of sociocultural learning theory for educational leaders’ understanding of how to assist teachers in this process of data use.
Sociocultural learning theory and its recent application
We draw on sociocultural learning theory to ground our understanding of capacity-building efforts for teachers’ data use. According to this approach, learning is inherently a social phenomenon where individuals make sense of information and construct new knowledge based on prior knowledge, beliefs, attitudes, and experiences, and through activity and social interactions in everyday contexts (Vygotsky, 1978). Since Vygotsky’s original scholarship, a range of sociocultural theories of learning have emerged and evolved (e.g. Brown et al., 1989; Cole and Engeström, 1993; Collins et al., 1991; Engeström, 1999, 2000; Lave and Wenger, 1991; John-Steiner and Mahn, 1996; Nasir and Hand, 2006; Rogoff, 1990, 1993; Wenger, 1998). We present these ideas as complementary ways to understand capacity-building efforts.
Sociocultural theory assumes learning is embedded within social events, so in order to understand development, one must focus on how individuals participate in everyday, authentic activities involving their social peers, activities, and artifacts (Brown et al., 1989; Johnson and Golombek, 2003; Vygotsky, 1978). From Vygotsky’s (1978) original writings, a teacher or “more knowledgeable other” played a key role in supporting learning, where he or she assists, models, discusses, and supports an activity to increase the learner’s understanding and independent performance. Other scholars further developed this idea of a mentor–apprentice relationship as a “model of instruction that works to make thinking visible…[by] showing the apprentice how to do a task and helping the apprentice to do it” (Collins et al., 1991: 1–2). A mentor offers not only specific domain knowledge—the concepts, facts, and routines within a content area—but also heuristic strategies (“tricks of the trade”) and meta-cognitive strategies (Collins et al., 1991; Tharp and Gallimore, 1988). Although some describe this relationship in one-way, novice–expert terms, it can also be framed as a dynamic, two-way relationship in which both parties strengthen their knowledge, skills, and thinking (Rogoff, 1990, 1993; Tharp and Gallimore, 1988; Vygotsky, 1978).
Other sociocultural theorists have suggested that learning occurs as a function of a “community of practice,” a group of people with a shared interest who, through their regular “joint work,” improve upon that practice (Lave and Wenger, 1991; Wenger, 1998). Within a community of practice, individuals establish norms, build collaborative relationships, and negotiate meaning within the group (Wenger, 1998). Participants gradually absorb and are absorbed in a “culture of practice,” acting as each other’s exemplars, leading to the development of shared meanings, a sense of belonging, and increased understanding. Still others have attended to how activity systems mediate learning and development within cultural, historical, and social settings (Cole and Engeström, 1993; Engeström, 2000). Within an activity system, development may unfold vertically through a set of stages, as well as horizontally with the introduction of new voices and perspectives across activity systems (Engeström, 1999; Engeström et al., 1995).
Several scholars have used sociocultural learning theory to understand school and district improvement efforts (Gallucci, 2008; Gallucci et al., 2010; Honig and Ikemoto, 2008; Ikemoto and Honig, 2010; Knapp, 2008). From this set of studies, we glean three important insights. First, many of these authors argue that reconceptualizing practice for educational leaders through a lens of sociocultural learning theory is an important, but underdeveloped, area of study. Second, the findings across these studies generate a list of practices important to learning as understood by sociocultural learning theory: brokering, modeling, authentic practice, dialogue, opportunities for “joint work,” and development and use of tools. Third, they affirm the reciprocal relationship between assistance provider and learner. Our proposed framework builds on theory and extant literature to expand to a particular area of practice: building teacher capacity to use data for instructional improvement.
A framework for building teachers’ data-use capacity
Collectively, the theoretical and empirical literature presented from the sociocultural learning tradition helps frame our understanding of how school leaders can choose and support interventions that build teachers’ knowledge and skills to use data. Through this lens, capacity building is not conceived of as the transmission of a set of skills and body of knowledge, but instead as a learning process in which individuals make sense of information and construct new knowledge through activity and social interactions, mediated by prior knowledge, beliefs, and experiences.
As noted, there are multiple leverage points at which capacity building may occur when supporting data use, illustrated by the bold dotted arrows in Figure 1. The literature also suggests that as a CBI lead (e.g. coach or lead/member(s) of a data team) helps a teacher learn new skills, knowledge, and ways of thinking, these interactions provide the CBI lead with feedback and opportunities to reflect and improve on her own practice (represented by the double-headed arrows in Figure 1). This work varies along the following dimensions—unit of interaction, practices, and artifacts—and is moderated by contextual factors.
Unit of interaction
The work of CBIs can vary by the unit of interaction (i.e. in a group and/or one-on-one). As noted, one stream of sociocultural learning theory describes the importance of the close interaction between novice and mentor (Collins et al., 1991). However, a CBI may also be represented by a group of practitioners or peers working together. Benefits of working within a larger group include collective problem solving, confronting a greater number of ineffective strategies and misconceptions, and providing collaborative work skills (Brown, 1997; Lave and Wenger, 1991; Stoll et al., 2006; Wenger, 1998).
Practices
CBIs can also vary in the practices employed in this work. By formally and informally assessing teacher needs around data-use and instructional capacity, a CBI lead can best target data-use activities that will meet the teacher slightly above her current level of independent practice. 2 The CBI lead can then work with the teacher to advance her data-use knowledge and skills through other practices described below, such as modeling, feedback, and dialogue. Modeling includes two linked processes: a physical demonstration of an activity paired with an explicit verbal explanation of the thinking process. By “making the thinking visible,” the mentor helps a novice build a conceptual model and acquire an integrated set of cognitive and meta-cognitive skills needed to monitor and appropriate the tacit thinking processes that underlie an activity (Collins et al., 1991; Lave, 1988).
Underlying many of the other practices is the opportunity for observing. As part of modeling, the coach may watch as a teacher tries out a new practice and then corrects, re-directs, or supports as needed. For someone new to a practice, observing is equally critical as a way to study and examine modeled behaviors in situ (Brown and Duguid, 1991). Providing feedback and sharing expertise is closely related to observations. Feedback can be a component of vertical learning where a mentor observes a novice and provides advice on how to advance the practice, as well as part of horizontal learning among peers and across communities (Engeström, 1999; Engeström et al., 1995; Rogoff, 1990; Tharp and Gallimore, 1988). This feedback may include a range of information from pertinent content knowledge to “tricks of the trade” (Collins et al., 1991).
Dialogue and questioning play a fundamental role in learning. In their work together, a mentor can engage with a novice, explaining, questioning, and verbally encouraging the teacher to get to a deeper level of understanding (Brown, 1987, 1997; Tharp and Gallimore, 1988). Novices publically engage in reflection through dialogue, “[developing] ultimately, an internal cognitive model of expertise” (Collins et al., 1991: 14). Dialogue is also critical to a community of practice: peers have the opportunities to engage in conversations that can lead to new shared information and deeper understandings (Lave and Wenger, 1991; Wenger, 1998). Finally, CBI leads may engage in brokering: translating, coordinating, and aligning the interests across different communities. A broker needs to have enough legitimacy as a member of each group to be able to influence the development of practices but maintain enough independence as to not be rejected by the other group (Engeström, 1999; Engeström et al., 1995; Wenger, 1998).
Artifacts
The work of CBIs and teachers may also differ by the artifacts with which participants engage. Artifacts are physical and symbolic tools created and adapted over time (Cole, 1996; Vygotsky, 1978; Wertsch, 1998) and play a mediating role in the enactment of the practices described above. According to Wenger (1998), tools are reifications, the manifestation of new ideas. A tool may take different forms, for different purposes (Wartofsky, 1979). A conceptual tool, for example, is used to share ideas about “big picture” principles, concepts, frameworks, and theories to guide thinking while a practical tool includes “practices, strategies, and resources that…have more local and immediate utility” (Grossman et al., 1999: 14). Regardless of their type, tools are culturally constructed and modified through individual and group use over time (Johnson and Golombek, 2003). The appropriation over time by the user of the underlying conceptual underpinnings or the “way of thinking” within a particular social and cultural environment is of critical importance (Grossman et al., 1999). Norms and values are another type of artifact likely to mediate practice and learning. Theory suggests that a set of norms often guides a group’s time together (Wenger, 1998). Within successful professional learning communities, these norms promote shared values and vision, a sense of collective responsibility, reflective professional inquiry, collaboration, promotion of group and individual learning, and inclusion of all members (Stoll et al., 2006).
Context
Finally, capacity-building efforts are embedded in a broader context that likely mediates the process and results. As noted earlier, sociocultural learning theory suggests that environmental, cultural, and historical factors are a critical part of the capacity-building process (Schunk, 2008; Vygotsky, 1978). After providing an overview of our research methods, we present some of our findings as a way to illustrate these dimensions of capacity building and the application of this framework to data use in schools.
Research methods
We draw on data from a year-long comparative case study of three interventions intended to improve teachers’ capacity to use data to improve literacy teaching and learning in six low-income, secondary schools in four districts in the USA (Merriam, 1998; Ragin and Becker, 1992). We seek to answer the following research questions: 1) How do these key dimensions of CBI activity—the unit of interaction, core practices, and artifacts—unfold in the capacity-building process in schools? 2) What are the challenges to its enactment? 3) What contextual factors mediate this process?
Study sample
Districts and schools were purposefully selected to maximize the conditions identified by prior research as supporting effective interventions (e.g. training for CBI leads, organizational resources devoted to the intervention), to ensure that the CBI had been in place for a minimum of 2 years, and to provide variation in characteristics of CBIs (e.g. content-area expertise, type of data that is the focus of their work). Three of the four districts were medium-sized, located in one state, and had similar demographic profiles. More than 75% of students were Latino, more than 60% were eligible for free or reduced-price lunches (FRL), and more than 25% were English language learners (ELLs). The fourth district was in a different state and was much larger and diverse in a different way. Approximately half of all students were Caucasian and half African American; fewer than 5% were ELLs, and about 40% qualified for FRL.
The size of case study schools varied across districts (340 to 800 students), but all six enrolled significant proportions of students of color and/or ELLs 3 and had failed to meet state accountability targets for more than 5 years. Each school implemented a main CBI sponsored by their district: two schools had literacy coaches, two had data coaches, and two utilized data teams. 4
Data collection and analysis
During the 2011–2012 school year, we visited each of the schools and districts at least three times. In each school, we identified two to three case study teachers, who primarily taught language arts. In all, we conducted interviews with district leaders (n = 13); school administrators, CBI leads, and case study teachers (n = 79); focus groups (n = 6) with non-case study teachers (n = 24); observations (n = 20); and document analysis.
All interviews were taped, transcribed verbatim, and coded using NVivo qualitative analysis software. Coding and analysis were continuous and iterative (Miles and Huberman, 1994; Strauss and Corbin, 1998). We began our coding with an initial set of capacity-building practices predicted by sociocultural learning literature, as well as codes for practices that had emerged from early empirical work. We later modified the code list to capture a more comprehensive list of topics, including specific practices, artifacts, challenges, and contextual conditions. The findings presented below have been selected from data collected across the six schools to illustrate the dimensions of the capacity-building framework.
Elaborating on the framework: Our research findings
Next, we present core dimensions of CBI activity observed across the six schools.
Unit of interaction
Several districts modeled their interventions on the novice–mentor relationship with the expectation that teachers meet one-on-one with a coach. In one district, the literacy coach initiative was seen as a way of introducing content, curricular, and data analysis expertise at the school sites. An assistant principal (AP) at one school in the district believed that their coach’s one-on-one assistance was particularly helpful when data analysis was the focus of the work. She explained: Because different grade levels are at different collaboration abilities, [our coach] is able to have an opportunity to develop a thinking process in their approach to data analysis in an individualized way and in a very comfortable way for people who might not be comfortable asking questions amongst their peers…. [where] it would be too exposing or too revealing for them to open up in a way where they reveal that they don't know something.
In contrast, the CBI in another district focused on groups of teachers working together on a regular basis to analyze assessment results and plan instructional responses. One seventh grade teacher reported that working in a group was preferable to her time spent with her coach because of the shared identity of team members: I have [a] coach, but there’s just a sense of disconnect in that they don’t necessarily know what you’re going through… [While] the teachers know the exact standard you’re hitting, so they have more strategies to offer you. They have a bank of ideas to put forward.
However, at all sites, we found that the distinction between one-on-one and group interaction was more nuanced then we originally anticipated. Rather, it was more appropriate to consider the unit of interaction at a situated point in time. For instance, within a data team, individual teachers often asserted their expertise within the group setting. Although group members explicitly reported that they were on equal footing, a more experienced teacher frequently took on the role of mentor when working with a teacher new to the group. Likewise, literacy or data coaches at times engaged with teachers in department or grade-level meetings in addition to one-on-one meetings.
Some educators made deliberate choices at different points in time regarding the appropriate unit of interaction. One coach spoke about the benefits of facilitating critical conversations around instruction and doing “heavy” coaching in a group setting to avoid defensive posturing, leaving the “light” coaching to one-on-one interactions (for more, see Huguet et al., 2014; Marsh, et al., in press). Others, such as the AP above, expressed the opposite position, viewing one-on-one interactions as the more appropriate venue for potentially emotionally charged conversations about individual data and practice. Although the theory and our emerging research remain agnostic on the benefits of one-on-one versus group support, they suggest that lines may be blurred and that different stages of DDDM might call for different modes of interaction over time.
Practices
Our research sheds light on a core set of practices utilized across all six schools and the challenges to their enactment. As illustrated in Figure 1, these practices applied to multiple stages of the data-use process. For example, modeling at times assisted teachers with analyzing data and at other times adjusting their instruction in response to knowledge gained from analysis. Although we describe them separately below, many of these practices are inherently connected to one another.
Assessing teacher needs
Although not as pervasive as some of the other strategies, the practice of assessment generally occurred at the start of the year or a meeting. Literacy coaches in one district were encouraged to conduct a needs assessment of their teachers to create specific goals for their data-use work together. One literacy coach explained: I look at it like the teachers are my students…you need to know what their strengths are and what their weaknesses are. You build on their strengths, but then you look at, what’s the next area they could work on? … It varies with each teacher.
The literacy coach then designed her coaching plan to target the needs identified by the assessment.
Although often conducted independently by the CBI lead, these assessments were at times co-constructed. Prior to observing a teacher, one literacy coach asked teachers to identify where they were struggling and what they wanted her to look for. Similarly, data teams at times conducted joint assessment as part of a goal-setting process, helping to identify where they needed to channel their time and energy. Following up on these assessment results, however, was a challenge. A lack of time or other tasks directed from administration sometimes prevented the CBI lead from adequately responding to assessed needs.
Modeling
Modeling around data use involved both explaining and demonstrating ways to interpret, respond to, and act on data. One new teacher described how her data team’s lead teacher modeled the teaching of summary writing. “She showed me how she teaches it, she gave me a graphic organizer that she uses, and she gave me examples [of student work],” the teacher explained, “She gave a very thorough explanation of what she does.” By simultaneously modeling and articulating thoughts and reasons behind these actions, the CBI lead provided the teacher with meta-cognitive skills and explicit tips on how to do this work independently in the future.
Another approach to modeling we observed throughout our visits involved gradual release—providing more explicit and direct support at the outset and then, over time, removing the supports so that a teacher could continue unassisted (Collins et al., 1991). At the beginning of the year, one literacy coach determined that a teacher did not know how to use the data management system to group students by their proficiency levels and clusters of standards on the state assessment. During their first meeting, the coach provided the teacher with this analysis in the form of a spreadsheet and modeled how to do it on the computer program. In their subsequent meetings, the teacher “took on” this work independently, allowing for their joint conversations to focus on how to respond to the assessment data.
CBI leads, nevertheless, at times struggled with aligning their modeling approach with different teacher needs. Recognizing the need to adapt the way she modeled how to retrieve data from the district’s data system, one literacy coach explained: I showed them two different ways. I showed them the way I use [it with] my “techies.” My techies love the electronic versions where they just plug in the numbers and, boom. Then I have my teachers that like to use the sticky notes where they’re putting the students in this box and that box.
Observing
We observed a range of observational practices, both formal (e.g. lesson study using protocols) and informal (e.g. walkthroughs). CBI leads regularly observed a teacher to monitor how she tried out or engaged in a particular phase of the data-use process or its impact on her instruction. For example, one new teacher found that being observed by the school’s literacy coach was extremely helpful for her own practice. After modeling an instructional strategy, the literacy coach immediately watched as the teacher attempted it herself. The coach took notes and provided the teacher with immediate feedback.
Although several CBI leads used observations effectively to move teachers from actionable knowledge to improvement of instruction, others struggled greatly with this practice. Many of these difficulties stemmed from teachers’ perceptions that the CBI lead lacked the relevant expertise, experience, and skills to provide credible and useful insights based on the observed instruction. These perceptions limited teachers’ willingness to either invite the lead educator into their classrooms or their openness to receiving feedback (described further below).
Providing feedback and sharing expertise
Feedback often involved suggested next steps for practice. For instance, within a group setting where teachers analyzed student work together, one teacher greatly valued her peers’ suggestions: The feedback that you get from the other eyes… Those are always beneficial because you’re usually stuck in your mind, so it’s hard to see outside the box. They bring in that new perspective, [asking] ‘have you ever considered this?’
Much like observations, provision of feedback relied greatly on the relationships among educators within the CBI and the interpersonal skills of CBI leads. Several coaches struggled with this practice and found it challenging to provide useful and critical information to push teachers’ thinking and practice, while delivering the message in a non-judgmental way. One coach, for example, refused to provide unsolicited feedback after conducting walkthroughs for fear of being perceived as an evaluator. Data team members also struggled at times with this practice. Several veteran teachers acknowledged discomfort in recommending instructional ideas to their peers. One worried that she would be seen as a boasting “shining star” for suggesting others adopt her ideas and usually provided the feedback in emails instead of in person.
Dialogue and questioning
In all four districts, dialogue allowed teachers and CBI leads to discuss and reflect on the meaning of data and how to act on new knowledge (for more on the role of dialogue, see Marsh et al., in press). Again, we observed a range of formal (e.g. meetings with protocols guiding discussion) and informal (e.g. casual conversations in hallways) practices across the six schools. The questioning techniques of CBI leads proved to be an important element of this practice. One educator noted that well-crafted questions such as “What stands out to you here?” helped her to facilitate, but not dominate, conversations.
In one district, most of the teachers interviewed described discussion as the main activity of their data teams, with a focus on instructional response rather than data analysis. The general pattern in their dialogue was to compare results on a common assessment, identify groups performing better, recognize what those particular teachers did instructionally, and then decide what all other teachers could do to improve results. The dialogue gave teachers the opportunity to reflect on how their own previous instructional choices may have led to particular student outcomes.
Not all examples of dialogue reflected deep analysis and reflection, and in many instances, educators struggled to move beyond superficial discussion or attribution of problems to students to richer analyses of instructional practice. Another challenge pertained to the perceptions among teachers of the evaluative role of the CBI lead. In cases where teachers clearly understood that the CBI lead would not report information back to administrators, teachers were more open in their dialogue. Without this climate of safety, CBI leads in other schools clearly struggled to facilitate meaningful and non-defensive dialogue.
Brokering
We saw CBI leads brokering, at a micro-level, the divide between data and application through their ability to connect teachers to expertise and resources that support the data process. For instance, data coaches in one district reported accessing data, disaggregating it, printing out data reports, and providing resources on how to act upon conclusions drawn from the data. One data coach explained: “The major component that’s missing for majority of teachers… is ‘what to do next’? So we come up with, what are the corrective strategies?”
CBI leads also acted as brokers between different communities within schools or districts. For example, one data coach referred to herself as the “middleman” between the principal and the teaching staff. In some cases, CBI leads communicated or translated district data initiatives to the faculty while simultaneously providing district administration with teachers’ feedback, which often shaped future curricular development. According to one district administrator, coaches were “a kind of conduit between the two groups [district and school sites].”
Echoing past research, we found that brokering was often fraught with difficulties (Marsh et al., 2010; Weatherly and Lipsky, 1977). At times, confusion around the CBI lead role complicated her ability to effectively bridge between teachers and administrators and serve in a support role. One literacy coach reported: It can be very nebulous. You are an instructional coach, but you have to continue to work on these district curriculum maps, and you support the administrative team with whatever direction they want to go in, and you monitor students, and it’s just—it’s a lot.
Artifacts
CBIs used multiple types of tools, either created by CBI leads, co-constructed with teachers, or adopted from external sources. CBI leads provided teachers with conceptual tools (e.g. a framework for thinking about the data-use cycle) and practical tools (e.g. a worksheet for recording analysis) to help teachers engage in the process and negotiate meaning from the examination of data. Another common tool was a data report or presentation. CBI leads constructed these documents as way to filter and present data in simpler, more understandable forms for teachers. These lead educators used the tool to spark dialogue and reflection (“you can talk about it, but to actually see it is something different”), at times increasing the sophistication of the reports over time to deepen the conversation.
The appropriation by a teacher of these tools occurred as a process over time as a teacher tried out a tool, used it superficially, grew to understand the conceptual underpinnings of it, and, in the mastery of the tool, individualized and further adapted it. In one school, the data teams were required to fill out specific worksheets to guide their data analysis and describe their next instructional steps. At the beginning of the year, teachers in one team believed these tools were “too structured” and used by administrators for “top-down” monitoring. Later in the school year, this team saw them as more helpful as a record of their past and future instructional plans, particularly after they were able to revise them to better serve their needs.
In both the group and one-on-one settings we observed, the work of CBIs included articulating and enforcing rules of engagement, values, and expectations that supported open, critical inquiry around data and instruction. For instance, a teacher in a data team described how her team established norms for effective inquiry: We talk about what it is that is important in effective communication. We go around and if the idea is accepted by the majority of the group, we post it as a norm, [such as] one person speaks at a time, assume positive intent, be here on time,…have a pre-established agenda, have a facilitator.
Of course, tensions sometimes surrounded the use of tools and norms, particularly when they were seen as externally imposed. For example, teachers in one school believed data protocols were used to micro-manage their data teams and were overly burdensome. One teacher argued: It’s a ‘Let me check up on you, I want to make sure you're doing what you're supposed to be doing’ thing… You're just wasting your time with it, the fact that we have to record all of these different components… Because when you have to figure out what box to put it in on the form, then that becomes your discussion for five minutes. Times that by five boxes, you're wasting half of your time figuring out, ‘Where do I put this?’
Confirming past research (Marsh et al., 2006; Gearhart and Osmundson, 2009; Honig and Ikemoto, 2008), without an understanding of the purpose and theory behind the tool and the opportunity to participate in its development, the tool was not well received and implemented superficially. Similarly, teachers in a school with a data coach expressed less buy-in to the DDDM process when they viewed norms and processes being imposed externally. According to one teacher: The previous meeting [of grade-level teachers] was a real disaster.… The data coach came in, and she had an agenda with the norms of collaboration. She imposed that on to the meeting…. She wasn’t really letting us be a professional learning community or be a lesson planning group of teachers.
Conditions that moderated the capacity-building process
Consistent with sociocultural learning theory (Schunk, 2008; Vygotsky, 1978), our data analysis confirmed that environmental, cultural, and historical factors were a critical part of the capacity-building process and frequently mediated CBI activities. In fact, the challenges of implementing practices described in the previous section are the direct result of these factors. We group these conditions into four categories: intrapersonal, interpersonal, structural-organizational, and environmental. We highlight some of our findings to illustrate these factors and, when relevant, some of the supporting empirical evidence from other studies (for further reviews of literature see Marsh, 2012; Coburn and Turner, 2011).
Intrapersonal factors
Echoing past research, certain characteristics of the teacher and CBI lead appeared to affect capacity building, such as the level of engagement; prior understandings around data use and content knowledge, personal values, experiences, and expectations; and level of alignment between individual goals and new strategies (Honig, 2008; Johnson and Golombek, 2003; Stoll et al., 2006; Young, 2006). For example, one of our case study teachers who had long believed that using assessment data helped inform her instruction was more willing to be observed and engage in dialogue than others without this orientation.
The CBI lead’s expertise and skills also affected interactions with teachers. A CBI lead with expertise in both literacy and data use was valued as a “more knowledgeable other” more so than a CBI lead without one or both of those areas of expertise. Our emerging findings suggest that expertise in data use (e.g. experience using data systems, disaggregating data, identifying patterns) supported CBI–teacher work around data access and analysis, while content-area expertise (e.g. knowledge and experience teaching literacy) was particularly important to bridging the “knowing–doing” gap (Pfeffer and Sutton 2000), helping teachers select instructional responses to data. In addition, the CBI lead’s ability to work with adults often trumped the relevance of all other types of expertise. For instance, one CBI lead had strong data and content-area expertise but lacked interpersonal skills; she failed to make progress with teachers at all stages of the DDDM process.
In relation to her role as “broker” between different perspectives (e.g. research and practice), a CBI lead needed the skills to translate, coordinate, and align multiple perspectives (Wenger, 1998). In her role as “broker” between different groups (e.g. administrators and teachers), a CBI lead needed the ability to ensure confidentiality that sustained legitimacy across both groups. One literacy coach, for instance, maintained her credibility with teachers by co-teaching classes and creating relationships with students. At the same time, the administrative team valued her for her ability to “communicate their message” to teachers.
Interpersonal factors
Several group-level factors appeared to influence the collective’s ability to engage in productive learning. For example, in one of our data team schools, two members’ 4-year history of working together positively shaped group dynamics and facilitated an openness to sharing data and critically examining practice. This group engaged in deep reflection of practice not observed in settings lacking strong interpersonal ties. Other studies have similarly shown the importance of the level of commitment to standards for participation and sustained engagement (Gallucci, 2008; Grossman et al., 1999; Honig, 2008; Stoll et al., 2006).
Across all of our case study schools, teachers and CBI leads also pointed to trust as a pre-condition facilitating data analysis and critical examination of teacher practice. Those lacking trust often resisted attempts to use tools, participate in observations, and engage in dialogue. Research and theory have similarly identified the development of trust and perceived credibility as important conditions fostering positive relationships and adult learning in both one-on-one and group settings (Marsh et al., 2008; Ertmer et al., 2005; GWU, 2001; Means et al., 2010; Nelson et al., 2008; Park and Datnow, 2009).
Structural-organizational and environmental factors
Across our sites, district and school leadership, ongoing professional development, and dedicated time were commonly cited as important facilitators of teacher–CBI learning. Conversely, a lack of time, training, and leadership frequently inhibited this work. For example, leaders bringing a compliance orientation to district policy often translated this mindset to the implementation of the CBI, as evidenced by superficial and compliance-oriented use of data analysis tools. Once again, other studies of data use also indicate that district-level conditions can influence CBI–teacher work, including structures that support a data-use initiative, leadership, funding, and intervention alignment with other policies (Farrell, in press; Marsh, 2012). Principal commitment, strategic selection of participants, and resources of time, space, and funding are other school-level factors that our research and other studies have found to enable CBI efforts (Cosner, 2012; Levin and Datnow, 2012).
Conclusions and implications
To date, data-use literature has been largely atheoretical, based largely in “how-to” guides and advocacy work. In this article, we have developed a framework for educational leaders—drawn from sociocultural learning theory, extant literature, and our own empirical analysis—that specifies a data-use process and strategies for supporting teachers. Next, we identify implications for leadership practice and future research and theory development.
Implications for educational leaders
With the continued focus on student outcomes, the expectation to use data is likely to persist for years to come. Yet given the current economic climate, administrators at all levels face difficult choices in how to invest scarce resources to support data use, and once invested, how to ensure that teachers gain and sustain the needed capabilities once the supports are removed. The framework provided herein presents a set of concepts that may be useful in guiding this decision-making process. Specifically, the framework suggests leaders consider a range of questions when designing data-use interventions.
What is the current level of data literacy in my organization, and at what stage in the data-use process is more support needed? For example, if teachers lack skills and knowledge in how to access and collect data, then perhaps investment in data management systems is needed. If teachers are well seasoned in interpreting data but weak in their ability to respond, then perhaps the emphasis belongs on interventions that provide opportunities to reflect on, observe, and receive feedback on instruction or access to greater expertise to help them adjust instruction.
When designing supports for teachers, what unit of interaction can be adequately supported? For example, if resources cannot support a one-on-one coach in schools, then what organizational resources will be dedicated to support a group-based approach? Is there enough time set aside to ensure regular meetings? Do teams members provide adequate access to needed content-area and technical expertise, and if not, from where might this expertise be leveraged (e.g. other teachers, consultants, district personnel)?
When designing supports for teachers, to what extent do they reflect the practices sociocultural learning theorists suggest are most effective for learning? For example, when a coach models, does she make her reasons explicit and “visible”? If data teams are used, do members jointly establish norms to guide their interactions and are structures in place to facilitate dialogue? What tools are available, and to what extent do they facilitate reflection and learning?
To what extent are conditions in place that foster these supports? While some of the conditions mentioned in the framework are not within the control of administrators (e.g. funding, alignment with national policies), there are many factors that merit particular attention such as leadership, dedicated time, and alignment of internal policies. Leaders may also want to consider ways to build interpersonal relations and trust among teachers and CBI leads, such as providing dedicated time for educators to get to know each other, protecting the confidentiality of data and safety of dialogue that occurs around these data, and ensuring clarity of coach roles.
Theoretical implications and directions for future research
Although we benefit from drawing on sociocultural learning theory, there are several important matters on which the theory is silent or underdeveloped. We highlight a few of these gaps and suggest directions for future research.
Sociocultural theory suggests that development goes from elementary processes to higher order, complex cognitive processes (Vygotsky, 1978). Future research might draw on these concepts to elaborate on the trajectory for developing data literacy skills and knowledge. Specific questions to consider include: What are the different components to the development of data literacy, and how does the process unfold over time? As a teacher works with a coach or within a data team, is the relationship ongoing with continuous development, or is there a point where the teacher reaches a certain skill level and no longer needs the CBI support?
Next, there is a set of unanswered questions around the artifacts for data use. For example, what are the culturally significant tools that mediate teachers’ data use and their interactions with CBI leads? Under what circumstances are external, technical tools appropriated by the teachers to become internal, psychological signs and symbols? Finally, sociocultural learning theory does not foreground questions of power, resistance, or conflict inherent in data use (Coburn and Turner, 2011; Henig, 2012; Knapp, 2008). At the micro-level, the distinction between mentor and novice inherently assumes a power dynamic that may exist between teachers or between a coach and teacher. Given that the current emphasis on data is intricately linked with high-stakes accountability systems in the USA and other countries, the role of CBIs within a school setting may have a political or power-based dimension that is critical to understand (Coburn and Woulfin, 2012).
Adopting a sociocultural learning approach provides a critical first step to understanding how school leaders and administrators can best support teachers’ use of data. This conceptual framework and future studies that draw on it for guidance can offer important insights into the practices of CBIs and practices for developing knowledge and skills for data use.
Footnotes
Acknowledgment
The authors gratefully acknowledge support for this research from the Spencer Foundation (grant number 201100106). We also greatly appreciate the cooperation of educators in our case schools and districts, as well as contributions from other members of our research team, including Alice Huguet, Melanie Bertrand, Jennifer McCombs, Beth Katz, and Brian McInnis.