Testing a Framework of Math Progress Indicators for ESSA: How Opportunity to Learn and Instructional Leadership Matter

Abstract

Purpose: To advance a framework of indicators that promotes implementation of math standards under the Every Student Succeeds Act (ESSA), we tested a conceptual model of the resources and processes within schools that influence the opportunity to learn mathematics (OTL) in the classroom using a recent administration of the 2011 Trends in International Math and Science Study (TIMSS 2011). There is potential benefit to state departments and system-level practitioners from more information about how schools might influence student access to college and career-ready curriculum. Such information has the potential to shape the kinds of indicators stakeholders put in place to target problems and formulate solutions for math learning. Research Design: To test the fit of our hypothesized model of math content and instruction OTL, we applied structural equation modeling techniques to school and classroom data from 425 fourth-grade U.S. math teachers in TIMSS 2011. Findings: We found a direct influence of instructional leadership on OTL math instruction in the classroom and teacher participation in math professional development. Content-specific resources indirectly influenced both OTL math instruction and content through teacher preparedness. Conclusions: The results of this study demonstrate a potential framework through which school leaders can support the preparedness of teachers in providing students with equitable access to coherent, focused, and rigorous math content.

Keywords

accountability policy math instruction math curriculum instructional leadership opportunity to learn teacher professional development Every Student Succeeds Act

In the years since the authorization of the No Child Left Behind (NCLB) Act, public, researcher, and policymaker concerns about the overreliance on standardized test scores as measures of student and school effectiveness have prompted efforts to build a framework of more robust measures of student learning as well as the school resources and processes that contribute to it (see, e.g., National Research Council [NRC], 2011; SRI International, 2014; U.S. Department of Education, 2016). While maintaining NCLB’s focus on student equity, the authorization of the Every Student Succeeds Act (ESSA) in late 2015 provided a minimum standard of “next-generation” accountability along two important dimensions: First, it allows states latitude to select from a broader set of indicators of school functioning and performance, including more rigorous learning and preparation for college and careers. Second, it removes the AYP (Adequate Yearly Progress) system of sanctions tied to underperformance, and instead allows states to set their own goals for performance based on these indicators (Darling-Hammond et al., 2016).

While the enactment of ESSA is recent, over the past several years, the NRC has been working to identify indicators of schools’ promotion of students’ understanding and development of inquiry and reasoning skills in STEM (science, technology, engineering, and mathematics; see NRC, 2011, 2013). For example, the NRC’s (2013) report identified 14 indicators of how to monitor progress toward successful K-12 STEM education, which are grouped into three overarching categories: (a) access to quality STEM learning, (b) educators’ capacity, and (c) policy and funding. Means, Mislevy, Smith, Peters, and Gerard (2015) highlighted the five NRC indicators most relevant to school success in math: (a) adoption of standards-aligned instructional materials, (b) classroom coverage of content and practices, (c) teachers’ content knowledge for teaching, (d) teachers’ participation in content-specific professional development, and (e) instructional leaders’ participation in professional development to support STEM learning (which includes learning climate, allocation of resources, teacher feedback, direction of content-specific instruction). Of these indicators, the NRC (2013) has placed a high priority on the investigation and use of indicators for aligned instructional materials, classroom coverage of math content and practices, as well as teachers’ math knowledge for teaching. Similarly, recent studies of the Common Core State Standards for Mathematics (CCSSM), which use large international data sets such as the Trends in Mathematics and Science Study (TIMSS), suggest that these standards are coherent and offer a more rigorous expectation of performance to help prepare students for college and careers beyond access to basic skills (Cobb & Jackson, 2011; Porter, McMaken, Hwang, & Yang, 2011; Schmidt & Houang, 2007).

While math performance remains a state reporting priority under ESSA, the measurement of resources and processes in schools that provide the conditions under which such performance improves would substantially extend practitioners’ understanding about what needs to change and why and when improvement is needed (Adams et al., 2017; Lach, 2016). One important indicator of math teaching and learning is student opportunity to learn (OTL), which has been used as both a research concept and a policy lever to assess student access to curriculum and instruction. Since the 1960s, with Carroll’s (1963) operationalization as time in and pacing of instruction, as well as content covered for achievement tests in the First International Mathematics Survey (FIMS) in 1964, student OTL has a long history of identifying the extent to which students have equitable access to learning (McDonnell, 1995; Schmidt & Maier, 2009). Most recently, scholars have expanded discussions around student OTL to include (a) access to more advanced courses or content and instruction with increased cognitive demand; (b) availability of resources for instruction, including teaching materials, as well as (c) measures of the training and expertise of teachers (Aguirre-Muñoz & Boscardin, 2008; Boykin & Noguera, 2011; Callahan, 2005; Cogan, Schmidt, & Wiley, 2001; Gamoran, 1987; Kurz, Elliott, Kettler, & Yel, 2014; Oakes, 1990). Not surprisingly, ESSA requires at least one indicator of “school quality,” or the resources or processes that influence student learning outcomes—many of these are subsumed under the broader umbrella of “opportunity to learn” (Darling-Hammond et al., 2016).

Where the literature remains sparse, however, is with respect to the role of leadership in supporting student OTL in mathematics. For the past 30 years in the field of educational administration, researchers have attempted to uncover a direct effect of instructional leadership on student achievement (see Robinson, Lloyd, & Rowe, 2008). Hallinger and Heck (1996) argued that this relationship was not direct, but largely indirect. Indeed, these findings of indirect relationships have demonstrated multiple paths, many of which flow through the work of teachers, as well as other school and social supports such as teacher professional community, teacher preparedness, and learning climate (Leithwood & Jantzi, 2008; Leithwood, Patten, & Jantzi, 2010; Sebastian & Allensworth, 2012; Seashore Louis, Dretzke, & Wahlstrom, 2010; Urick & Bowers, 2011, 2014). However, scholars still question the extent to which instructional leadership is proximal to student learning. In this study, we posit that it is instructional leadership that drives OTL via the paths of school supports, teacher preparedness, and delivery of content and instruction in the classroom (Bryk, Sebring, Allensworth, Easton, & Luppescu, 2010). To date, however, the direct effect of instructional leadership on student OTL has not been explicitly tested. Furthermore, leadership has been frequently tied to general instruction and teacher supports, but less often to content-specific initiatives. By addressing these gaps in the literature, we not only add further nuance to a long-standing debate within the field of educational administration but also stand to shed more light on the ways in which the school leader, through his or her actions, can influence a comprehensive set of indicators for teaching and learning.

Building on prior research on STEM indicators (Means et al., 2015) and math OTL content aligned with CCSSM (Schmidt & Houang, 2012), we used the U.S. sample of the 2011 TIMSS to test the paths through which school supports, teacher preparedness, and instructional leadership influence student OTL math content and instruction in the classroom. A purpose of this study was to provide state departments as well as system-level practitioners with more information about the potential resources and processes within schools that might influence student OTL in math as they begin to implement ESSA.¹ Such information has the potential to shape the kinds of indicators these stakeholders might put in place to target problems and formulate solutions with respect to math learning in their districts and schools. This study design also has the potential to inform the broader research literature on instructional leadership and to highlight possible policy levers for influencing important student outcomes in math. In what follows, we advance a conceptual framework of school resources and processes—teacher preparedness, schoolwide supports, and principal instructional leadership (PIL)—posited to influence math OTL in the classroom. Through a review of the relevant literature, we define each of these constructs, delineate the relationships among them, and explicate their purpose in our framework.

Conceptual Framework

Opportunity to Learn

While seemingly a straightforward concept, OTL has been conceptualized and operationalized in different ways in research since it was introduced in the 1960s. In his model of school learning, Carroll (1963) conceptualized OTL in terms of time, emphasizing the relationship between the time needed to learn a concept and the time spent engaged in learning. Around the same time, OTL was operationalized by the International Association for the Evaluation of Educational Achievement (IEA) for the FIMS. For FIMS, OTL served as a measure of content coverage to ensure validity when making comparisons in student achievement between international samples (McDonnell, 1995; Schmidt & Maier, 2009; Schmidt & McKnight, 2012). This construct was refined in later iterations of the study, the Second International Mathematics Survey (SIMS) and Third International Mathematics and Science Study (TIMSS), and time was also incorporated as a measure of OTL in TIMSS (McKnight et al., 1987; Robitaille & Garden, 1989; Schmidt, Cogan, Houang, & McKnight, 2011). For the Instructional Dimensions Study (IDS), Cooley and Leinhardt (1980) similarly used OTL measures that included how time was spent in the classroom and the overlap between the curriculum and the tests. In addition to time and content, scholars have responded to concerns about disparities within and across classrooms by expanding OTL measures to include school and classroom context variables, such as the availability of and student access to coursework and academic programs (Callahan, 2005; Cogan et al., 2001; Gamoran, 1987; Oakes, 1990), instructional quality and practices (Kurz et al., 2014; Smithson, Porter, & Blank, 1995; Stevens & Grymes, 1993; Wang, 1998), teacher expertise and experience (Aguirre-Muñoz & Boscardin, 2008; Goertz, 1994), and resources (Boykin & Noguera, 2011; Elliott, 1998; Herman & Klein, 1996; Kimura-Walsh, Yamamura, Griffin, & Allen, 2009; Oakes, 1990).

Despite expanded conceptualizations of OTL that have been introduced over the past several decades, content coverage has remained the most common, and often only, measure of OTL employed in research studies. Schmidt, Cogan, Houang, and McKnight (2011) have justified this more narrow, “seminal definition of OTL as content” by stating that “the provision of content is the fundamental rationale of schooling and the education system,” and “this is an aspect of schooling that both reflects education policy and is amenable to education policy reform” (p. 400; see also Schmidt, Cogan, & Mcknight, 2011). However, cognitive demand (Gamoran, Porter, Smithson, & White, 1997; Porter, 2002) is an important concept that complicates a straightforward interpretation of OTL as content by acknowledging the important role that teachers play in delivering content to students. Cognitive demand refers to the levels of skills or knowledge that students are expected to demonstrate for various topics, with higher categories of demand requiring students to “communicate understanding of concepts”; “solve nonroutine problems/make connections”; and “conjecture, generalize, prove” (Porter, 2002, p. 13). Porter (2002) has referred to these decisions—such as “how much time to allocate to a particular school subject, what topics to cover, when and in what order, to what standards of achievement, and to which students” (p. 3)—as the content of instruction. While content coverage has been the primary measure of OTL (e.g., Schmidt, Cogan, Houang, McKnight, 2011), scholars (Gamoran et al., 1997; Porter, 2002) have argued for the inclusion of the depth and rigor in which this content is taught as an extended measure of OTL.

The national focus on school improvement and accountability, coupled with concerns about equitable access for all students, have resulted in various policy efforts around OTL. Whereas early policy efforts to ensure equality of educational opportunity focused on inputs such as school finances and resources (Elmore & Fuhrman, 1995), the push for higher curriculum standards and improved student achievement outcomes shifted attention to school process indicators (Baratz-Snowden, 1993; McDonnell, 1995). Beginning in the 1980s, expanded indicator systems were proposed that “advocated including process measures such as teacher background and experience, school- and grade-level organization, course offerings and student course-taking patterns, curriculum content, instructional materials availability and usage, and instructional strategies” (McDonnell, 1995, p. 309). In the 1990s, the National Council on Education Standards and Testing (NCEST) incorporated school delivery standards in its Raising Standards for American Education report (Dougherty, 1996; McDonnell, 1995), which informed the framework for the Goals 2000: Educate America Act. This piece of legislation shifted the language to “OTL standards,” which were to be used to assess the “sufficiency or quality of the resources, practices, and conditions necessary at each level of the education system” (McDonnell, 1995, p. 312) in preparing students to meet content standards (Dougherty, 1996; Lucas, 1999; McDonnell, 1995; Porter, 1995). Most recently, the newly enacted ESSA (2015) requires states to include at least one OTL indicator—such as school climate, student engagement, or access to advanced coursework—in their state accountability systems (Darling-Hammond et al., 2016; ESSA Explained, 2015; Klein, 2016). In policy, then, OTL indicators tend to reflect a broader conceptualization of OTL than most scholars address through research studies. However, a commonality across research and policy efforts is the connection between OTL and achievement where OTL functions as a lever for improving standardized student achievement outcomes rather than as an important outcome itself.

These conflicting approaches to addressing OTL, including expanded measures and indicators of OTL, call attention to the importance of understanding how OTL is enacted in practice as a result of the hierarchical nature of educational policy and schools. Schmidt and McKnight (2012) have demonstrated that a trickle-down effect occurs at various levels of the education system, leading to differential learning opportunities for students. Thus, it is necessary to consider how student OTL is shaped by teachers, schoolwide supports, and school leadership.

Teacher Preparedness

The role of teachers as the “ultimate arbiters of what is taught [and how]” (Porter, 2002, p. 3) has been acknowledged in the OTL literature. Within schools, teachers can contribute to differential learning opportunities because of their individual choices around instructional strategies, learning materials, support for diverse learners, and sequencing and pacing of instruction, and these choices are shaped, in part, by their degree of preparedness in these areas of content and pedagogy. According to Ball (2000), there are three areas that define a math teacher’s preparedness to teach “what they need to know, how they have to know it, and helping them learn to use it” (p. 246). The NRC (2011) identified key elements of effective STEM instruction, including teachers with both content knowledge and pedagogical expertise specific to the content. This finding aligns with Bryk et al.’s (2010) framework for school improvement, which also addressed the importance of content-specific teacher capacity. Math teachers feel prepared through their knowledge of the content, confidence in how to instruct it, and opportunities to continue to learn how to use it (Ball, 2000). Additionally, the relationship of these issues around teacher preparedness to math OTL has implications for teacher preparation and ongoing in-service professional development (Means et al., 2015; NRC, 2011, 2013), a concern that is reinforced by scholars who have extended the concept of OTL to include teacher OTL (see Blömeke & Kaiser, 2012; Blömeke, Suhl, Kaiser, & Döhrmann, 2011; Schmidt, Cogan, & Houang, 2011; Schmidt et al., 2008; Wang & Tang, 2013; Wong, Boey, Lim-Teo, & Dindyal, 2012). By seeking to understand how teacher preparation programs influence teacher capacity, these studies not only acknowledge teachers’ proximity to student OTL through delivery of content and instructional practices but also contribute to a more nuanced view of teacher capacity that distinguishes between content knowledge, content-specific pedagogy, and general pedagogy.

Schoolwide Supports

Whereas teacher preparedness informs their curricular and pedagogical decisions within a classroom, teachers’ interactions with each other from within broader school structures can either support or constrain classroom teaching (Li, Hallinger, & Ko, 2016). In the school improvement literature, school structures such as teacher professional community, learning climate, and content-specific resources help create a system of capacity around teachers, which promotes their preparedness to use instruction and deliver content (Bryk et al., 2010; Li et al., 2016). Because OTL measures tend to emphasize instructional content and processes that students experience within classrooms, this sphere of OTL indicators at the school level is less developed in the OTL literature.

One of the most salient of the schoolwide supports is teacher professional community. As Newman et al. (1996) point out, a strong professional community, when focused on the technical core, has the potential to greatly enhance the individual preparedness of the teachers who comprise it. While the literature on these communities is vast, beneficial professional communities share some common features: shared values and vision, collective responsibility for student learning, reflective inquiry, and collaboration (Stoll, Bolam, McMahon, Wallace, & Thomas, 2006). As features of a teacher social network, these characteristics of strong professional community combine to generate social capital that can then be utilized in the building of teacher human capital (Coleman, 1988; Frank, Zhao, & Borman, 2004; Gamoran, Gunter, & Williams, 2005; Marks & Louis, 1999; Supovitz, Sirinides, & May, 2009; York-Barr & Duke, 2004). Effective instructional leadership also involves providing teachers with constructive feedback and opportunities for reflection (Blase & Blase, 1999).

Learning climate is another important dimension of school improvement frameworks (Bryk et al., 2010) addressed in the NRC (2011) report as a common element to promote learning gains in math. This dimension involves teachers’ expectations for student achievement, including academic press and learning norms (Bryk et al., 2010). The elements of learning climate demonstrate how schoolwide goals and expectations for learning are developed and shared to influence teachers’ understanding and implementation in their individual classrooms (Hallinger, Bickman, & Davis, 1996; Hoy & Hannum, 1997; Hoy, Tarter, & Hoy, 2006; Urick & Bowers, 2011, 2014).

A third schoolwide support involves availability and access to content-specific resources. Resources have been conceptualized as, a direct measure of OTL, variation of access to resources across students, or a support condition, which are school instructional materials available to teachers. Resources commonly refer to the materials, tools, and facilities that promote teaching and learning. Textbooks, for example, are primary resources related to content, sequencing, and pacing of instruction (Schmidt & McKnight, 2012). Similarly, materials related to standards, curriculum, and assessment bear on the focus, coherence, and rigor of content and instruction enacted in schools and classrooms (see Schmidt & Houang, 2012; Schmidt, Wang, & McKnight, 2005). The NRC (2011) has also documented that disparities in access to “adequate” facilities, resources, and supplies exacerbate achievement gaps in science and math for underrepresented groups. Finally, resources can also refer to aspects of teacher quality of that address content-specific needs, such as hiring of “in-field” teachers for math, or teachers who specialize in math content and pedagogy (Darling-Hammond, 1996; O’Day & Smith, 1993). Students with in-field math teachers are more likely to be affluent and to have higher math achievement when compared with out-of-field math teachers (Goldhaber & Brewer, 1996; Greenberg, Rhodes, Ye, & Stancavage, 2004).

In sum, a schoolwide system of teacher professional community to expand teacher knowledge, a learning climate with rigorous expectations, and material and human resources as content-specific tools and teacher qualifications represent the broader capacity within schools, which explains the extent to which teachers are prepared to deliver content (Bryk et al., 2010; Li et al., 2016). These schoolwide supports provide an extended set of indicators to measure the degree at which a school can successfully support math teaching and learning.

Principal Instructional Leadership

As the “driver for change” (Bryk et al., 2010, p. 69; NRC, 2011, p. 24), school leaders have an important role in the promotion of OTL. The relationship between instructional leadership and OTL has been neglected in the OTL literature, although research on school improvement suggests that this is a promising line of inquiry (e.g., Cuban, 1984; Edmonds, 1979; Hallinger, 2005; Hallinger & Heck, 1996, 1998; Hallinger & Murphy, 1985). School improvement scholars agree that “principals exercise a measurable, though indirect effect on school effectiveness and student achievement” (Hallinger & Heck, 1998, p. 186; see also Hallinger & Heck, 1996). While various types of leadership have been explored, Robinson et al. (2008) concluded that the effect of instructional leadership on student outcomes was three to four times greater than transformational leadership. These findings are also supported by other studies that demonstrate the importance of targeted efforts around the technical core of teaching and learning (Marks & Louis, 1999; see also Hallinger & Heck, 1996; Marks & Printy, 2003).

To explain how this focus on teaching and learning translates to improved student outcomes, Supovitz et al. (2009) have noted that by influencing changes in instructional practice, educational leaders have a significant effect on student learning. School leaders help develop and maintain a school climate conducive to improvement, collaboration, professional growth, and a schoolwide focus on teaching and learning (Blase & Blase, 1999; Firestone & Wilson, 1985; Hoy & Hannum, 1997; Leithwood & Mascall, 2008; Marks & Louis, 1999; Schmidt & McKnight, 2012; Urick, 2016). One way that school leaders accomplish this is by setting direction through the development, implementation, and monitoring of the school’s vision, mission, and goals that characterize direct instructional leadership (Bendikson, Robinson, & Hattie, 2012; Hallinger & Heck, 1998, 2002). Effective instructional leadership also involves providing teachers with constructive feedback and opportunities for reflection (Blase & Blase, 1999).

These findings suggest that principals have the potential to influence student OTL through their work with teachers. School leaders shape supporting conditions in the school related to instruction, including learning climate (Hallinger et al., 1996; Hoy & Hannum, 1997; Hoy et al., 2006; Urick & Bowers, 2011, 2014), teacher professional community (Blase & Blase, 1999; Frank et al., 2004; Hallinger & Heck, 1998; Louis & Marks, 1998; Marks & Louis, 1999; Supovitz et al., 2009; York-Barr & Duke, 2004), and content-specific resources (Darling-Hammond, 1996; Hoy & Hannum, 1997; O’Day & Smith, 1993; Spillane, Hallett, & Diamond, 2003). In turn, these schoolwide supports influence individual teacher preparedness (Leithwood & Mascall, 2008; Marks & Louis, 1999; Marks & Printy, 2003).

Collectively, as seen in Figure 1, these conditions are expected to influence student math OTL. In the hypothesized model, we imply that PIL is central to the conditions that directly and indirectly influence student math OTL. Finding this to be the case, these results would demonstrate that principals are instrumental in implementing curricular and instructional policy changes in math. Furthermore, if principals have a direct effect on students’ OTL in mathematics, then efforts to increase OTL through instructional leadership could create more equitable access to math programs. Finally, a main purpose of this study is to test the extent of the relationships in this hypothesized framework to help create a more comprehensive set of math OTL indicators.

Method

Data Sources

This study employs secondary data analysis of U.S. mathematics teachers within the Grade 4 population of the TIMSS 2011. The primary goal of TIMSS 2011 was to provide comparative information across countries for the improvement of teaching and learning (Foy, Arora, & Stanco, 2013). Because a wide range of information on teaching, leading, and learning was collected, TIMSS 2011 was ideal for an examination of the connections between school resources, teacher preparation, school leadership, and math OTL of interest in this study. Furthermore, as the most recent administration, TIMSS 2011 provided more extensive leadership measures in the area of instruction as well as more measures capturing the work environment of teachers than prior administrations.

TIMSS 2011 utilized a two-staged procedure to ensure a nationally representative sample for each participating country. First, schools in each participating country were randomly selected, then whole intact classrooms within each school were randomly selected from the target grade level. In the majority of cases, this sampling procedure resulted in one to two randomly selected classrooms per school (and thus one to two teachers), for a target of 150 schools and 4,000 students per country (Joncas & Foy, 2013). The analysis presented here focused on only the teacher/school level of TIMSS. A final sample of N = 425 mathematics teachers were included in the models. Some missingness was present due to the merging of school and teacher data, and this resulted in some listwise deletion of cases. To address concerns about the effects of these lost cases on preserving the representativeness of the sample with respect to the study variables, an analysis was conducted. A series of t tests between the study variables in Table 1 for the full TIMSS sample and our analytical sample revealed no statistically significant differences. All three structural equation models (SEMs) included the teacher weight, MATWGT, to appropriately adjust for sampling techniques.

Table 1.

Descriptives for Context Variables and Composites Included in Analyses.

	N	Minimum	Maximum	Mean	SD
School variables
School affluence (>50%)	425	0.00	1.00	0.16	0.36
School readiness	425	0.00	3.00	1.33	0.94
School size: 400-750 students	425	0.00	1.00	0.50	0.50
School size: 750+ students	425	0.00	1.00	0.16	0.36
Urbanicity: Urban school	425	0.00	1.00	0.20	0.40
Urbanicity: Medium city	425	0.00	1.00	0.21	0.40
Urbanicity: Town/village/rural	425	0.00	1.00	0.28	0.45
Teacher-reported measures
OTL instruction	425	0.00	2.00	1.15	0.75
OTL content	425	0.00	20.00	15.39	3.45
Teacher preparedness: Content	425	0.00	5.00	3.96	1.49
Teacher preparedness: Instruction	425	0.00	3.00	2.29	0.95
Teacher preparedness: Math PD	425	0.00	4.00	2.41	1.53
Teacher professional community	425	0.00	3.00	1.69	0.80
Learning climate	425	0.33	4.00	3.37	0.56
Leader-reported measures
Math resources	425	0.00	3.00	1.95	0.84
Instructional leadership: Goals	425	0.00	4.00	2.75	1.34
Instructional leadership: Teacher feedback	425	0.00	2.00	0.76	0.87

Note. Sample size and descriptives presented here were for Model 3. Models 1 and 2 only fluctuated a few cases (423 and 428, respectively). OTL = opportunity to learn; PD = professional development.

Measures and Instrumentation

To construct the needed measures for the study, we utilized school and teacher questionnaires. The school questionnaire is addressed to school leaders and includes information on school characteristics, climate, resources, as well as the leadership activities of the principal. The teacher questionnaire is addressed to teachers of intact classrooms of fourth-grade students randomly selected for participation. It collects information on teachers’ backgrounds, classroom resources, instructional knowledge and practices, and attitudes toward teaching and colleagues. Teachers are instructed to respond to these questions with reference to the specific TIMSS class selected for the study, even though they might teach other classes within the same school (Foy et al., 2013). To construct the measures referred to in more detail below, we employed a combination of theory and factor analysis in examining clusters of items on each of the school and teacher questionnaires. For a detailed list of the measures and the items that comprise them, please see Appendix A. Table 1 displays the descriptive statistics for the raw study measures discussed below.

Teacher-reported measures

Opportunity to learn: Math content (α = .66)

This outcome variable was constructed using research on the sequence of fourth-grade math content as it relates to CCSSM and past TIMSS math curriculum studies (see Schmidt & Houang, 2012). Ten items covering content aligned with fourth-grade national and international standards were measured on a categorical scale of not yet taught (0), mostly taught before this year (1), mostly taught this year (2). The largest value was assigned to the content taught this year based on past curriculum studies that identified coherent and focused math topics for fourth grade (see Appendix B; only bolded math topics are included in the measure). A sum of the responses for these items was recorded for each teacher.

Opportunity to learn: Math instruction (α = .56)

This outcome variable captured the degree of “high cognitive demand” (see Porter, 2002) for math instructional strategies. A total of eight items covering various math instructional strategies are included in the survey and range in their cognitive demand. For example, listening to teacher explain how to solve problems, or memorizing rules, procedures, or facts would be examples of a low cognitive demand instructional strategies. From this list of eight, two items were selected: asking students to explain their answers and relating math to everyday life (see Appendix C for justification). These were recoded to daily (1) and half lessons/some lessons/never (0) due to limited variation across this scale. A sum of the responses for these items was recorded for each teacher.

Teacher preparedness: Prepared—math content (α = .80)

This series of items on the TIMSS teacher questionnaire asked teachers how comfortable they feel (from very well prepared to not well prepared) in teaching various topics in math such as decimal concepts and number sentences as well as shapes and other geometric concepts. These items reflect teacher exposure to math content, which has been found to influence student learning in the classroom (Schmidt, Burroughs, Cogan, & Houang, 2017; Schmidt et al., 2001; Schmidt & Maier, 2009). Ten items from this list were selected for their alignment with CCSSM standards for Grade 4 (same topics at OTL content, see Appendix B). These items were dichotomized as either well prepared (1) or not/somewhat prepared or not applicable (0). A sum of these items was taken to create a composite. Six categories were then created to consolidate these sums for analysis because of the nature of the distribution: 0-5 content areas well prepared and 6, 7, 8, 9, 10 content areas well prepared.

Teacher preparedness: Confident—math instruction (α = .66)

On the TIMSS teacher questionnaire, the three of five items selected asked teachers how confident they feel about employing content-specific, high cognitive demand (Porter, 2002), instruction: showing students a variety of problem-solving strategies, helping students appreciate the value of math, and provide more challenging tasks for advanced students. These items were dichotomized as either very confident (1) or not confident (0), and then a sum of the three items was taken.

Teacher preparedness: Participated math professional development (α = .79)

On the TIMSS teacher questionnaire, six items asked teachers about their participation in content-specific professional development for mathematics over the past 2 years for content, pedagogy, curriculum, and assessment and/or technology integration. Content-specific professional development related to both content and instruction have been linked to student learning in the classroom (Desimone, Smith, & Phillips, 2013). These items were dichotomous; teachers answered either yes (1) or no (0). A sum of five of the six items in this measure was taken to capture the amount of math professional development the TIMSS teacher has received in the past 2 years.

Schoolwide supports: Teacher professional community (α = .87)

Teacher professional communities, described as a norm of collaboration and sharing around teaching topics, materials, and experiences, have been found to influence a teacher’s preparedness to influence student learning in the classroom (see Stoll et al., 2006). This measure consisted of four of five items on the teacher questionnaire, which ask teachers about the nature and frequency of their interactions with colleagues from almost never (0), 2 to 3 times per month (1), 1 to 3 times per week (2), to daily or almost daily (3). An average of their responses for four of five of these items was used as a composite. An item about visiting other classrooms was omitted due to lack of variability in the responses.

Schoolwide supports: Learning climate (α = .80)

Perceptions of the learning climate, specific to schoolwide academic goals and expectations, have been found to influence the work of teachers (Bryk et al., 2010) and math learning in the classroom (Urick & Bowers, 2014). On the TIMSS teacher questionnaire, three items elicit the perceptions of teachers with respect to the learning climate of the school, in particular the degree of teachers’ academic press and perceptions of school goals or norms. These items were on a 5-point scale from very high (4) to very low (0). An average of these items was computed as a composite for each teacher.

Leader-reported measures

Schoolwide supports: Math resources (α = .89)

Math resources, both curriculum-aligned instructional materials and teachers with content-knowledge qualifications, influence the work of teachers and what is ultimately delivered in the classroom (Darling-Hammond, 1996; Schmidt & Houang, 2012). This measure was constructed from items on the school questionnaire that asked leaders to report on the degree to which lack of resources affects quality content-specific, math instruction: math specialist teachers, computer software, library materials, and audio-visual materials. These items were reverse-coded from a lot (0) [low resources, lack affects instruction] to not at all (3) [high resources, lack does not affect instruction]. An average of the four items was taken as a composite.

Principal instructional leadership: Goals (α = .69)

PIL is measured as a direct focus on improving teaching (see Bendikson et al., 2012), which includes items about their direction setting of curricular goals (Hallinger & Heck, 1998, 2002). This measure was constructed from items on the school questionnaire that asked leaders to report on how much time in the past year they had spent on leadership activities toward developing, promoting, and monitoring their school’s educational vision and/or goals. The original item 3-point scale ranged from no time to a lot of time, but these were collapsed into categories of no/sometime (0) to a lot of time (1) due to low variation. A sum of these dichotomous items was then calculated and used in the analysis.

Principal instructional leadership: Teacher classroom feedback (α = .80)

The second direct instructional leadership (Bendikson et al., 2012) construct includes items that measure the frequency of feedback to teachers regarding their practice (see Blase & Blase, 1999). This measure from the school questionnaire asked leaders to report on the degree to which leaders advised teachers and/or initiated discussion who are experiencing problems with their teaching/classroom. Similar to the prior measure of PIL, item responses were collapsed into a dichotomous category, and then summed for the purposes of the analysis.

School measures

Several characteristics of school context were used as controls in the analysis from the school questionnaire: school size, urbanicity, and percent of economic affluent students (e.g., Opdenakker & Van Damme, 2007). These were each recoded as dummy variables (refer to Table 1). School readiness was composed of eight items from the school questionnaire that elicited the percentage of students in the school who came to elementary prepared with literacy and numeracy knowledge. These items were averaged into a composite, then four categories were created due to modal distributions around the original ranges, 0, 1, 2, and 3.

Analytical Approach

The purpose of the analysis was to systematically parse out and test the hypothesized relationships between the constructs that comprised the framework of progress indicators in schools. Thus, we organized our analysis and reporting of the results in three stepwise SEMs. In Model 1, our goal was to more closely examine the direct effects of teacher preparedness on math OTL. Model 2 examined the addition of schoolwide supports on teacher preparedness and math OTL. The final phase, Model 3, tested a full analysis of paths for math OTL, which included PIL as a potential moderator of schoolwide and teacher variables, but also as a direct predictor of math OTL.

To test the fit of our hypothesized model of math content and instruction OTL to the TIMSS fourth-grade mathematics data, we applied a general SEM approach via Mplus, version 7.4. All three phases of the modeling process were run as aggregate models (i.e., single school/teacher level; all composites, no item-level variables) with the MATWGT math teacher weighting variable applied. For all models, the weighted least squares estimator (WLSMV) was used because of both categorical and continuous dependent variables, but in Phase 3 parameterization switched from delta to theta, given that a categorical dependent variable (OTL Instruction) influenced and was influenced by another observed dependent variable (OTL Content).

Model-fit indices and parameter estimates were analyzed throughout model specification to determine alignment to our hypothesized specification. Several model-fit indices were used to test the fit between our hypothesized model and the sample data and estimated model covariance matrix. Using the recommendations of Schreiber, Nora, Stage, Barlow, and King (2006), we used the root mean squared error of approximation (RMSEA) for model parsimony, and the comparative fit index (CFI) for comparative fit. Widely accepted “good fit” thresholds for RMSEA of less than .05 and for CFI of greater than .900 to .950 were used as criteria for analyzing fit (Hu & Bentler, 1999; Kline, 2015). Because a weighted least squares estimator was used, chi-square estimates can be compared across models, but the significance tests are not interpreted. All results reported below and in the subsequent tables are the standardized estimates. Finally, using a method outlined in MacCallum, Browne, and Sugawara (1996), a power analysis was conducted on each of our three hypothesized models to estimate, based on the degrees of freedom and sample size, the statistical power available for testing the fit of our models to the underlying covariance structure of the data (i.e., RMSEA). The results of this analysis revealed that the statistical power of each ranged from .95 in Model 1 to .99 for Model 3, with 45 degrees of freedom and 105 degrees of freedom, respectively.

Results

Model 1: Influence of Teacher Preparedness on Math Opportunity to Learn

The multivariate regression model presented in Figure 2 displays the relationships of the teacher preparedness composite variables to OTL math content and instruction while controlling for school context. In the instances where the relationships between aspects of school context (i.e., school size, urbanicity, school poverty, and readiness) and teacher preparedness and math OTL were statistically significant, their standardized coefficients are displayed at the bottom of Figure 2. School affluence and urban school locale had a negative relationship with teachers’ preparedness to teach the math content (β = −.213, p < .01; β = −.132, p < .10). Midsize and large school enrollment (β = .205, p < .001; β = .244, p < .001), and all three urbanicity classifications had positive relationships with OTL math content (β = .126, p < .10 for urban; β = .168, p < .05 for medium/large; β = .116, p < .10 for small), while only midsize enrollment had a significant positive relationship with OTL math instruction (β = .178, p < .05). As will be seen in the following figures, this precise pattern of relationships between school context variables, teacher preparedness and math OTL in Model 1 held throughout all analysis phases, with only slight changes in the standardized estimates.

Returning to the focus of Model 1, which was the analysis of the relationships between teacher preparedness variables and math OTL in content and high cognitive demand instruction, Figure 2 reveals significant relationships between teachers’ confidence in math instruction (β = .290, p < .001) and participation in math professional development (β = .171, p < .05) and the use of math instructional strategies with high student cognitive demand (OTL math instruction). Furthermore, and somewhat not surprisingly, the degree to which teachers feel prepared in the various areas of math content predicted the degree to which they reported that these content areas are present in their everyday practice (β = .328, p < .001).

Model 2: Influence of Schoolwide Supports and Teacher Preparedness on Math Opportunity to Learn

The second model in our analysis added the schoolwide support measures (learning climate, teacher professional community, and math resources) to teacher preparedness and math OTL. The SEM path model (Figure 3) displays the relationship of these schoolwide supports and teacher preparedness on the math OTL outcomes (instruction and content), while controlling for school context. The number of school context variables was reduced for parsimony based on the initial multivariate regression model (Figure 2). Several model statistics were used to examine the fit between our hypothesized model and the sample data and estimated model covariance matrix. Model 2 fit statistics indicated a generally acceptable model fit (χ² = 226.079, CFI = .922, RMSEA = .025).

The results shown in Figure 3 reveal the same underlying relationships between teacher preparedness and math OTL instruction and content as reported in Model 1. For example, an increase in teachers’ perceived preparedness to teach math content with coherence has a positive relationship with whether or not these math topics were covered in the classroom (β = .314, p < .001). Furthermore, an increase in teachers’ confidence in their ability to use math instructional strategies with high student cognitive demand is associated with an increase in the actual use of these strategies in their daily mathematics instruction (β = .322, p < .001). Where statistically significant, the schoolwide support variables added in Model 2 exhibited positive relationships to teachers’ perceived content preparedness, confidence in math instruction, and participation in math professional development. For example, an increase in school learning climate, as measured by teachers’ understanding and implementation of curricular goals and high student learning expectations, has a positive relationship to teacher confidence in math instruction (β = .206, p < .001). Furthermore, an increase in access to school-specific math resources, such as computer, library, and audio-visual materials for instruction and teachers specializing in math, has a positive association with the degree to which teachers reported feeling confident in the use of math instructional strategies with high student cognitive demand (β = .175, p < .01).

Model 3: Influence of Principal Instructional Leadership on Math Opportunity to Learn

The final model of our analysis included PIL as a potential moderating variable of schoolwide supports and teacher preparedness as well as a direct predictor of math OTL content and instruction (Figure 4). All possible relationships between instructional leadership, schoolwide supports, teacher preparedness, and OTL were tested, then nonsignificant paths were reduced for parsimony. A few nonsignificant correlates or “with” paths remained due to the nature of the significant direct and indirect paths tested in the final model—see grey lines with two-headed arrows between teacher professional community and math resources as well as educational goals (instructional leadership) and each schoolwide support. Model statistics (χ² = 261.794; CFI = .894, RMSEA = .023) suggest a reasonable fit.

PIL, for the purposes of our analysis, was composed of two composite variables—teacher classroom feedback and educational goals. As observed in Model 2, schoolwide supports were found to be predictive of teacher preparedness as measured by confidence and preparation in math content and instruction. As is evidenced in Figure 4, these key relationships held in Model 3. On including the PIL measure in Model 3, we observed a strong direct effect of PIL on OTL math instruction, and a smaller effect on teacher math PD participation. In other words, as principals increase their focus on developing, promoting, and monitoring the achievement of schoolwide vision and goals, there is a corresponding increase in the frequency with which teachers use high cognitive demand instructional strategies in math (β = .281, p < .001). Similarly, this increased focus also had a direct effect on teacher participation in math-related professional development (β = .155, p < .05). This finding is also notable because of the mediating role that math-related professional development participation plays between schoolwide supports such as teacher professional community (β = .247, p < .001) and access to school-specific resources (β = .167, p < .05) and math OTL instruction. The key ways in which school supports, PIL, and teacher preparedness lead to opportunities to learn in mathematics are explained in the discussion.

In addition to the main paths in this final model, several context variables were significant covariates on the dependent variables included in the analysis. First, schools with more than 50% affluent students were associated with teachers who reported feeling less prepared to teach math content. Interestingly, an urban location was also a negative predictor of teacher preparedness of content, but had a positive influence on OTL math content. Urban schools and affluence of the student population were significant covariates along with content-specific resources on teacher preparedness of content. In other words, teachers feel more prepared to deliver content in suburban, compared with urban, locations, and to communities with less than a majority of affluent students. The availability of math resources may moderate these teachers’ feelings of preparedness within these settings. Future research should further disaggregate categories of affluence, or family income, into high and low extremes to compare teacher feelings of preparedness. Based on past literature on low-income schools (e.g., Halvorsen, Lee, & Andrade, 2009) and these results on a majority affluent student population, both contexts might have inverse relationships with teachers feeling prepared. Second, compared with suburban locations, such as urban areas, schools in medium and small towns had a positive relationship with teacher delivery (OTL) of math content. With small enrollment as a reference, or less than 400 students, midsize or large enrollment had a positive effect on teacher confidence in math instruction, and OTL math instruction and content. In sum, in 2011, at the start of implementation of Common Core State Standards in the United States, the size and location of a school, compared with small enrollment and suburban schools, were positively associated with access to math content.

Discussion

With the shift to ESSA from NCLB—which includes the use of college and career readiness standards, CCSSM, as well as the tracking of multiple measures of school and student success beyond test scores—it is important to understand how to implement changes in curriculum and instruction, but also what school practices might indicate progress toward increased quality and equity in the delivery of content. The results of this study provide a potential framework through which school leaders can support the preparedness of teachers to ensure equitable student access to coherent, focused, and rigorous math content in the classroom. While the indirect effects of school leadership on student achievement have been studied in the school improvement literature (see Hendriks & Scheerens, 2013), our study focused instead on the effects of school supports on a more proximal measure of student learning: student OTL. Based on the final Model 3 (Figure 4), we found three distinct paths from instructional leadership and schoolwide supports to OTL. We will discuss each of these three paths re-represented in Figures 5 to 7.

Foremost, we found a direct relationship between the amount of time a principal spends on instructional leadership of educational goals and OTL math instruction in the classroom (see Figure 5). This is a nuanced and important finding. Decades of research has sought a direct effect from leadership on student achievement. Several authors (e.g., Heck & Hallinger, 1996) have advised to stop searching for leadership direct effects on student achievement. With the shift in policy from summative student achievement measures to OTL indicators that show school and student progress, this finding directly connects the principal to classroom processes through which students learn. Figure 5 demonstrates a direct influence from educational goals on OTL, an indirect relationship through professional development, but no indirect influences from these goals through schoolwide supports or the collaboration of teachers. Using data over 3 years from different contexts in England, Day, Gu, and Sammons (2016) explain that, in the improvement process, schools need to develop foundational goals first, which influence instruction and professional development of teachers, before initiatives are distributed to other leaders or groups of teachers. These results could show that this particular path between goals, professional development, and OTL represents this foundational goal phase, which Day and colleagues explain as a clear focus from the principal on raising expectations for students throughout training, redesign of teams, instruction, and overall environment. Furthermore, as predictors in Figure 5, teacher professional community and resources may represent the degree in which these goals have been developed and, thus, influence the extent to which teachers need professional development and direct principal guidance of goals to meet “high cognitive demand” expectations in the classroom (see Day et al., 2016). However, the direct path from goals in instructional leadership to professional development and OTL instruction provides evidence that principals have an effect on student access to “high cognitive demand” instruction in the classroom, which supports college and career readiness goals in STEM.

This direct effect of instructional leadership on OTL supports arguments for an increased priority placed on principal training in content-specific instruction (see Means et al., 2015; NRC, 2013) for better leadership of educational goals, teacher classroom feedback, delivery of instruction, and teacher professional development. The path reinforces other conclusions on content-specific, math, instructional leadership in which administrators have been found to guide instruction anchored in their past teaching experiences rather than content understanding (Lochmiller, 2016; Lochmiller & Acker-Hocevar, 2016).

Second, we found a separate path to OTL math instruction through teachers’ confidence in instruction (see Figure 6), which was not linked to teachers’ participation in professional development as discussed in the first main path. Both learning climate and content-specific resources directly influenced teachers’ confidence in instruction, which predicts OTL. Furthermore, learning climate and resources have a possible increased effect together, compared with individual results, since they are related. While our analysis did not show it, given findings in past literature, a principal has the ability to influence instruction through the learning climate as well as available resources (e.g., Blase & Blase, 2000; Hallinger, 2005; Heck, Larsen, & Marcoulides, 1990; Marks & Printy, 2003; Urick & Bowers, 2011, 2014). As discussed earlier, Day et al. (2016) conclude that a developed learning climate around established initiatives and the availability of supports, such as resources, is evidence of increased progress toward improvement in student outcomes, compared with a school in which a principal is just beginning to direct goals. Furthermore, previous research has grouped measures of available instructional resources and learning climate into broader conceptualizations of “successful” leadership (e.g., Leithwood, Day, Sammons, Harris, & Hopkins, 2006), whereas this study seeks to separate them out from the direct practices of the principal. Bendikson et al. (2012) found that principals in schools that were improving in performance displayed direct leadership practices. These principals were able to respond appropriately to the context of their school with these direct practices. Future research can continue to better understand this overlap and separation of principal practices and broader school supports as conceptualizations of instructional leadership, and their influence on OTL in the classroom.

Finally, while we did not find a direct relationship from instructional leadership to OTL math content, resources were indirectly related to OTL math content through teachers’ preparedness to teach content (see Figure 7). Again, learning climate is related to these available content-specific resources. In original explanations of distributed leadership (Spillane, 2006), routines and tools, such as norms in a learning climate and available content-specific resources, were a means through which leaders interacted with followers. These routines and tools were a way in which principals could offer support to teachers or to include teachers in the leadership of teaching and learning. Our findings reinforce this idea with a direct influence on teacher preparedness and indirect relationship with the delivery of sequenced and focused math content in the classroom. As mentioned earlier and given the past literature, future research should more closely examine the relationship between leadership, learning climate, content-specific resources, and teacher preparedness to teach math content. These relationships determine the focus and coherence of content that is delivered within the classroom. Policy shifts in standards and curriculum are implemented and supported through this final path. While state curriculum standards are largely dictated, there remains variation in how they are delivered, to whom they are delivered, and the rate of implementation when changed. The study of student OTL helps us understand how to more equitably monitor shifts in academic programs and to whom and to what extent they are available.

One indicator that consistently predicted teacher preparedness to deliver content and instruction in our models was available content-specific resources. The NRC (2013) has argued for the importance of evidence-based, standard-aligned instructional materials for STEM. The findings reported in this study support the argument that these materials promote the delivery of content in the classroom. Teachers and principals should understand these curriculum goals and expectations so that instructional materials can be made available and used to prepare teachers for content delivery. More important, this demonstrates a great need for increased school funding so that system leaders can hire specialized content teachers as well as purchase necessary materials aligned with new college and career readiness curriculum and supply teachers with training to incorporate it into the classroom.

The overall purpose of this study was to connect instructional leadership to OTL. These findings extend current literature because they provide a comprehensive framework for how principals might direct school supports, and teacher preparedness, to provide students with greater access to coherent math content and rigorous instruction in their classrooms. First, this study argues for the importance of OTL as both a research concept and a policy lever. Our OTL measure includes purposefully selected mathematics topics that represent coherent, sequenced, and focused content, as well as high cognitive demand instruction that asks students to explain answers and relate to daily life. The intention behind measures of OTL is to assess the degree to which students have equitable access to more rigorous standards and curriculum. This approach differs from the use of achievement scores as an outcome. Achievement scores are often measures of basic skills and are not necessarily as representative of school and classroom learning processes and structures. Second, just as discussions around OTL are missing from educational administration literature, school leadership and improvement findings are often not linked to discussions around equitable student access to curriculum, standards-based reform, or accountability in policy. These findings demonstrate the way in which a system of school structures leads to a teacher’s delivery of select math content and instruction in the classroom.

Limitations of the Study

One important limitation to the findings presented here is that our model tested one-way relationships to math OTL, and, as such, these relationships cannot be interpreted as causal in nature. There are plausible paths of influence that remain unexplored in this article with respect to math OTL. In particular, future research should investigate the relationship between teacher feedback and educational goals on teacher professional development and how this might inform content-specific resources and the professional community of teachers. Furthermore, the findings here were constrained, to some degree, by the nature of the sampling procedure and nesting of data within the TIMSS 2011 study. Because the TIMSS 2011 randomly selected only one to two classrooms (and thus teachers) to participate, the school and teacher/classroom levels were not able to be partitioned into separate levels in the analysis (i.e., a multilevel model). To more fully model variation in math OTL and understand its relationships to school-level supports and teacher preparation in math, future studies should attempt to leverage nested data on these characteristics of teachers and schools.

One final limitation of note pertains to our measure of OTL, which was a teacher’s perception of instructional strategies used in the classroom rather than a student’s perception. The rationale behind this choice was twofold. First, we held reservations about the validity and reliability of fourth-grade students’ perceptions of classroom teaching. Second, because schoolwide supports and teacher preparedness were predominately measured via teacher perceptions, it made more sense to align the measurement of our outcomes to the teachers as a unit of analysis. While it might be argued that we could have used the eighth-grade TIMSS sample instead to overcome the first of these two issues, we felt that the need to measure OTL before a student progresses too far in an academic track superseded our desire to use student perceptions of classroom practice over those of teachers. In contrast, other studies, such as the 15-year-old student sample from the Programme for International Student Assessment (PISA), offer student perceptions of math content familiarity as a measure of student OTL (see Schmidt, Zoido, & Cogan, 2013). Future work might utilize these data to investigate patterns in OTL from the standpoint of the individual student’s experience.

Implications for Policy and Practice

For policymakers embarking on changes to state accountability systems in anticipation of ESSA, our findings beg the careful consideration of OTL (as measured by access to rigorous content and instruction in mathematics) as a central indicator of school quality related to math learning. Of course, math learning outcomes, such as those measured by performance on state assessments, will likely continue to be a priority for states, our findings emphasize the role of resources and processes in affecting these long-term outcomes. In particular, content-specific resources are within the purview of states to control through strategic investments in access to certified and “in-field” math teachers as well as access to technology, media, and other curricular materials, while ensuring that such access is equitably distributed across schools. Furthermore, school leaders need to be provided with professional development for instructional leadership, which is focused on, among other things, fostering content-specific leadership toward greater OTL. Finally, while ESSA requires only one indicator of school quality, we would also encourage states to consider tracking multiple indicators of school quality, including math OTL, with the understanding that no single indicator is a panacea for understanding and/or improving patterns of student learning within a school.

Regardless of what states decide to require with respect to the measure of school quality under ESSA, we recommend that principals and superintendents consider tracking OTL in math in their respective schools or districts. Our findings suggest that tracking and manipulating this proximal outcome as needed through indirect and direct instructional leadership action increases the likelihood of affecting important accountability outcomes such as student achievement. Thus, while improved student achievement might be the long-term goal for leaders, changing this for the better requires a focus not only on these outcomes but also on the resources and processes within schools and districts that produce them. It is in focusing on resources and processes that school leaders can begin to understand why outcomes are stagnant and how or what must change for them to improve (Adams et al., 2017).

Conclusion

The importance of OTL as an indicator of access to high-quality math content and instruction cannot be overemphasized. As various stakeholders track student achievement results for rigorous content standards such as the CCSSM, the assumption is often that students have had access to ideal CCSSM content and methods from the start. This is a very dangerous assumption and can lead stakeholders to draw the wrong conclusions about how and what to change (or, alternatively, what not to change). What good does it do a teacher to create formative assessments for CCSSM if students are not being adequately exposed to the rigorous content and instruction in math needed to be successful? OTL can be an important check on this assumption by capturing access to ideal CCSSM content and methods at the start, thus eliminating equivocation about whether or not students have had the proper exposure.

Developing a set of informationally significant indicators of performance for a variety of stakeholders and purposes is critical to ensuring that the improvement decisions we make are based on solid evidence about the resources, processes, and outcomes at work in our schools (Adams et al., 2017). The evidence presented here, it is hoped, provides states and local practitioners tasked with implementing ESSA with more information about indicators of math teaching and learning that might assist in guiding their improvement efforts.

Footnotes

Appendix A

Item Information on Study Measures.

Study Measure	TIMSS Survey Name	Item Description
Opportunity to learn: Math instruction	ATBM03F, G	In teaching mathematics to this class, how often do you usually ask students to do the following . . .
		“explain their answers”
		“relate what they are learning in mathematics to their daily lives”
Opportunity to learn: Math content	ATBM07AE-BE, BG	If topic was in the curriculum before fourth grade . . . “mostly taught before this year” . . . if taught half this year but not yet completed, . . .” mostly taught this year” . . . not in the curriculum “not yet taught”
Opportunity to learn: Math content	ATBM07AE-BE, BG	Math topics: Concepts of decimals, adding and subtracting decimals, number sentences, number patterns, lines, angles, coordinate systems, shapes, reflections and rotations, area
Teacher preparedness: Content	ATBM12AE BE, BG	How well prepared do you feel you are to teaching the following . . .
Teacher preparedness: Content	ATBM12AE BE, BG	Math topics: Concepts of decimals, adding and subtracting decimals, number sentences, number patterns, lines, angles, coordinate systems, shapes, reflections and rotations, area
Teacher preparedness: Instruction	ATBM02B,C,E	In teaching mathematics to this class, how confident to do you feel . . .
		“show students a variety of problem solving skills”
		“provide challenging tasks for capable students”
		“help students appreciate the value of learning mathematics”
Teacher preparedness: Math professional development	ATBM11A-E	In the past 2 years, have you participated in the professional development in any of the following . . .
		“mathematics content”
		“mathematics pedagogy/instruction”
		“mathematics curriculum”
		“integrating information technology into mathematics”
		“mathematics assessment”
Teacher professional community	ATBG10A-C, E	How often do you have the following interactions with teachers . . .
		“discuss how to teach a particular topic”
		“collaborate in planning and preparing instructional materials”
		“share what I have learned about my teaching experiences”
		“work together to try out new ideas”
Learning climate	ATBG06B-D	How would you characterize each of the following within your school . . .
		“Teachers’ understanding of the school’s curricular goals”
		“Teachers’ degree of success in implementing the school’s curriculum”
		“Teachers’ expectations for student achievement”
Math resources	ACBG10CA-CD	How much of your school’s capacity to provide instruction affected by a shortage of inadequacy of the following . . .
		“teachers with a specialization in mathematics”
		“computer software for mathematics instruction”
		“library materials relevant to mathematics instruction”
		“audio-visual resources for mathematics instruction”
Instructional leadership: Goals	ACBG15A-D	During the last year, approximately how much time have you spent on the following school leadership activities in your role as a school principal . . .
		“promoting the school’s educational vision or goals”
		“developing the school’s curricular and educational goals”
		“monitoring teachers’ implementation of the school’s educational goals in their teaching”
		“monitoring students’ learning progress to ensure that the school’s educational goals are reached”
Instructional leadership: Teacher feedback	ACBG15I, J	During the last year, approximately how much time have you spent on the following school leadership activities in your role as a school principal . . .
		“initiating a discussion to help teachers who have problems in the classroom”
		“advising teachers who have questions or problems with their teaching”
School readiness	ACBG16A-H	About how many of the students in your school can do the following when they begin elementary school . . .
		“recognize most of the alphabet”
		“read some words”
		“read sentences”
		“write letters of the alphabet”
		“write some words”
		“count up to 100 or higher”
		“recognize all 10 written numbers from 1 to 10”
		“write all 10 numbers from 1 to 10”

Appendix B

Math Curriculum Coherence for Opportunity to Learn Content.

TIMSS 2011	Broad Coding Topics^a	CCSSM, Grades Covered^a	A+ Countries, Grades Covered^a
Number
Concepts of whole numbers, including place value and ordering	Whole number meaning	1-5	1-5
Adding, subtracting, multiplying, and/or dividing with whole numbers	Whole number operations	1-5	1-5
Concepts of fractions (fractions as parts of a whole or of a collection, or as a location on a number line; comparing and ordering of fractions)	Fractions	1-6	3-6
Adding and subtracting with fractions	Fractions	1-6	3-6
Concepts of decimals, including place value and ordering	Decimals	4-6	4-6
Adding and subtracting with decimals	Decimals	4-6	4-6
Number sentences (finding the missing number, modeling simple situations with number sentences)	Equations and formulas	3-8	3-8
Number patterns (extending number patterns and finding missing terms)	Equations and formulas	3-8	3-8
Geometric shapes and measures
Lines: measuring, estimating length of; parallel and perpendicular lines	Measurement units	1-8	1-7
	Estimating quantity and Size	NA	4-5
	2D geometry basics	2 and 4-8	3-8
Comparing and drawing angles	2D geometry basics	2 and 4-8	3-8
Using informal coordinate systems to locate points in a plane (e.g., in square B4)	2D geometry basics	2 and 4-8	3-8
Elementary properties of common geometric shapes	2D geometry basics	2 and 4-8	3-8
Reflections and rotations	2D geometry basics	2 and 4-8	3-8
Relationships between two-dimensional (2D) and three-dimensional (3D) shapes	3D geometry	1-2 and 5-8	7-8
Finding and estimating areas, perimeters, and volumes	Perimeter, area and volume, estimating computations	3-8	4-8
Data display
Reading data from tables, pictographs, bar graphs, or pie charts	Data representation and analysis	1-8	3-6 and 8
Drawing conclusions from data displays	Data representation and analysis	1-8	3-6 and 8
Displaying data using tables, pictographs, and bar graphs	Data representation and analysis	1-8	3-6 and 8

Note. CCSSM = Common Core State Standards for Mathematics; TIMSS = Trends in International Math and Science Study. Only bolded math topics are included in the measure.

Compares TIMSS 2011 topics with prior math curriculum, CCSSM, and A+ top performing country research studies (see Schmidt & Houang, 2012).

Appendix C

Performance Expectations for Opportunity to Learn Instruction.

TIMSS 2011 Teacher Questionnaire	TIMSS 2011 Assessment	TIMSS 1995 Assessment	Porter (2002)
In teaching, I ask student to . . .	“Cognitive domains”	“Performance expectations”	“Cognitive demand”
Listen to me explain how to solve problems	Knowing	Knowing Using routine procedures	Perform procedures
Memorize rules, procedures, and facts	Knowing	Knowing Using routine procedures	Memorize
Work problems (individually or with peers) with my guidance	Applying [Reasoning]	Investigating and problem solving [Communicating]	Perform procedures [Solve nonroutine problems]
Work problems together in the whole class with direct guidance from me	Applying [Reasoning]	Investigating and problem solving [Communicating]	Perform procedures [Solve nonroutine problems]
Work problems (individually or with peers) while I am occupied by other tasks	Applying [Reasoning]	Investigating and problem solving [Communicating]	Perform procedures [Solve nonroutine problems]
Explain their answers	Reasoning	Reasoning [Communicating]	Conjecture/generalize/prove Communicate understanding
Relate what they are learning in mathematics to their daily lives	Reasoning	Reasoning [Communicating]	Conjecture/generalize/prove [Communicate understanding] [Solve nonroutine problems]

Note. The entries in bold indicate the two items that were selected. TIMSS = Trends in International Math and Science Study.

Declaration of Conflicting Interests

The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The authors disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This research was supported by a grant from the American Educational Research Association, which receives funds for its “AERA Grants Program” from the National Science Foundation under NSF Grant #DRL-0941014. Opinions reflect those of the authors and do not necessarily reflect those of the granting agencies.

Notes

ORCID iDs

A. Urick

T. G. Ford

Author Biographies

Angela Urick is an assistant professor in the Department of Educational Leadership and Policy Studies, Jeannine Rainbolt College of Education, at the University of Oklahoma. She studies how to lead and measure school improvement.

Alison S. P. Wilson is a PhD student and full-time graduate research assistant in Educational Administration, Curriculum, and Supervision at the University of Oklahoma. Her research interests include opportunity to learn, equity, and ethical leadership.

Timothy G. Ford is an assistant professor of Educational Leadership and Policy Studies and a research scientist with the Oklahoma Center for Educational Policy at the University of Oklahoma. His research interests include teacher evaluation policy and practice, leadership for school improvement, teacher motivation, and trust in schools.

William C. Frick is an associate professor in the Department of Educational Leadership and Policy Studies, Jeannine Rainbolt College of Education, University of Oklahoma. His research interests include ethics, reform and community revitalization efforts, and the intersection of identity and schooling.

Meredith L. Wronowski is a doctoral candidate in Educational Leadership and Policy Studies at the University of Oklahoma. Her research interests include evaluation of federal accountability policy and teacher retention in urban schools.

References

Adams

C. M.

Ford

T. G.

Forsyth

P. B.

Ware

J. K.

Barnes

L. B.

Khojasteh

. . . Lepine

J. A.

(2017, February). Next generation school accountability: A vision for improvement under ESSA. Palo Alto, CA: Learning Policy Institute.

Aguirre-Muñoz

Boscardin

(2008). Opportunity to learn and English learner achievement: Is increased content exposure beneficial? Journal of Latinos and Education, 7, 186-205.

Ball

(2000). Bridging practices: Intertwining content and pedagogy in teaching and learning to teach. Journal of Teacher Education, 51, 241-247.

Baratz-Snowden

(1993). Opportunity to learn: Implications for professional development. Journal of Negro Education, 62, 311-323.

Bendikson

Robinson

Hattie

(2012). Principal instructional leadership and secondary school performance. SET: Research Information for Teachers, 2012(1), 2-8.

Blase

(1999). Principals’ instructional leadership and teacher development: Teachers’ perspectives. Educational Administration Quarterly, 35, 349-378.

Blase

(2000). Effective instructional leadership: Teachers’ perspectives on how principals promote teaching and learning in schools. Journal of Educational Administration, 38, 130-141.

Blömeke

Kaiser

(2012). Homogeneity or heterogeneity? Profiles of opportunities to learn in primary teacher education and their relationship to cultural context and outcomes. ZDM, 44, 249-264.

Blömeke

Suhl

Kaiser

Döhrmann

(2011). Family background, entry selectivity and opportunities to learn: What matters in primary teacher education? An international comparison of fifteen countries. Teaching and Teacher Education, 28, 44-55.

10.

Boykin

Noguera

(2011). Creating the opportunity to learn: Moving from research to practice to close the achievement gap. Alexandria, VA: Association for Supervision and Curriculum Development.

11.

Bryk

A. S.

Sebring

P. B.

Allensworth

Easton

J. Q.

Luppescu

(2010). Organizing schools for improvement: Lessons from Chicago. Chicago, IL: University of Chicago Press.

12.

Callahan

R. M.

(2005). Tracking and high school English learners: Limiting opportunity to learn. American Educational Research Journal, 42, 305-328.

13.

Carroll

(1963). A model of school learning. Teachers College Record, 64, 723-723.

14.

Cobb

Jackson

(2011). Assessing the quality of the common core state standards for mathematics. Educational Researcher, 40, 183-185.

15.

Cogan

L. S.

Schmidt

W. H.

Wiley

D. E.

(2001). Who takes what math and in which track? Using TIMSS to characterize US students’ eighth-grade mathematics learning opportunities. Educational Evaluation and Policy Analysis, 23, 323-341.

16.

Coleman

J. S.

(1988). Social capital in the creation of human capital. American Journal of Sociology, 94, S95-S120.

17.

Cooley

W. W.

Leinhardt

(1980). The instructional dimensions study. Educational Evaluation and Policy Analysis, 2, 7-25.

18.

Cuban

(1984). Transforming the frog into a prince: Effective schools research, policy and practice at the district level. Harvard Educational Review, 54, 129-151.

19.

Darling-Hammond

(1996). The right to learn and the advancement of teaching: Research, policy, and practice for democratic education. Educational Researcher, 25(6), 5-17.

20.

Darling-Hammond

Bae

Cook-Harvey

C. M.

Lam

Mercer

Podolsky

Stoslch

E. L.

(2016). Pathways to new accountability through the Every Student Succeeds Act. Palo Alto, CA: Learning Policy Institute.

21.

Day

Sammons

(2016). The impact of leadership on student outcomes: How successful school leaders use transformational and instructional strategies to make a difference. Educational Administration Quarterly, 52, 221-258.

22.

Desimone

Smith

Phillips

(2013). Linking student achievement growth to professional development participation and changes in instruction: A longitudinal study of elementary student and teachers in Title I schools. Teachers College Record, 115(5), 1-46.

23.

Dougherty

K. J.

(1996). Opportunity-to-learn standards: A sociological critique. Sociology of Education, 69, 40-65.

24.

Edmonds

(1979). Some schools work and more can. Social Policy, 17(5), 17-18.

25.

Elliott

(1998). School finance and opportunities to learn: Does money well spent enhance students’ achievement? Sociology of Education, 71, 223-245.

26.

Elmore

Fuhrman

(1995). Opportunity-to-learn standards and the state role in education. Teachers College Record, 96, 432-457.

27.

Every Student Succeeds Act . (2015). Retrieved from https://www.gpo.gov/fdsys/pkg/BILLS-114s1177enr/pdf/BILLS-114s1177enr.pdf

28.

The Every Student Succeeds Act: Explained. (2015, December 7). Education Week, p. 17. Retrieved from http://www.edweek.org/ew/articles/2015/12/07/the-every-student-succeeds-act-explained.html

29.

Firestone

W. A.

Wilson

B. L.

(1985). Using bureaucratic and cultural linkages to improve instruction: The principal’s contribution. Educational Administration Quarterly, 21(2), 7-30.

30.

Foy

Arora

Stanco

G. M.

(2013). TIMSS 2011 user guide for the international database. Boston, MA: TIMSS & PIRLS International Study Center, Boston College Lynch School of Education.

31.

Frank

K. A.

Zhao

Borman

(2004). Social capital and the diffusion of innovations within organizations: The case of computer technology in schools. Sociology of Education, 77, 148-171.

32.

Gamoran

(1987). The stratification of high school learning opportunities. Sociology of Education, 60, 135-155.

33.

Gamoran

Gunter

Williams

(2005). Professional community by design: Building social capital through teacher professional development. In Hedges

L. V.

Schneider

B. L.

(Eds.), The social organization of schooling (pp. 111-126). New York, NY: Russell Sage Foundation.

34.

Gamoran

Porter

A. C.

Smithson

White

P. A.

(1997). Upgrading high school mathematics instruction: Improving learning opportunities for low-achieving, low-income youth. Educational Evaluation and Policy Analysis, 19, 325-338.

35.

Goertz

M. E.

(1994). Opportunity to learn: Instructional practices in eighth-grade mathematics: Data from the 1990 NAEP trial state assessment (Report No. 32). New Brunswick, NJ: Consortium for Policy Research in Education.

36.

Goldhaber

D. D.

Brewer

D. J.

(1996). Evaluating the effect of teacher degree level on educational performance. Rockville, MD: Westat.

37.

Greenberg

Rhodes

Stancavage

(2004, April). Prepared to teach: Teacher preparation and student achievement in eighth-grade mathematics. Paper presented at the annual meeting of the American Educational Research Association, San Diego, CA.

38.

Hallinger

(2005). Instructional leadership and the school principal: A passing fancy that refuses to fade away. Leadership and Policy in Schools, 4, 221-239.

39.

Hallinger

Bickman

Davis

(1996). School context, principal leadership, and student reading achievement. Elementary School Journal, 96, 527-549.

40.

Hallinger

Heck

R. H.

(1996). Reassessing the principal’s role in school effectiveness: A review of empirical research, 1980-1995. Educational Administration Quarterly, 32, 5-44.

41.

Hallinger

Heck

R. H.

(1998). Exploring the principal’s contribution to school effectiveness: 1980-1995. School Effectiveness and School Improvement, 9, 157-191.

42.

Hallinger

Heck

R. H.

(2002). What do you call people with visions? The role of vision, mission and goals in school leadership and improvement. In Leithwood

K. A.

Hallinger

(Eds.), Second international handbook of educational leadership and administration (pp. 9-40). Amsterdam, Netherlands: Springer.

43.

Hallinger

Murphy

(1985). Assessing the instructional leadership behavior of principals. Elementary School Journal, 86, 217-248.

44.

Halvorsen

Lee

Andrade

(2009). A mixed-method study of teachers’ attitudes about teaching in urban and low-income schools. Urban Education, 44, 181-224.

45.

Heck

R. H.

Larsen

T. J.

Marcoulides

G. A.

(1990). Instructional leadership and school achievement: Validation of a causal model. Educational Administration Quarterly, 26, 94-125.

46.

Hendriks

Scheerens

(2013). School leadership effects revisited: A review of empirical studies guided by indirect-effect models. School Leadership & Management, 33, 373-394.

47.

Herman

J. L.

Klein

D. C.

(1996). Evaluating equity in alternative assessment: An illustration of opportunity-to-learn issues. Journal of Educational Research, 89, 246-256.

48.

Hoy

W. K.

Hannum

J. W.

(1997). Middle school climate: An empirical assessment of organizational health and student achievement. Educational Administration Quarterly, 33, 290-311.

49.

Hoy

W. K.

Tarter

C. J.

Hoy

A. W.

(2006). Academic optimism of schools: A force for student achievement. American Educational Research Journal, 43, 425-446.

50.

L.-T.

Bentler

P. M.

(1999). Cutoff criteria for fit indices in covariance structure analysis: Conventional criteria versus new alternatives. Structural Equation Modeling, 6, 1-55.

51.

Joncas

Foy

(2013). Sample design in TIMSS and PIRLS. Boston, MA: TIMSS & PIRLS International Study Center, Boston College Lynch School of Education.

52.

Kimura-Walsh

Yamamura

Griffin

Allen

(2009). Achieving the college dream? Examining disparities in access to college information among high achieving and non-high achieving Latina students. Journal of Hispanic Higher Education, 8, 298-315.

53.

Klein

(2016, January 6). New law, fresh challenges. Education Week, pp. 10-12.

54.

Kline

(2015). Principles and practice of structural equation modeling (4th ed.). New York, NY: Guilford Press.

55.

Kurz

Elliott

Kettler

Yel

(2014). Assessing students’ opportunity to learn the intended curriculum using an online teacher log: Initial validity evidence. Educational Assessment, 19, 159-184.

56.

Lach

(2016). Using indicator data to drive K-12 STEM improvements in states and districts: Implications and recommendations for leaders and policymakers. Menlo Park, CA: SRI International.

57.

Leithwood

Day

Sammons

Harris

Hopkins

(2006). Seven strong claims about successful school leadership. Nottingham, England: National College for School Leadership.

58.

Leithwood

Jantzi

(2008). Linking leadership to student learning: The contributions of leader efficacy. Educational administration quarterly, 44, 496-528.

59.

Leithwood

Mascall

(2008). Collective leadership effects on student achievement. Educational Administration Quarterly, 44, 529-561.

60.

Leithwood

Patten

Jantzi

(2010). Testing a conception of how school leadership influences student learning. Educational Administration Quarterly, 46, 671-706.

61.

Hallinger

(2016). Principal leadership and school capacity effects on teacher learning in Hong Kong. International Journal of Educational Management, 30, 76-100.

62.

Lochmiller

(2016). Examining administrators’ instructional feedback to high school math and science teachers. Educational Administration Quarterly, 52, 75-109.

63.

Lochmiller

Acker-Hocevar

(2016). Making sense of principal leadership in content areas: The case of secondary math and science instruction. Leadership and Policy in Schools, 15, 273-296.

64.

Louis

K. S.

Marks

H. M.

(1998). Does professional community affect the classroom? Teachers’ work and student experiences in restructuring schools. American Journal of Education, 106, 532-575.

65.

Lucas

S. R.

(1999). Tracking inequality: Stratification and mobility in American high schools. New York, NY: Teachers College Press.

66.

MacCallum

R. C.

Browne

M. W.

Sugawara

H. M.

(1996). Power analysis and determination of sample size for covariance structure modeling. Psychological Methods, 1, 130-149.

67.

Marks

H. M.

Louis

K. S.

(1999). Teacher empowerment and the capacity for organizational learning. Educational Administration Quarterly, 35, 707-750.

68.

Marks

H. M.

Printy

S. M.

(2003). Principal leadership and school performance: An integration of transformational and instructional leadership. Educational Administration Quarterly, 39, 370-397.

69.

McDonnell

L. M.

(1995). Opportunity to learn as a research concept and a policy instrument. Educational Evaluation and Policy Analysis, 17, 305-322.

70.

McKnight

C. C.

Crosswhite

F. J.

Dossey

J. A.

Kifer

Swafford

J. O.

Travers

K. J.

Cooney

T. J.

(1987). The underachieving curriculum: Assessing U.S. school mathematics from an international perspective. Champaign, IL: Stipes.

71.

Means

Mislevy

Smith

Peters

Gerard

S. N.

(2015). Measuring the monitoring progress K-12 STEM indicators: A road map. Washington, DC: SRI Education.

72.

National Research Council. (2011). Successful K-12 STEM education: Identifying effective approaches in science, technology, engineering, and mathematics. Washington, DC: National Academies Press.

73.

National Research Council. (2013). Monitoring progress toward successful K-12 STEM education: A nation advancing? Washington, DC: National Academies Press.

74.

Newman

, & Associates. (1996). Authentic achievement: Why are some schools more successful than others. San Francisco, CA: Jossey-Bass.

75.

Oakes

(1990). Multiplying inequalities: The effects of race, social class, and tracking on opportunities to learn mathematics and science. Santa Monica, CA: Rand.

76.

O’Day

J. A.

Smith

M. S.

(1993). Systemic reform and educational opportunity. In Fuhrman

S. H.

(Ed.), Designing coherent education policy (pp. 250-313). San Francisco, CA: Jossey-Bass.

77.

Opdenakker

Van Damme

(2007). Do school context, student composition and school leadership affect school practice and outcomes in secondary education? British Educational Research Journal, 33, 179-206.

78.

Porter

(1995). The uses and misuses of opportunity-to-learn standards. Educational Researcher, 24(1), 21-27.

79.

Porter

(2002). Measuring the content of instruction: Uses in research and practice. Educational Researcher, 31(7), 3-14.

80.

Porter

McMaken

Hwang

Yang

(2011). Assessing the common core standards opportunities for improving measures of instruction. Educational Researcher, 40, 186-188.

81.

Robinson

V. M.

Lloyd

C. A.

Rowe

K. J.

(2008). The impact of leadership on student outcomes: An analysis of the differential effects of leadership types. Educational Administration Quarterly, 44, 635-674.

82.

Robitaille

D. F.

Garden

R. A.

(1989). The IEA study of mathematics II: Contexts and outcomes of school mathematics (Vol. 2). Oxford, England: Pergamon Press.

83.

Schmidt

Burroughs

Cogan

Houang

(2017). The role of the subject-matter content in teacher preparation: An international perspective for mathematics. Journal of Curriculum Studies, 49, 111-131.

84.

Schmidt

Cogan

Houang

(2011). The role of opportunity to learn in teacher preparation: An international context. Journal of Teacher Education, 62, 138-153.

85.

Schmidt

W. H.

Cogan

L. S.

Houang

R. T.

McKnight

C. C.

(2011). Content coverage differences across districts/states: A persisting challenge for US education policy. American Journal of Education, 117, 399-427.

86.

Schmidt

W. H.

Cogan

L. S.

McKnight

C. C.

(2011). Equality of educational opportunity: Myth or reality in US schooling? American Educator, 34(4), 12-19.

87.

Schmidt

W. H.

Houang

R. T.

(2007). Lack of focus in the mathematics curriculum: Symptom or cause? In Loveless

(Ed.), Lessons learned: What international assessments tell us about math achievement (pp. 65-84). Washington, DC: Brookings Press.

88.

Schmidt

W. H.

Houang

(2012). Curricular coherence and the common core state standards for mathematics. Educational Researcher, 41, 294-308.

89.

Schmidt

W. H.

Houang

R. T.

Cogan

Blömeke

Tatto

M. T.

Hsieh

F. J.

. . . Paine

(2008). Opportunity to learn in the preparation of mathematics teachers: Its structure and how it varies across six countries. ZDM, 40, 735-747.

90.

Schmidt

W. H.

Maier

(2009). Opportunity to learn. In Sykes

Schneider

B. L.

Plank

D. N.

(Eds.), Handbook on education policy research (pp. 541-559). New York, NY: Routledge.

91.

Schmidt

W. H.

McKnight

(2012). Inequality for all: The challenge of unequal opportunity in American schools. New York, NY: Teachers College Press.

92.

Schmidt

McKnight

Houang

Wang

Wiley

Cogan

Wolfe

(2001). Why schools matter: A cross-national comparison of curriculum and learning. San Francisco, CA: Jossey-Bass.

93.

Schmidt

W. H.

Wang

H. C.

McKnight

C. C.

(2005). Curriculum coherence: An examination of US mathematics and science content standards from an international perspective. Journal of Curriculum Studies, 37, 525-559.

94.

Schmidt

Zoido

Cogan

(2013). Schooling matters: Opportunity to learn in PISA 2012 (OECD Education Working Paper No. 95). Paris, France: OECD.

95.

Schreiber

J. B.

Nora

Stage

F. K.

Barlow

E. A.

King

(2006). Reporting structural equation modeling and confirmatory factor analysis results: A review. Journal of Educational Research, 99, 323-338.

96.

Seashore Louis

Dretzke

Wahlstrom

. (2010). How does leadership affect student achievement? Results from a national US survey. School Effectiveness and School Improvement, 21, 315-336.

97.

Sebastian

Allensworth

(2012). The influence of principal leadership on classroom instruction and student learning: A study of mediated pathways to learning. Educational Administration Quarterly, 48, 626-663.

98.

Smithson

J. L.

Porter

Blank

(1995). Describing the enacted curriculum: Development and dissemination of opportunity to learn indicators in science education. Washington, DC: Council of Chief State School Officers. Retrieved from https://files.eric.ed.gov/fulltext/ED385430.pdf

99.

Spillane

(2006). Distributed leadership. San Francisco, CA: Jossey-Bass.

100.

Spillane

J. P.

Hallett

Diamond

J. B.

(2003). Forms of capital and the construction of leadership: Instructional leadership in urban elementary schools. Sociology of Education, 76, 1-17.

101.

SRI International. (2014). Developing a STEM education indicator system. Retrieved from http://stemindicators.org

102.

Stevens

Grymes

(1993). Opportunity to learn: Issues of equity for poor and minority students (Report). Washington, DC: U.S. Department of Education, Office of Educational Research and Improvement, National Center for Education Statistics.

103.

Stoll

Bolam

McMahon

Wallace

Thomas

(2006). Professional learning communities: A review of the literature. Journal of Educational Change, 7, 221-258.

104.

Supovitz

Sirinides

May

. (2009). How principals and peers influence teaching and learning. Educational Administration Quarterly, 46, 31-56.

105.

Urick

(2016). Examining US principal perception of multiple leadership styles used to practice shared instructional leadership. Journal of Educational Administration, 54, 152-172.

106.

Urick

Bowers

A. J.

(2011). What influences principals’ perceptions of academic climate? A nationally representative study of the direct effects of perception on climate. Leadership and Policy in Schools, 10, 322-348.

107.

Urick

Bowers

A. J.

(2014). What are the different types of principals across the United States? A latent class analysis of principal perception of leadership. Educational Administration Quarterly, 50, 96-134.

108.

U.S. Department of Education. (2016). Every Student Succeeds Act. Retrieved from http://www.ed.gov/essa?src=rn

109.

Wang

(1998). Opportunity to learn: The impacts and policy implications. Educational Evaluation and Policy Analysis, 20, 137-156. doi:10.3102/01623737020003137

110.

Wang

Tang

(2013). Profiles of opportunities to learn for TEDS-M future secondary mathematics teachers. International Journal of Science and Mathematics Education, 11, 847-877.

111.

Wong

Boey

Lim-Teo

Dindyal

(2012). The preparation of primary mathematics teachers in Singapore: Programs and outcomes from the TEDS-M study. ZDM, 44, 293-306.

112.

York-Barr

Duke

(2004). What do we know about teacher leadership? Findings from two decades of scholarship. Review of Educational Research, 74, 255-316.