Abstract
The authors synthesize the findings of a research project to extend what is known about educative curriculum materials, or curriculum materials designed with the intent of supporting teacher learning as well as student learning. Drawing on a three-year program of research, including several close observational case studies and a large-scale quasi-experiment, the authors demonstrate how teachers use curriculum materials, what evidence there is of teachers’ uptake of ideas in educative curriculum materials, and what evidence there is of impact on teacher and/or student knowledge. These findings are situated in the literature, and the authors discuss how, taken together, the findings suggest design principles for educative curriculum materials. The authors close with implications for research.
Keywords
Two decades ago, Ball and Cohen (1996) argued for curriculum materials as a lever for effecting change in classrooms, describing how these everyday resources could support teacher learning and teacher change. Remillard (2000) refers to such texts as materials that “speak to teachers, not merely through them” (p. 347). Building on the work of Ball and Cohen, Remillard, and others, Davis and Krajcik (2005) advanced design heuristics (or design guidelines) for educative curriculum materials, or curriculum materials designed explicitly to support teacher learning as well as student learning. Davis and Krajcik identified a gap in the literature when explaining their terminology: Note that we use the term “design heuristics” rather than the more common “design principles.” Our recommendations are intended to be useful rules of thumb and not principles, which would imply a level of empirical testing that researchers have not yet undertaken. . . . Nonetheless, these design heuristics take us one step closer to the principled design of educative curriculum materials, necessary for this early stage of the research. (p. 4)
Here, we synthesize findings from our program of empirical research conceived to address this gap by integrating the results of a three-year research project in which we designed, tested, refined, and conducted a quasi-experiment with a set of educative curriculum materials. The work we describe spans design and development studies (including in-depth case studies) and efficacy 1 studies (a large quasi-experimental study) (Institute of Education Sciences & National Science Foundation, 2013)—an unusual scope for a relatively short project. While some findings have been presented in other research papers, they have not yet been synthesized to demonstrate the broad significance of the work.
Such syntheses across a program of research can provide useful guidance for the field. For example, Romance and Vitale (2001) reported on a multiyear study of science instruction; the authors used their project’s findings to discuss implications for policy and research. Hill and Charalambous (2012a) synthesized a set of related case studies, presented together in a special issue, to develop lessons learned about the relationships among teacher knowledge, mathematics instruction, and curriculum materials. Such research syntheses serve a different role than reviews of the literature or empirical reports on a single study and have the potential to provide important design, policy, and research guidance for the field.
The themes across our studies—situated within the literature—allow us to present a set of design principles for educative curriculum materials. To date, most design work specific to educative curriculum materials has been grounded in “best bets.” Curriculum developers rarely describe their full development process but are primarily guided by intuition, theory, and/or Davis and Krajcik’s (2005) design heuristics (see e.g., Beyer & Davis, 2009; Schneider & Krajcik, 2002). The empirically based and literature-grounded design principles presented here should interest curriculum developers seeking to make their materials supportive of teacher learning.
We organize our findings and design principles around three questions—questions that undergirded the entire program of research:
Research Question 1: How do teachers use and adapt curriculum materials in their enactment? What are some of the key influences on their decision making while using curriculum materials, including educative curriculum materials?
Research Question 2: What evidence do we have of teachers’ uptake of ideas from educative features in the educative curriculum materials?
Research Question 3: What evidence do we have of an impact of educative curriculum materials on teacher and student outcomes?
By curriculum materials, we mean resources designed to be used by teachers in classrooms to guide their instruction (Stein, Remillard, & Smith, 2007). By educative curriculum materials, we mean curriculum materials designed to support teacher learning as well as student learning (Davis & Krajcik, 2005). We use educative features to refer to the elements in curriculum materials specifically intended to provide support for teacher learning. These might take the form of “callout” boxes with teacher tips, graphics illustrating conceptual relationships among the ideas in a unit, guides to the use of readings, or suggestions for providing students feedback on their writing (e.g., Arias, Bismack, Davis, & Palincsar, 2016; Arias, Palincsar, & Davis, 2015; Cervetti, Kulikowich, & Bravo, 2015; Drake, Land, & Tyminski, 2014; Hill & Charalambous, 2012b). By uptake, we mean the ways teachers adopt, in language and/or action, ideas or practices recommended in the educative features.
We make two key assumptions in presenting our design principles. First, building on Ball and Cohen (1996), Remillard (2005), Davis and Krajcik (2005), Hill and Charalambous (2012a), and others, curriculum designers should support multiple domains of teacher knowledge and practice, including subject matter knowledge and pedagogical content knowledge as well as associated teaching practices for supporting the content and disciplinary practices students are learning. Second, building on Davis and Krajcik’s design heuristics, educative curriculum materials should reveal the rationales underlying recommendations. Davis and Krajcik argued that when teachers understand the rationale behind a recommendation, they are better able to apply the ideas across lessons, units, and contexts. These assumptions guide our design of educative features.
Our project focused on supporting elementary teachers of science. However, most of our implications should apply across content areas and grade bands given that educative curriculum materials are used across subject areas and grades (e.g., Collopy, 2003; Grossman & Thompson, 2008; McNeill, 2009). Four of the six design principles we propose are domain general. Furthermore, we conclude the paper with implications for the design and conduct of research regarding educative curricula; while grounded in our studies of science curriculum materials, these should apply to other areas as well.
Teachers, Curriculum Materials, and Elementary Science Education
Curriculum use entails transforming the written curriculum to the intended curriculum to the enacted curriculum (Stein et al., 2007). Teacher characteristics (e.g., knowledge and beliefs about teaching, learning, and learners) and characteristics of the curriculum materials (e.g., how the materials represent content) shape how teachers use curriculum materials. In our designs, we aim to support teachers’ engagement with curriculum materials by helping teachers recognize the rationales for recommendations and how they can productively adapt the recommendations in their classrooms. Indeed, any engagement with curriculum materials will involve adaptation to support teachers’ needs and contexts (e.g., Remillard, 2000). Curriculum materials are well suited to support teachers because they are situated in teachers’ daily work (Ball & Cohen, 1996; Putnam & Borko, 2000).
Educative curriculum materials can support positive outcomes across subject areas, grade levels, and contexts. For example, in secondary English language arts (Grossman & Thompson, 2008), elementary mathematics (Collopy, 2003; Hill & Charalambous, 2012a, 2012b), middle school science (Schneider & Krajcik, 2002; McNeill, 2009), and elementary science (Cervetti et al., 2015), teachers using educative curriculum materials changed their ideas about teaching and/or their practice, and/or their students experienced positive learning outcomes. Hill and Charalambous (2012b) found that educative features in curriculum materials seemed to support reform-oriented instruction. Grossman and Thompson (2008) found that curriculum materials provided opportunities for teacher learning, and for some teachers, this supported development of subject matter knowledge and pedagogical content knowledge. McNeill (2009) found that when teachers’ enactments departed from recommendations in educative features, their students’ learning suffered. Complementing these studies of impact on enactment is a study of use of specific educative features. Research with preservice teachers revealed differential use of educative features (Beyer & Davis, 2009). For example, they were more influenced by narratives (i.e., brief vignettes describing a teacher’s enactment of a lesson) than expository texts when identifying specific adaptations to teaching plans.
Elementary science teaching, with its associated challenges, provides an especially fruitful arena for studying the effects of educative curriculum materials. First, most elementary teachers teach all academic subject areas, including all science disciplines, even though they may not have robust science knowledge (Abell, 2007). Second, elementary science is often last in an ever-growing list of demands on teachers. It is both infrequently taught and inadequately supported through professional learning opportunities (Banilower et al., 2013; National Academies of Sciences Engineering and Medicine [NASEM], 2015). Supporting elementary science teachers is particularly critical now (NASEM, 2015) given current reforms in the United States that call for integrating disciplinary core ideas (e.g., heredity), crosscutting concepts (e.g. patterns), and scientific practices (e.g., arguing from evidence; National Research Council [NRC], 2012). Educative curriculum materials have potential to provide sustained, scalable, job-embedded, discipline-specific, professional learning opportunities that teachers need.
However, little is known about how specific features of educative curriculum materials support specific changes in practice among teachers. That is, how are educative features with particular characteristics (e.g., narratives, graphics, definitions) used, interpreted, and incorporated into instruction by teachers? Our first research question lays a foundation, and our second research question directly addresses this gap in the literature. Furthermore, while a handful of studies have attempted to show large-scale effects of educative curriculum materials (e.g., Cervetti et al., 2015; McNeill, Pimentel, & Strauss, 2013), continued work is needed—hence our third research question regarding impacts. Studying the effects of any instructional intervention, including educative curriculum materials, is challenging and involves tradeoffs in research design because so many factors can shape student outcomes (see e.g., Cheung, Slavin, Kim, & Lake, 2017). Thus, we also address the methodological lessons we have learned to provide guidance for future research.
Overview of Methods Used in Program of Research
Here we overview the methods we used throughout our program of research.
Overview of Context
We developed and tested the efficacy of educative features in the context of two kit-based, commercially available curricular units designed with National Science Foundation funding in the United States. These upper elementary units—Science and Technology Concepts’ (STC) Ecosystems and Electric Circuits units (National Science Resources Center [NSRC], 2004a, 2004b)—are intended to promote conceptual understanding through scientific investigation.
We worked with “typical” elementary teachers, not science specialists, because of our interest in educative curriculum materials as a way of effecting large-scale change. All teachers were certified in elementary education. None had much, if any, experience with STC curriculum materials, and they varied in their confidence in and experience with teaching science. We worked with ethnically diverse schools and districts that were mainly in underserved communities, especially in our initial work. We aspired to support those schools, and we anticipated that educative curriculum materials might be well suited to effect change in these contexts.
Overview of Three Phases of Work
Our work proceeded through three phases. Tables 1 and 2 summarize the research foci and data.
Timeline and Research Summary
Teacher and School Characteristics and Data Available in Years 1, 2, and 3
Note. In Years 1 and 2, we video-recorded and took field notes for every lesson and collected as many student notebooks as were available. Year 3 refers to data for the case study teachers only. FRL = free and reduced price lunch.
In Year 1, we observed three teachers using the original STC curriculum materials to inform the design of the educative features we would overlay onto the materials. As a result of this pilot work, complemented by insights from the literature and support from content-area experts, we designed educative features to enhance the existing STC curriculum materials. (This theoretically and empirically informed design process is detailed elsewhere; Davis et al., 2014.)
In Year 2, four teachers used the enhanced curriculum materials, including the educative features we designed, and we collected a similar corpus of data as in Year 1. Based on Year 2 outcomes, we refined the educative features.
In Year 3, the design and development work culminated in a quasi-experimental efficacy study (Smith & Smith, 2014). We recruited 20 elementary schools (ranging from rural to urban fringe and from well-resourced to underresourced), randomly assigning them to the treatment condition, which received the enhanced curriculum materials, or the comparison condition, which received the original materials. 2 (See Figure 1 for a sampling of educative features in the enhanced materials.) The treatment group included 28 teachers and the comparison group 22 teachers. All but 1 teacher taught both units. We conducted one-day professional development workshops before each unit. We gathered common data for all 50 teachers (i.e., assessments of teacher content knowledge at multiple timepoints, lesson logs describing each enactment, pre- and postassessments of student content knowledge, and samples of student notebooks), and we supplemented those data with more extensive data (see Table 2) from 6 case study teachers: 4 from the treatment condition and 2 from the comparison condition. We purposefully selected these case study teachers to reflect the classrooms and teachers in the larger sample. In addition to school-level demographic characteristics, we considered each teacher’s content knowledge and beliefs, their willingness to have their instruction video-recorded, and logistics (e.g., travel time to the school). We oversampled from our treatment teachers to gain insight into the variability of the teachers’ enactment and because we already had data, from our pilot work, on teachers’ use of the original curriculum materials (as would be provided in the comparison condition).

Sampling of the educative features developed: reading guide, content storyline, narrative, rubric
Overview of Data Analysis
In several of the smaller scale studies, we coded 2-minute segments of video-records; this convention enables analysis of teaching practices (Borko, Jacobs, Eiteljorg, & Pittman, 2008). We used partial-interval time sampling to mark each segment for the relevant code(s). We employed “tracers” (Duncan & Frymier, 1967), which allowed us detect the influence of educative features on teachers’ enactments. We used tracers as both a design element (seeding keywords or phrases in the design of the educative features) and an analytic tool (using those keywords or phrases in our coding to characterize uptake from the educative features). For example, we designed educative features that suggested teaching moves to discuss how and why to make justified predictions and included language such as justification and claim; these teaching moves and language were not in the original curriculum materials and thus served as tracers. To use these tracers as an analytic tool, we designed coding schemes that included the teaching moves and language tracers and analyzed the teachers’ enactments of lessons, teacher-created materials, and student work for evidence of these tracers. Our use of tracers is described in depth elsewhere (e.g., Arias, Bismack, et al., 2016; Bismack, Arias, Davis, & Palincsar, 2015), as are our analytic approaches in the small-scale studies more generally. The analyses for the quasi-experiment, which employed multilevel modeling, are described in Smith and Smith (2014).
Identifying Themes Across Studies
We took several steps to identify patterns across the studies within this larger program of research. We summarized the studies to identify main findings. We developed matrices that characterized case study teachers across studies (e.g., how a teacher used content, practice, or literacy-focused features). We developed initial themes, then reread the body of work to see if there was additional support or counterevidence for a theme. Throughout this process, we had ongoing conversations as a research group about the potential themes to sharpen our insights and identify counterevidence. We also used the themes to point toward potential design principles, which we tested in much the same manner.
Findings, Principles, and Connections to the Literature Base
Overall, we found that, as expected, teachers adapted the curriculum materials as they enacted them. Their decision making was informed by competing goals, which sometimes promoted positive changes and sometimes limited students’ opportunities to learn. We found extensive evidence of uptake of the ideas and recommendations in the educative features. Yet our evidence of impact on outcomes was more limited. In the following sections, we provide background literature related to each research question and then explicate the results and our design principles.
Use of Curriculum Materials and Key Influences on Decision Making
The literature documents how teachers adapt curriculum materials as they enact them (e.g., Collopy, 2003; McNeill, 2009; Remillard, 2000). Not all adaptations maintain students’ opportunities to learn (Hill & Charalambous, 2012a; Stein et al., 2007). Teachers’ decision making is shaped by factors related to the teachers themselves (e.g., knowledge, beliefs, goals, and identities), their curriculum materials (e.g., how inquiry-oriented they are; their look, structure, or voice), and the context (e.g., how supportive they are of innovative teaching) (for reviews of this literature, see Davis, Janssen, & Van Driel, 2016; Remillard, 2005). Here, we show how patterns in our findings complement the existing literature and inform our design principles.
Teachers adapted both the original and the enhanced curriculum materials as they enacted them. Table 3 summarizes the findings associated with Research Question 1. The changes teachers made sometimes maintained or augmented students’ opportunities to learn and sometimes limited those opportunities. Factors that shaped teachers’ decision making included time pressure, the strengths and limitations of the curriculum materials, and teachers’ understanding of the scientific practices and their students. Because most research related to this question stems from mathematics education, our findings extend the literature while being consistent with what is already understood.
Findings Regarding Teachers’ Use of Curriculum Materials
These findings, along with the literature reviewed briefly, lead us to our first design principle
3
: Design Principle 1: Teachers will adapt curriculum materials. These adaptations are likely to be informed by teachers’ concerns about time and student capabilities and experiences. By anticipating these adaptations, educative features can facilitate principled and productive adaptations. Therefore, educative features should provide suggestions for adaptations of lessons that would take different amounts of time and meet a range of students’ needs while still meeting the intent of the reforms embedded in the curriculum materials. Examples of such educative features could include narratives describing choices that may reduce time needed while maintaining opportunities to learn.
Our findings highlight the need for educative curriculum materials to anticipate and support teachers’ adaptations to curriculum materials.
Uptake From Educative Features
Studies have demonstrated uptake of ideas in educative curriculum materials through characterizing teacher practice, teacher learning, or student outcomes (e.g., Beyer & Davis, 2009; Cervetti et al., 2015; Collopy, 2003; McNeill, 2009; Schneider & Krajcik, 2002). Studies have not, however, untangled which ideas teachers take up from specific kinds of educative features. We use tracers, as described previously, to discern the effects of specific educative features.
Students’ work, teachers’ comments on that work, and teachers’ practice all reflected teachers’ uptake of ideas, pointing to effects of specific educative features. Table 4 summarizes these and other findings associated with this research question. Educative features supported teachers in engaging students in some scientific practices, learning the content themselves, and teaching with scientific text.
Findings Regarding Teachers’ Uptake of Ideas in Educative Features
Based on the findings summarized in Table 4, we extend the existing literature by identifying specific effects of specific educative features. Overall, patterns from this set of studies include: (a) the salience of highly situated educative features, including teaching tools, and (b) the need for multiple vectors for supporting content knowledge and scientific practices. These studies collectively also demonstrate the variation in how different teachers take up different educative features.
These are important contributions because they provide direct design guidance for curriculum developers, captured in Design Principles 2 and 3.
Design Principle 2: Educative features that provide representations of practice can support teachers’ uptake of the ideas in the features. Educative features that can be used as teaching tools can support concrete changes in teachers’ practice. Furthermore, sample student work can help teachers set higher expectations for their students than they might otherwise have. Therefore, educative features should be situated and grounded in teachers’ practice. Some features grounded in practice can be directly applied as teaching tools in the classroom. Examples of educative features that are situated in teachers’ practice include (a) rubrics that illustrate essential features of key ideas of reforms along with sample student work and possible teacher comments that reflect those key ideas and (b) narratives that describe teachers’ enactment of lessons in ways that demonstrate key ideas of reforms. Examples of teaching tools include rubrics, examples of key scientific ideas, and student-friendly definitions of terms.
Design Principle 2 reflects the situated nature of teacher learning (Putnam & Borko, 2000). This design principle is also bolstered by limited previous empirical findings indicating that preservice elementary teachers used ideas from narratives more readily than those that appeared in general, nonsituated 4 educative features (Beyer & Davis, 2009). Collopy’s (2003) work also suggests the importance of tight connections between educative features and instruction. For example, one teacher from Collopy’s study did not shift in her fundamental beliefs about mathematics learning, but she made sustained changes to the kinds of tasks she provided to students to focus more on conceptual understanding and mathematical reasoning; these changes were consistent with her use of the educative features.
Design Principle 3 further highlights the forms of support designers might provide, specifically with respect to enhancing teachers’ subject matter knowledge.
Design Principle 3: As expectations for students and teachers change, demands on teachers’ subject matter knowledge grow. Multiple vectors may help teachers identify the “big ideas” to highlight in lessons. Therefore, designers should use multiple forms of support for highlighting important content. Examples include content storylines, student-friendly definitions, and graphics.
Hill and Charalambous’s (2012b) study of teachers’ use of educative mathematics curriculum materials highlights the importance of educative features that can bolster teachers’ own content knowledge for teaching. These authors compared a teacher’s prior incorrect understanding of a mathematical concept to her ability to support her students in understanding that concept after using the curriculum materials, which provided support for the concept. Our empirical work demonstrating uptake suggests specific ways of providing such support.
Findings across Research Questions 1 and 2 lead us to Design Principles 4 and 5.
Design Principle 4: Different teachers will need and take up different kinds of educative features (in terms of substance and form). Teachers’ variable uptake will be based on the needs they perceive in themselves (e.g., their knowledge of content, assessment, or reading strategies) and their students (e.g., their typical content struggles). Therefore, designers should develop a constellation of educative features that have the potential to meet these various needs. Designers also should help teachers recognize how the recommendations differ from their current practice, in part through emphasizing the rationales for the recommendations. A constellation could be constituted purposefully using different forms of educative features (e.g., narratives and call-out boxes) with different foci (e.g., subject matter knowledge and pedagogical content knowledge). An example of an educative feature that would highlight how recommendations differ from current practice could include a reading guide that signals changes from typical literacy practices.
Providing a range of educative features and calling attention to how these features enable practice that may differ from the status quo is consistent with the field’s understanding of the nature of learning and the role of tools in supporting learning (e.g., Putnam & Borko, 2000).
Our fifth design principle is science-specific.
Design Principle 5: Teachers take up the practice of scientific explanation in a limited way. Therefore, educative features should help teachers (a) appreciate the definition, intention, and value of constructing scientific explanations and (b) learn how to support students in engaging in explanation construction and argumentation. Examples include narratives, expository text, capstone questions, and rubrics that synergistically define, illustrate, and guide explanation construction and argumentation in the classroom.
Our curriculum use studies highlighted the uneven nature of teachers’ understanding of scientific practices. Our uptake studies and curriculum use studies revealed the challenges teachers had specific to supporting scientific explanation. McNeill’s (e.g., 2009) work on the challenges of scientific explanation and argumentation provides further support for this design principle. Students struggled with explanation construction and developing arguments. While supporting a claim with evidence is not straightforward for students, providing reasoning—that is, explicating how their data count as evidence to support a claim and what scientific principles connect a claim and evidence—tends to be particularly difficult. Furthermore, teachers’ instructional practices helped shape students’ engagement in these practices in that a clear definition of the practices seemed to support students’ engagement in them, while an oversimplification seemed to undermine students’ engagement. Because explanation and argumentation are key scientific practices, inherent in student sensemaking, yet present significant challenges for teachers and students, providing extensive support through educative science curriculum materials—and other venues—seems crucial.
Impact on Teachers and Students
Our third research question relates to the impact of educative curriculum materials on teachers and students. Some studies have found effects of educative curriculum materials on student learning (e.g., McNeill et al., 2013), and others suggest possible relationships with learning (e.g., Cervetti et al., 2015). We were interested in whether we would find similar effects.
Our project culminated in Year 3 with an efficacy study intended to discern effects of educative curriculum materials on teacher and/or student learning (for details, see Smith & Smith, 2014). Table 5 summarizes these findings. The primary outcome measures for teachers were multiple-choice assessments of science content knowledge. The lack of difference between conditions shown in Table 5 is not surprising given what we learned about how teachers took up the features, as described previously. Although several features were designed to support content knowledge, teachers used them unevenly; furthermore, some supports for content knowledge were provided as unit front matter; hence, they were less situated in the day-to-day teaching of the unit than our findings suggest would have been optimal. The findings for student content knowledge outcomes were similar to those for teacher outcomes.
Findings Related to the Impact on Teachers and Students
Finally, the student assessments included one constructed-response question asking students to construct and justify a prediction related to the content of the unit. (Due in part to constraints on project funding, we included only a single item to assess science practices.) These were scored for the extent and quality of justification students included. On this out-come measure, we did detect a difference between students in treatment and comparison groups (Arias, Smith, Davis, Marino, & Palincsar, 2017).
In sum, our efficacy study did not, for the most part, reveal significant effects on teacher or student knowledge. The exception was that students in the treatment condition did perform better on an item requiring the justification of a prediction. We interpret these findings in light of our findings on teachers’ uptake of educative features, which indicated extensive variability, and in light of the many factors that matter in supporting student learning, of which teacher subject matter knowledge is just one.
Our final design principle, which is also science-specific, is grounded in the findings across all three research questions, about teachers’ use of curriculum materials, teachers’ uptake of ideas from educative curriculum materials, and the impact studies.
Design Principle 6: Certain scientific practices, including making and recording observations and making and justifying predictions, were taken up effectively by most teachers. Therefore, educative features should support easier-to-enact scientific practices, with the idea of moving incrementally toward more ambitious science teaching in elementary classrooms. Designers should connect to teachers’ existing teaching practice to create leverage points while helping teachers recognize salient differences. Examples of such educative features could include narratives and how-and-why support for prediction that reinforce the need for justification, point the path toward argumentation, and connect to ways teachers may already use prediction (e.g., in English language arts).
Our findings indicated that some scientific practices were more accessible for teachers to take up than others. The NRC’s (2012) Framework for K–12 Science Education emphasizes a range of scientific practices but provides little guidance about how to support teachers to include these in their instruction. Star’s (2015) recommendation for incremental change toward reform-based teaching and Janssen, Westbroek, and van Driel’s (2014) bridging progression to support incremental change might help designers conceptualize how to move toward implementing these practices in typical classrooms. Rather than expecting a wholesale change in lesson design, these models focus on small, more manageable changes. Starting with more straightforward practices and using them as entry points to the more complex practices could be a version of such incremental change. For example, prediction, as a practice that involves justification based on evidence and/or reasoning, may serve as an “entrée” for explanation and argumentation by providing a lower stakes need for justification. Carefully recorded observations can provide evidence for explanation and argumentation and thus make engagement in those more sophisticated practices possible.
In summary, Table 6 presents all six design principles.
Design Principles for Educative Curriculum Materials
Implications for Studying Educative Curriculum Materials
This program of research, like any such program, has limitations. Many of our results stem from rich but small case studies of a few teachers. We had 50 teachers in our quasi-experimental study but uneven numbers in each condition. We had a single item to measure scientific practice despite evidence that in retrospect suggests teachers’ uptake of support for scientific practice was more clear than their uptake of support for content.
With the hindsight afforded by a completed project, we can glean insights about methodological lessons learned for studying educative curriculum materials. Here we first discuss the tradeoffs we considered and the rationales for our decisions in three areas: the curriculum materials themselves, the specification and development of outcome measures, and the timing of the efficacy study.
The first group of research design decisions centered on the curriculum materials. We decided on a 6- to 8-week instructional unit to allow for sustained opportunity for students to engage with a coherent set of disciplinary concepts, but we stipulated two units to allow for the possibility of generalizing claims across content areas. A second critical decision was whether to create these instructional materials de novo, along with the educative features, or to overlay the features on existing materials. We chose to develop features and overlay them on highly respected, commercially available materials. This decision saved time and money as we were able to build on a strong foundation. It enhanced ecological validity and eased the process of creating a comparison condition. As noted, we systematically built tracers into the educative features so we could study teachers’ uptake and the influence on student learning opportunities. This turned out to be an important way to identify uptake even when we did not see large-scale impact; other researchers should consider incorporating tracers into their materials as well.
The second group of decisions centered on outcome measures. The ultimate dependent variable for the efficacy study was a measure of student learning of the targeted science concepts (see Smith & Smith, 2014). We created our own measures, aligned closely with the science content in the two units and with good evidence of validity and reliability. We opted not to focus on science practices in the teacher or student assessments, with the exception of the single practice question in the student assessments, due to limitations in time and resources. We recognized this as a design tradeoff with implications about the claims we would be able to make. 5 In retrospect, we should have divided assessment resources more evenly. Evidence from our constructed response item supports our claim that the features positively shaped student learning, but we had limited opportunity to collect this evidence due to our assessment development decisions. A pilot efficacy study might have helped us better align the assessments with the anticipated outcomes; other researchers should consider incorporating such a study.
Deciding when to measure the dependent variable represents a third study design decision. The instructional materials we chose, though highly regarded, are not widely used (Banilower et al., 2013). Typically, teachers need at least two iterations with materials before they begin to use them purposefully (Loucks-Horsley, Love, Stiles, Mundry, & Hewson, 2003). Thus, our teachers’ uptake of the educative features may have been diminished by their preoccupation with the instructional materials themselves. Our decision to use existing materials afforded the time needed for conducting the pilot work, which informed development and refinement of the educative features. A more ideal design, though not feasible within our funding constraints, would have allowed all teachers to become more familiar with the materials before initiating the efficacy study. Teachers’ familiarity with the materials might then have freed them to attend more purposefully to the supports—or given them a false sense of security and led them away from the supports (a question that itself bears study). This aspect of study design also highlights the importance of including a valid and reliable measure of the extent of teachers’ use of educative features. The educative features were used differently by different teachers. However, because teachers inconsistently completed the lesson logs that asked about use of each feature, we did not have a measure of curricular use from all teachers that could be included in the quantitative analysis—like the others mentioned here, a methodological issue likely to be relevant regardless of discipline. Other researchers should try to ensure the incorporation of such a measure of curricular use.
The previous discussion focuses on implications related to study design. Our work also points to implications for the focus of future research. Our evidence that the educative features impacted student learning is mixed. The field needs studies across contexts that examine whether and how specific aspects of educative curriculum materials affect student learning. Further, future research should continue to explore the tensions described by Davis and Krajcik (2005): finding the balance between adequate support and too much, navigating between prescription and autonomy, and determining the role of technology. Progress is being made particularly on the last of these issues (e.g., Marco-Bujosa, McNeill, González-Howard, & Loper, 2016), and further work could explore how advances in assessment could help tailor the educative features provided to teachers. Another area ripe for continued work is exploring how teachers learn to use educative features in curriculum materials (cf. Drake et al., 2014). Furthermore, educative curriculum materials are not a panacea and should be supported by strong professional development (e.g., Pringle, Mesa, & Hayes, 2017). Empirical research could investigate how professional development and educative curriculum materials work together to support teacher learning. Finally, of course, we hope that the design principles presented here provide guidance for the development of and research on new educative curriculum materials across a range of subject areas, grade levels, and purposes.
Conclusion
As demands on teachers and students increase, educative curriculum materials have potential for improving teaching and learning in elementary, middle school, and high school classrooms (cf. Krajcik & Delen, 2017). When used in conjunction with other professional learning opportunities, they are uniquely positioned to support such change. As we noted previously, a decade ago, the field lacked an empirical basis for design principles for educative curriculum materials, and Davis and Krajcik (2005) instead put forward a set of design heuristics. In the ensuing years, a stronger empirical foundation has been laid, allowing the development of design principles for curriculum development with an eye toward supporting teacher learning as well as student learning.
In brief, our domain-general design principles suggest that educative features should:
suggest adaptations of lessons that would take different amounts of time and meet a range of students’ needs,
be situated and grounded in teachers’ practice,
take multiple forms,
work together to meet a range of teacher needs.
Our domain-specific design principles suggest that educative science curriculum materials should provide purposeful support for scientific argumentation as well as “entry-level” scientific practices as a mechanism to effect change. Going forward, the recommendations made here can help shape an agenda for design-based research on educative curriculum materials.
