Professional Development for Data-Based Instruction in Early Writing: Tools,Learning,and Collaborative Support

Abstract

Few teachers receive adequate preparation to provide effective individualized instruction for children with intensive early writing needs. In this article, the authors describe an attempt to close this learning gap, by developing Data-Based Instruction-Tools, Learning, and Collaborative Support (DBI-TLC), a comprehensive professional development (PD) system that provides tools, learning opportunities, and ongoing collaborative supports for teachers to implement DBI in early writing. They describe the theoretical framework that has guided this work, the teacher population with whom they worked, their approach to assessing important teacher outcomes, and their development process. They highlight key findings that align with their theory of change, and discuss implications for further research and teacher preparation.

Keywords

general special education quasi-experimental research research methods writing instruction response to intervention assessment curriculum-based measurement

Recent national efforts (e.g., National Center for Intensive Intervention; intensiveintervention.org) have aimed to improve teachers’ individualization of instruction. These efforts have focused primarily on reading and math, with much less attention on writing. Yet, learning to write is critical to students’ literacy development (Biancarosa & Snow, 2004; Graham & Hebert, 2010), their overall academic performance in school (Shanahan, 2004), and their future postsecondary and employment opportunities (Graham & Perin, 2007). Students with disabilities are particularly at risk for writing failure; for example, in the 2011 National Assessment of Educational Progress (NAEP), only 5% of eighth graders with learning disabilities (LD) reached proficiency in writing; 60% were below basic levels (National Center for Education Statistics, 2012). Furthermore, these statistics have not improved since the 2007 NAEP results.

To address students’ writing development needs, particularly for those who experience significant difficulties, teachers require knowledge, skills, and tools for effective writing instruction and intervention. Yet, there is little evidence that teachers are prepared to deliver effective writing instruction. In a national survey of elementary teachers, more than half cited their teacher education programs as poor to merely adequate in preparing them to deliver effective writing instruction (Cutler & Graham, 2008). Furthermore, in a study of the relation between teacher knowledge and time allocation in literacy instruction (Spear-Swerling & Zibulsky, 2014), many general and special education teachers—particularly those in primary grades—allocated little to no time to writing assessment or basic writing skills instruction, despite evidence that such instruction is foundational for later writing proficiency (e.g., Berninger, Nielsen, Abbott, Wijsman, & Raskind, 2008).

To address these gaps in teacher preparation, our research team was funded a development and innovation project by the Institute of Education Sciences (IES). These development and innovation—or Goal 2—projects provide the opportunity to explore research questions aligned to three phases: development, feasibility testing, and pilot testing. The purpose of our Goal 2 project was to develop a professional development (PD) system that provides tools, learning opportunities, and ongoing collaborative supports for teachers to effectively implement and individualize early writing instruction using Data-Based Instruction (DBI). DBI is a hypothesis-driven, empirical approach to individualizing instruction (Deno & Mirkin, 1977) that entails a systematic, ongoing cycle of assessment and intervention delivered in addition to (or instead of) general core instruction (Danielson & Rosenquist, 2014; see Figure 2 for basic steps in the DBI process).

For the purposes of this project, we characterize DBI as a Tier 3 intervention (in a Response to Intervention model) or as an intervention delivered to students who receive special education services. As part of this project, we developed and revised DBI-TLC, which includes the following components:

Tools, which include Curriculum-Based Measures (CBM) for monitoring student progress in writing (see McMaster, Ritchey, & Lembke, 2011, for a review), research-based intervention lessons and materials (see McMaster, Kunkel, Shin, Jung, & Lembke, in press, for a review), and decision-making tools;

Learning modules (a series of face-to-face workshops designed to train teachers to use the tools and implement all components of DBI with fidelity); and

Collaborative support (ongoing coaching to support teachers’ implementation of DBI).

Across 3 years, we tested DBI-TLC through a series of iterative phases that included development, feasibility testing, and finally, a pilot study that entailed a small randomized control trial. The focus of this article is the process and teacher outcomes that were part of each phase of iterative development and refinement. Below, we describe our theoretical framework, the teacher population that we aimed to support, how we assessed critical teacher outcomes, our process of development and refinement, and key findings. We end with implications for future research and teacher preparation and PD.

Theoretical Framework

Desimone (2009) proposed that researchers studying PD should base their work on a common conceptual framework that would “elevate the quality of professional development studies and subsequently the general understanding of how best to shape and implement teacher learning opportunities for the maximum benefit of both teachers and students” (p. 181). This framework assumes that effective PD should lead to increased teacher knowledge, skills, and attitudes, leading to positive changes in practice and improved student outcomes. Desimone proposed several features that effective PD should include. We incorporated these features into DBI-TLC in the following ways: (a) focus on content, by emphasizing knowledge and skills needed to implement each DBI step; (b) active learning opportunities over an extended duration, by providing multiple chances to view models, practice, and apply content with feedback during workshops and in the classroom; (c) coherence, by ensuring that DBI components align with theory and core academic standards, and are integrated into existing instructional routines; and (d) collective participation, by ensuring that teachers have frequent opportunities to collaborate with coaches and peers.

Our goal was to provide teachers with support to implement DBI with their students, including PD in critical areas, ongoing coaching visits, and supported practice of activities that would benefit implementation fidelity. Our PD and coaching was focused both on the DBI elements and also on implementation of writing instruction. As illustrated in Figure 1, our work was guided by a theory of change that aligns with Desimone’s (2009) framework, and that relies on five assumptions:

Figure 1.

Theory of change.

Figure 2.

Steps in the DBI process.

An important goal is to improve student outcomes in early writing.

Given that no single approach will work for all students (Deno, 1990), teachers must make timely and appropriate instructional decisions for students with significant needs.

To make timely and appropriate instructional decisions, teachers need a strong framework—such as DBI—and must implement it with fidelity (Stecker, Fuchs, & Fuchs, 2005).

To implement a practice such as DBI with fidelity, teachers require knowledge and skills of that practice (e.g., Cunningham, Perry, Stanovich, & Stanovich, 2004; Spear-Swerling, 2009). Teachers also require self-efficacy, or “confidence that they can perform the actions that lead to student learning” (Graham, Harris, Fink, & MacArthur, 2001, p. 178).

We can increase teacher knowledge, skills, and self-efficacy through DBI-TLC. By providing in-depth PD in the theoretical and empirical underpinnings, as well as classroom applications, of DBI, we can increase teacher knowledge and skills. By providing ongoing support that allows teachers to experience success in applying new knowledge and skills to their instructional practices, we can increase teacher efficacy related to DBI and early writing.

Teacher Population Studied

In our project, we aimed to support teachers who provided direct instructional services to children in Grades 1 to 3 with intensive early writing needs, including special education teachers, Title I teachers, and teachers supporting English Learners. We also aimed to support teachers in different types of settings serving diverse student populations. Thus, teachers who participated in this project were located in two demographically diverse districts in two states. The districts were in a small city and a large urban area, with K-12 enrollments ranging from 17,000 to more than 36,000. The free or reduced price lunch percentage ranged from 41% to 64%. Students receiving special education services ranged from 10% to 18%, and were predominantly White in one district (62.1%) and predominantly non-White in the other district (66%).

The number of teacher participants varied across phases of the research, with 50 total teacher participants across all phases. In the development phase of the project, 17 teachers participated across sites; all worked in elementary settings and identified as female. Participants were primarily special education teachers, but also included four administrators, a reading specialist, and an English as a Second Language (ESL) teacher. Participants were predominantly White, and had an average of 10 years teaching special education. For the feasibility phase of the project, 11 special education, ESL, and intervention teachers participated across sites. All were female and the majority self-identified as White (93%) and had earned at least a master’s degree (87%). Teachers had been in their current positions an average of 8 years. In the pilot study phase of the project, 22 teachers across sites who provided direct support to students in Grades 1 to 3 who had specific needs in beginning writing skills and who had or were at risk for disabilities were randomly assigned to treatment and control groups within school or within demographically similar schools (n = 12 treatment and 10 control). The majority of participants were female (95%) and self-identified as White/European American (86%). Most were special education teachers, with one ESL teacher in each group. More than 60% had at least a master’s degree and had taught in their current positions between 2 and 5 years on average, but had an average of 8 years of total experience.

Assessments of Teacher and Student Learning

In line with our theory of change, we collected data on important teacher outcomes that would lead to a greater understanding of their knowledge, skills, self-efficacy, and use of DBI with fidelity, including timely and appropriate individualization of instruction. Each type of outcome is described in detail below.

DBI Knowledge and Skills

In the feasibility and pilot phases of the project, all teacher participants completed a DBI Knowledge and Skills pre–posttest. This assessment was developed and field-tested as part of the project. It included 40 multiple-choice questions that assessed teachers’ knowledge related to children’s writing development and writing instruction, the purpose of DBI, specific DBI steps, and skills related to administering, scoring, and using CBM writing data to make instructional decisions, with number of items correct serving as the final score. Field-tests yielded internal consistency coefficients (Cronbach’s α) of .58 to .78. Individual items were revised based on item-level analyses; the final version was used for the pilot study.

Teacher Efficacy and Writing Orientation

In the pilot study phase of the project, teachers completed pre- and posttest surveys to explore the effect of DBI-TLC on teachers’ efficacy for teaching writing as well as their writing orientation. Critical to teachers’ successful use of student data are their knowledge of literacy development and practices, skill in selecting appropriate instruction to promote student learning (Cunningham et al., 2004; Spear-Swerling, 2009), and self-efficacy. Self-efficacy relates to teachers’ “confidence that they can perform the actions that lead to student learning” (Graham et al., 2001, p. 178). Teachers with strong self-efficacy are “better organized . . . more willing to try new ideas . . . more positive about teaching” (Graham et al., 2001, p. 178) and more likely to persist in helping struggling students (Allinder, 1994). In addition to assessing self-efficacy, one of our expert consultants on the project suggested that we examine teachers’ writing orientation, which influences how teachers teach and what they focus on while teaching (Graham, Harris, MacArthur, & Fink, 2002; Troia, Lin, Cohen, & Monroe, 2011). Given strong evidence that explicit instruction is critical to support children who experience difficulties in learning to write (Berninger et al., 2008) and that the research-based intervention tools that we provided in this project focused on explicit instruction, we were interested in whether and how teachers’ writing orientation might change as a result of participating in DBI-TLC.

Teacher efficacy was assessed using Graham et al.’s (2001) modified version of Gibson and Dembo’s (1984) Teacher Efficacy Scale (TES), which includes 16 items related to writing instruction at the elementary level. The teacher responds to statements such as, “When students’ writing performance improves, it is usually because I found better ways of teaching” (1 = strongly disagree; 6 = strongly agree). A factor analysis based on a national sample of teachers indicated two dimensions: personal teaching efficacy and general teaching efficacy, with Cronbach’s alphas of .84 and .69, respectively (Graham et al., 2001).

Writing orientation was assessed using Graham et al.’s (2002) Writing Orientation Scale, a 13-item survey using a 6-point Likert-type scale, with questions such as, “A good way to begin writing instruction is to have children copy good models of each particular type of writing.” The questions align with the subscales Natural Writing, Correct Writing, and Explicit Instruction (Graham et al., 2002). Natural Writing includes an emphasis on incidental and informal learning; Correct Writing places emphasis upon spelling, grammar, copying models, and using standard English; and Explicit Instruction includes teaching skills overtly and systematically.

CBM Reliability and Fidelity of DBI Components

During the pilot study phase of the project, we periodically checked teachers’ reliability of CBM scoring, given that accurate scoring of student writing samples was critical to instructional decision-making. Members of the research team collected students’ writing samples from each teacher each month, rescored them, and compared the teachers’ scores with their own. Reliability was calculated as number of agreements divided by agreements plus disagreements.

Fidelity of DBI was assessed using the Accuracy of Implementation Rating Scales (AIRS), originally created by Fuchs, Deno, and Mirkin (1984) and modified for this project. The AIRS consists of three core DBI components: CBM, Writing Instruction, and Decision Making (DM). AIRS-CBM and Writing Instruction include detailed checklists of critical steps involved in assessment and intervention. AIRS-DM involves comparing teachers’ graphed data with a “decision log” in which they recorded information about instructional changes they made, including the date, type of change, and rationale for the change. Based on this information, we determined whether the teachers’ decisions (a) were timely (i.e., they made a decision every six to eight data points, as prescribed in training), (b) were appropriate (i.e., they used prescribed decision rules to determine whether to change instruction and how to change it), (c) indicated the change on the graph, and (d) noted the type of decision that was made. Fidelity was recorded as the number of items observed divided by the total number of applicable items.

In addition, during the pilot study phase, we administered a brief survey to DBI-TLC and control teachers, asking them to estimate how frequently they made instructional decisions based on individual student progress. This information would allow a cursory examination of the extent to which DBI-TLC led to increased instructional decision-making.

Student Outcomes

Although not the primary focus of this article, it is worth noting that we also included measures of student learning in the pilot study, because we hypothesized that student achievement would be influenced by DBI-TLC implementation. Each participating teacher identified two to three students in their classes who needed intensive early writing intervention. Students were then screened and monitored weekly using CBM tasks for early writing, and were given a standardized writing test, the Test of Early Written Language, 3rd Edition (TEWL-III; Hresko, Herron, Peak, & Hicks, 2012), pre–post. More detail about these measures can be found in McMaster, Lembke, and Shin (2016).

Development and Refinement of DBI-TLC

A central aim of this project was to develop and iteratively revise and refine DBI-TLC. This iterative development process occurred across three phases, described in detail below.

Development Phase

The first phase involved development and refinement of a DBI manual and tools for implementing research-based early writing assessment, intervention, and data-based decision-making, along with fidelity tools. We also developed learning modules to provide teachers with knowledge and skills needed to implement DBI. To develop these components, our research team reviewed the literature in CBM for beginning writers, early writing intervention, and data-based decision-making. We gathered existing materials that could be used or modified for DBI “toolkits” for teachers. We drafted the manual, drawing from similar existing manuals (e.g., Deno & Mirkin, 1977; Fuchs & Fuchs, 2007), and shared drafts with two national experts in CBM and DBI and with leaders from the participating districts. Their feedback on comprehensiveness, organization, clarity, accuracy, and utility for teachers was incorporated into a revised version of the manual.

CBM prompts and administration and scoring guides were drawn from previous research (e.g., Hampton & Lembke, 2016; Lembke, Deno, & Hall, 2003; McMaster, Du, & Pétursdóttir, 2009; McMaster, Ritchey, & Lembke, 2011) and refined for use in this project. We developed graphing tools and a decision rubric to support teachers’ examination, interpretation, and use of CBM data to inform instruction. In addition, results of a best-evidence synthesis (McMaster et al., in press) and input from a national expert in writing intervention were used to develop and refine research-based writing lessons and materials, along with guidance for teachers to analyze students’ strengths and weaknesses in writing and match interventions to their specific needs. We also modified the AIRS (Fuchs et al., 1984) to assess fidelity of the assessment, instruction, and decision-making components of DBI.

After developing the tools, we developed a series of learning modules, consisting of four daylong workshops that provided in-depth training in (a) the overall DBI process, (b) how to administer and score CBM for early writing and graph the data, (c) how to implement and individualize early writing instruction, and (d) how to use CBM data to intensify early writing intervention. Training included information about the writing process, including how students develop writing skills. The recommended research-based approaches we helped teachers to develop are based on theoretical models of early writing development. Our particular focus was on the three key components of the Simple View of Writing (transcription, text generation, and self-regulation; Berninger & Amtmann, 2003). Throughout the training, we emphasized the alignment of these components with the writing assessments and instructional approaches that were to be incorporated into the DBI process.

We delivered the learning modules via a 4-day Summer Institute to 17 teachers from the two participating districts. During the time in this Institute, teachers completed the DBI Knowledge and Skills assessment (pre- and posttest), learned all DBI procedures,received materials, and provided in-depth feedback regarding the learning modules and materials. Their feedback was elicited via semistructured group discussions, written surveys, and a brief survey following the Institute. We also asked teachers to help generate ideas for the “collaborative support” component of DBI-TLC.

All feedback was entered into a spreadsheet, categorized by themes, prioritized, and used to revise the tools and learning modules. Changes based on this feedback included creating a “frequently asked questions” document and quick-start guides for each module to make start-up more feasible, changing the order of the learning modules to better reflect the steps of DBI, and streamlining the module content into three daylong workshops to fit a more realistic schedule.

Based on teachers’ anticipated need for ongoing support, along with findings from a systematic literature review (Poch et al., in preparation) and input from a national PD expert, we also developed the collaborative support component, which we conceptualized as a cycle of coaching aligned to the learning modules. From the literature review (Poch et al., in preparation), we derived five principles of coaching: Coaching should (a) be teacher oriented, (b) focus on building mastery, (c) be observable and measurable, (d) emphasize alignment with existing curriculum and instructional practices, and (e) be geared toward teachers’ sustained implementation. The resulting collaborative support system included an opportunity for teachers to practice new content, share data, and discuss implementation issues with their coaches and peers. After each module, the teacher completed a performance assessment to gauge mastery of DBI content. For example, after learning about CBM, the teacher scored CBM samples, and was expected to meet a criterion of 85% scoring accuracy. The teacher’s coach then directly observed implementation of assessment and intervention, and provided feedback and support as needed via face-to-face (at least biweekly) and virtual (email, phone) sessions. Each coaching session followed a basic agenda with key steps that aligned with the coaching principles. Coaches were trained on the coaching process and principles at a “Coaches’ Institute.”

Finally, we developed tools to examine fidelity of implementation of the “TLC” components of the package. These fidelity measures included (a) Fidelity-T—an assessment of teachers’ accuracy in locating tools for the assessment, instruction, and decision-making components of DBI (all tools were provided in binders as well as on a Google drive); (b) Fidelity-L—detailed checklists of key points and activities to be covered in each of the learning modules; and (c) Fidelity-C—checklists of key points to be covered during each coaching session.

Feasibility Phase

In the second phase of the project, we conducted a feasibility test of DBI-TLC. Eleven special education teachers and intervention specialists from the two sites received all DBI tools, participated in the learning modules, and received Coaching supports while they implemented DBI with students with intensive writing needs for 12 weeks. Teachers kept logs of time spent on various DBI activities (e.g., collecting, scoring, and graphing CBM data; preparing instructional plans; making instructional decisions) for each student. Coaches kept logs of their interactions with teachers, including the frequency, focus, and outcomes of those interactions. At the end of the study, we invited teachers to attend a 2-hour focus group at each site. We developed a semistructured protocol to elicit teachers’ input on the feasibility of the entire DBI process, the learning curve, the cycle of implementation (including learning modules and coaching), and the DBI tools. Teachers also completed an anonymous survey about their satisfaction with the Coaching components of DBI-TLC, a 12-item, researcher-developed instrument where teachers were asked six open-ended questions about the strengths and weaknesses of coaching. This coaching survey was used to iteratively develop and revise our coaching methods.

Data from teachers’ and coaches’ logs, the focus group, and the coaching survey were entered into a spreadsheet. Then, two researchers developed a coding scheme to capture all feedback from these data sources. We began by independently identifying themes to categorize the feedback, then compared notes and came to agreement through an iterative process of generating and comparing themes until we agreed on all codes. This process resulted in three “levels” of coding: Level 1 distinguished among what teachers perceived to be actual facilitators (features of DBI-TLC that teachers perceived fostered implementation of DBI), suggested facilitators (features that teachers suggested would facilitate implementation if incorporated into DBI-TLC), and barriers/challenges (things that teachers perceived stood in the way or made DBI implementation difficult). Level 2 indicated whether the perceived facilitators and barriers were related to DBI, tools, learning modules, or collaborative support components of DBI-TLC.

Level 3 entailed more specific, descriptive categories. For example, teachers cited a specific actual facilitator (Level 1) of DBI (Level 2) to be its alignment (Level 3) with existing instructional programming or with students’ needs. A specific suggested facilitator (Level 1) of DBI tools (Level 2) was to improve their usability (Level 3) by making more ready-made materials. A specific barrier/challenge (Level 1) to coaching (Level 2) included external conflicts (Level 3) such as limited time to schedule coaching sessions. Complete definitions and examples of codes are reported in Poch, McMaster, and Lembke (2016).

Results of this analysis were used to both evaluate the feasibility of DBI-TLC and to inform further revisions (for a full description of analysis and results, see Poch et al., 2016). With respect to feasibility, teachers highlighted several key facilitators, including that the step-by-step process made DBI less overwhelming than other new programs, that the assessment and intervention procedures were well aligned with and easy to incorporate into existing instructional programs, that they found the data to be valuable, and that they were likely to continue using DBI in the future. From this feedback, we concluded that the overall DBI-TLC components and procedures were generally feasible to implement in school settings.

To further revise DBI-TLC, we attended closely to suggested facilitators and barriers/challenges that teachers identified. Most of this feedback related to challenges with organizing and managing DBI tools. Thus, we heeded teachers’ suggestions to make intervention materials more accessible, organized, and easy to use. For example, we had provided all materials electronically via a shared Google drive, but many teachers had difficulty navigating this drive. They preferred that we printed and organized the materials in large three-ring binders. Teachers also suggested that the initial learning modules occur before the school year begins so that they could begin administering CBM and delivering interventions right away. Also, many teachers found the learning curve to be steep at first but to level off over time, and thus desired more coaching initially with less coaching after DBI was up and running. This feedback was used to adjust the schedule of delivering learning modules and coaching activities.

Pilot Study

In the last phase of the project, we evaluated the promise of DBI-TLC to improve both teacher and student outcomes in a small, randomized pilot study using a pre- and posttest control group design. Participants were from two Midwestern districts, one large urban and one midsized city with diverse student populations, and included 20 special education teachers primarily serving children in Grades 1 to 3 (randomly assigned to DBI or Control) and 57 children with a range of mild to moderate disabilities and significant early writing needs.

As part of this study, we collected fidelity data on teachers’ CBM administration, intervention implementation, and decision-making, as well as fidelity of teachers’ access of DBI tools, our own delivery of the learning modules, and coaching. These fidelity observations revealed a couple of issues: (a) fidelity tools, particularly for intervention, did not necessarily differentiate among DBI and control teachers, nor did they capture the quality of implementation; and (b) teachers did not always make timely or appropriate decisions based on data. These findings led us to consider ways to improve both our fidelity tools (to make them more sensitive to the quality of DBI implementation) and our training and coaching activities (to improve teachers’ use of data to make instructional decisions). We discuss these issues in more depth in the “Summary and Implications” section.

Key Findings

Knowledge and Skills

At both the feasibility and pilot phases of the project, teachers’ DBI knowledge and skills appeared to improve following participation in DBI-TLC activities. During the Feasibility Study, teachers’ mean pretest performance was 28.45 items correct (SD = 4.68). Twelve weeks later, after receiving the learning modules and collaborative support, mean posttest performance was 36.27 (SD = 2.69). A paired-samples t test revealed statistically significant growth from pre- to posttest, t(10) = −6.659, p < .001, d = 1.90.

Whereas findings from the feasibility phase were encouraging, the lack of a control group precluded any causal inference we could make about the effect of DBI-TLC on teachers’ knowledge and skills. The randomized control trial conducted during the pilot phase, however, enabled us to examine whether gains made over time could be attributed to DBI-TLC. Using the pre- and posttest data collected during the Pilot Study, we ran a repeated-measures analysis of variance (RM-ANOVA) with time (pre- vs. posttest) as the within-subjects factor and condition (DBI-TLC vs. control) as the between-subjects factor (see Table 1 for means and standard deviations). This analysis revealed a statistically significant time by condition interaction, F(1, 16) = 25.99, p < .001. Whereas DBI-TLC and control teachers performed similarly at pretest (d = −.03), DBI-TLC teachers outperformed controls at posttest on the Knowledge and Skills measure, with a large effect size (d = 3.05). Thus, it appears that DBI-TLC did, in fact, explain changes in teachers’ DBI knowledge and skills.

Table 1.

Pilot Study Teacher Outcomes by Condition.

	DBI-TLC (n = 11)		Control (n = 9)
	M	SD	M	SD
Knowledge and Skills
Pretest	25.55	4.95	25.67	2.78
Posttest	32.32	2.26	23.50	3.52
Teacher Efficacy
General Efficacy
Pretest	4.18	0.88	4.50	0.80
Posttest	4.54	0.75	4.51	0.67
Personal Efficacy
Pretest	4.54	0.35	4.46	0.35
Posttest	4.94	0.51	4.46	0.50
Writing Orientation
Correct Writing
Pretest	3.03	1.05	2.50	3.03
Posttest	3.23	0.72	3.00	0.87
Explicit Instruction
Pretest	5.00	0.65	4.56	0.64
Posttest	5.77	0.25	5.06	0.51
Natural Learning
Pretest	4.37	0.56	4.24	0.55
Posttest	3.72	0.69	4.26	0.57

DBI-TLC: Data-Based Instruction-Tools, Learning, and Collaborative Support.

Teacher Efficacy and Writing Orientation

Teacher efficacy and writing orientation were assessed only during the pilot test phase of the project. Means and standard deviations for each subscale can be found in Table 1. Linear regression, in which efficacy posttest scores were regressed on condition, pretest scores, and an interaction between pretest scores and condition, revealed few differences between treatment and control teachers. Most teachers had high self-efficacy across conditions and times as well as mixed writing orientations (most teachers associated with more than one writing orientation). The explicit instruction subscale of the Writing Orientation Scale (Graham et al., 2002) was the only subscale in which a main effect of condition was significant when controlling for pretest score (R² = .63, p < .001); there were no significant interactions among the predictor variables. Teachers in the treatment condition scored an average of .56 (p = .0034) higher than the control group on the posttest.

CBM Scoring Reliability and Fidelity of DBI Implementation

Periodic reliability checks indicated that teachers scored their students’ CBM samples reliably (mean interrater agreement between teachers and scoring experts = 95%). Fidelity observations indicated that, on average, teachers administered CBM with 83% accuracy (range = 69%-100%), intervention with 79% accuracy (range = 40%-94%), and overall decision-making fidelity was 52% accuracy (range = 0%-88%). Furthermore, most DBI teachers (n = 10) reported implementing multiple instructional changes during the study period, whereas only one control teacher reported doing so.

Student Outcomes

As we mentioned earlier, student outcomes are not a primary focus of this article; these outcomes are described in detail in McMaster et al. (2016). Briefly, a series of hierarchical linear models (HLM; Raudenbush & Bryk, 2002), with students nested within teacher and controlling for students’ pretest performance, indicated a pattern of higher mean performance of DBI students compared with controls on proximal CBM tasks, with small to moderate effect sizes of d = .23 to .40, though these differences were not statistically significant. There were no reliable differences on the TEWL-III (Hresko et al., 2012), and effect sizes were small (d = .10-.18). Note that, given that the focus of this project was on iterative development and the aim was to provide preliminary evidence of feasibility and promise, the pilot study was underpowered. We are encouraged by the initial promising results and are hopeful that future efficacy work conducted with larger samples of students will produce stronger results.

Summary and Implications

In this project, we set out to develop and evaluate the feasibility and promise of DBI-TLC, a PD system designed to support teachers’ implementation of DBI to address children’s intensive early writing needs. In line with Desimone’s (2009) framework for PD research, our work was guided by a theory of change that posited that DBI-TLC would serve to improve teachers’ knowledge, skills, and self-efficacy related to DBI and writing instruction, which in turn would support their implementation of DBI with fidelity, including timely and appropriate instructional changes, which would ultimately lead to improved student outcomes.

Findings from this project support components of this theory of change to varying degrees. First, evidence from the DBI Knowledge and Skills assessment indicated that DBI-TLC did, indeed, lead to improved teacher knowledge and skills. This result likely reflects the focus on content prescribed by Desimone’s (2009) framework. Although teachers’ overall efficacy for writing instruction did not appear to change, we did observe that those who participated in DBI-TLC shifted to a more explicit writing orientation than did their control counterparts. We also observed that teachers were able to administer CBM with reasonable fidelity and score student samples reliably, and that they implemented instruction with some degree of fidelity (though instructional fidelity varied quite widely). These results are in line with Desimone’s (2009) emphasis on active learning opportunities over an extended period of time. Less evident was teachers’ fidelity of decision-making, with most teachers making timely and appropriate decisions less than half of the time. Finally, whereas children’s writing outcomes showed promising patterns on proximal measures as a result of teachers’ DBI implementation, these patterns were not statistically reliable, nor did they generalize to more distal measures of writing.

Implications for Research

Overall, we are encouraged that DBI-TLC improved teachers’ knowledge and skills, because knowledge and skills are likely necessary for changing teacher practice (e.g., Cunningham et al., 2004; Spear-Swerling, 2009). We are also encouraged that DBI-TLC teachers shifted to a more explicit writing orientation, as research indicates the importance of explicit instruction for fostering children’s early writing development (e.g., Berninger et al., 2008). In addition, research is needed to shed light on whether these factors serve as mediators of improved student outcomes; we were unable to explore such relations given our small sample, but intend to as part of future work.

Furthermore, although teacher efficacy did not appear to be affected by DBI-TLC, we are not necessarily discouraged by this outcome, because teacher efficacy was relatively high at first. This finding is in line with previous studies (e.g., Graham et al., 2001; Graham et al., 2002; Ritchey, Coker, & Jackson, 2015; Troia et al., 2011), and might reflect that teachers were study volunteers (a prerequisite of participating in this project)—Teachers motivated to participate in the research might already have a relatively strong sense of their capacity to effect improved student outcomes. Furthermore, research is needed to determine the extent to which teachers’ efficacy affects their instructional practices and, by extension, student outcomes (Graham et al., 2002; Ritchey et al., 2015; Troia et al., 2011). A related, important question for future DBI research is whether efficacy of a wider range of teachers who participate in PD such as DBI-TLC is altered, particularly for those who begin with relatively low self-efficacy. It might also be useful to determine whether levels of teacher knowledge, skill, self-efficacy, and writing orientation can be used to individualize teacher PD and support for DBI.

We are somewhat encouraged that teachers were able to implement aspects of DBI with reasonable fidelity (assessment and intervention components were implemented, on average, with close to 80% accuracy), though many questions remain. For example, it is not clear what level of fidelity is sufficient to lead to improved student outcomes. Also unclear is how best to assess fidelity when teachers are encouraged to individualize instruction (and thus possibly stray from prescribed intervention procedures). In our project, we discovered that it was particularly challenging to create a tool that was sufficiently detailed to capture critical instructional elements, but also sufficiently generic to allow for teachers’ instructional adjustments. In addition, we found our measures to be inadequate in terms of capturing instructional quality, including the extent to which teachers engaged students in meaningful ways and were appropriately responsive to students when engagement waned. Thus, we are currently exploring more in-depth ways to assess fidelity and instructional quality, including the fidelity of teachers’ intensification of instruction, and the extent to which teachers foster and reinforce student engagement during instruction.

We are somewhat discouraged that teachers largely failed to make what we operationalized as timely and appropriate instructional changes to individualize instruction for students. We do not necessarily view this outcome as a measurement problem (as we described for instructional fidelity), but possibly as an indicator of limitations to the learning and collaborative support opportunities provided through DBI-TLC that shored up teachers’ decision-making capacity. Perhaps Desimone’s (2009) notion of coherence, ensuring that DBI components align with theory and core academic standards, and are integrated into existing instructional routines, was limited as DBI was not something that most of our teachers were used to doing. Furthermore, work is needed to determine how best to teach teachers to correctly interpret data, as well as to act upon the data in a timely way. We are revisiting our learning modules and coaching procedures to determine where there might be areas where we need to shore up supports for teachers in their use of data. A recent special issue of Learning Disabilities Research & Practice (February 2017, Volume 32, Issue 1) on teachers’ use of data for instructional decision-making also provides valuable insights that we can draw from in this regard.

Implications for Practice

Although we view our findings as preliminary given the developmental nature of the above-described work, we believe that there are several implications for teacher development that can be applied to both preservice and in-service training. First, there is a reported widespread lack of attention to writing assessment and instruction in teacher preparation and PD programs (Cutler & Graham, 2008). We believe we can fill this gap with the research-based tools and learning modules that we have developed as part of this project, and that use of these products has great promise to improve teacher knowledge and skills related to DBI and early writing. We have already begun to do so, by incorporating these resources in our teacher preparation programs and making them available through http://arc.missouri.edu/dbi_early.aspx. Second, providing tools and learning opportunities is necessary, but not likely sufficient for changing teacher practices. Many teachers are likely to require ongoing supports to successfully adopt and implement DBI in early writing with fidelity.

Third, particular attention is needed for supporting teachers’ use of data for instructional decision-making. This component of DBI is essential, yet has remained one of the most difficult skills for teachers to learn and implement (Espin et al., in press; Stecker et al., 2005). It may be that data-based decision-making requires a level of expertise that can only be developed through more extensive and intensive PD, support, and experience than what we provided through DBI-TLC. Throughout our project, we utilized Desimone’s (2009) feature of effective PD, collective participation, ensuring that teachers have frequent opportunities to collaborate with coaches and peers. Outside of the project, however, this level of support may not always be available. Furthermore, research is needed to determine how best to achieve sustainable results, including an examination of the role of instructional leaders. Researchers and practitioners should continue to work together to find ways to develop this type of instructional expertise, so that children with the most intensive early writing needs can receive the maximum benefit possible from individualized writing instruction.

Footnotes

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This work was supported in part by the Institute of Education Sciences, U.S. Department of Education, through Grant R324A130144 to the University of Minnesota.

Author Biographies

Erica S. Lembke is a professor and chair in the Department of Special Education at the University of Missouri. Her research interests are in the area of data-based decision making for teachers.

Kristen L. McMaster, PhD, is a professor of Special Education in the Department of Educational Psychology, University of Minnesota. Her research interests include creating conditions for successful response to intervention of students at risk and students with disabilities, particularly in reading comprehension and written expression. Specific research focuses on (1) promoting teachers’ use of data-based decision making and evidence-based instruction and (2) developing intensive, individualized interventions for students for whom generally effective instruction is not sufficient.

R. Alex Smith conducts research in early writing, data-based instruction, and curriculum based measures in writing with at-risk learners and dual language learners.

Abigail Allen is an assistant professor of Special Education at Clemson University. Her research interests include early writing assessment and intervention, curriculum-based measures, and technology in teacher education.

Dana Brandes earned her PhD from the University of Minnesota in the Department of Special Education. She holds a teaching license in learning disabilities and is interested in reading interventions for academically diverse learners.

Kyle Wagner has worked as a school psychologist in Texas and Minnesota. His research interests include optimization of practice for academic skills, and using data-based interventions to intensify intervention.

References

Allinder

R. M.

(1994). The relationship between efficacy and the instructional practices of special education teachers and consultants. Teacher Education and Special Education, 17, 86-95.

Berninger

Amtmann

(2003). Preventing written expression disabilities through early and continuing assessment and intervention for handwriting and/or spelling problems: Research into practice. In Swanson

H. L.

Harris

Graham

(Eds.), Handbook of research on learning disabilities (pp. 345-363). New York, NY: Guilford Press.

Berninger

V. W.

Nielsen

K. H.

Abbott

R. D.

Wijsman

Raskind

(2008). Writing problems in developmental dyslexia: Under-recognized and under-treated. Journal of School Psychology, 46, 1-21.

Biancarosa

Snow

(2004). Reading next—A vision for action and research in middle and high school literacy: A report to Carnegie Corporation of New York. Washington, DC: Alliance for Excellence in Education.

Cunningham

A. E.

Perry

K. E.

Stanovich

K. E.

Stanovich

P. J.

(2004). Disciplinary knowledge of K-3 teachers and their knowledge calibration in the domain of early literacy. Annals of Dyslexia, 54, 139-167.

Cutler

Graham

(2008). Primary grade writing instruction: A national survey. Journal of Educational Psychology, 100, 907-919.

Danielson

Rosenquist

(2014). Introduction to the TEC special issue on data-based individualization. Teaching Exceptional Children, 46, 6-12. doi:10.1177/0040059914522965

Deno

S. L.

(1990). Individual differences and individual difference: The essential difference of special education. The Journal of Special Education, 24, 160-173.

Deno

S. L.

Mirkin

P. K.

(1977). Data-based program modification: A manual. Reston, VA: Council for Exceptional Children.

10.

Desimone

L. M.

(2009). Improving impact studies of teachers’ professional development: Toward better conceptualizations and measures. Educational Researcher, 38, 181-199.

11.

Espin

C. A.

Wayman

M. M.

Deno

S. L.

McMaster

K. L.

de Rooij

(in press). Data-based decision-making: Developing a method for capturing teachers’ understanding of CBM graphs. Learning Disabilities Research & Practice, 32(1), 8-21. doi: 10.1111/ldrp.12123

12.

Fuchs

L. S.

Deno

S. L.

Mirkin

P. K.

(1984). The effects of frequent curriculum-based measurement and evaluation on pedagogy, student achievement, and student awareness of learning. American Educational Research Journal, 21, 449-460.

13.

Fuchs

L. S.

Fuchs

(2007). A model for implementing responsiveness to intervention. Teaching Exceptional Children, 39(5), 14-20.

14.

Gibson

Dembo

M. H.

(1984). Teacher efficacy: A construct validation. Journal of Educational Psychology, 76, 569-582. doi:10.1037/0022-0663.76.4.569

15.

Graham

Harris

K. R.

Fink

MacArthur

C. A.

(2001). Teacher efficacy in writing: A construct validation with primary grade teachers. Scientific Studies of Reading, 5, 177-202.

16.

Graham

Harris

K. R.

MacArthur

Fink

(2002). Primary grade teachers’ theoretical orientations concerning writing instruction: Construct validation and a nationwide survey. Contemporary Educational Psychology, 27, 147-166.

17.

Graham

Hebert

(2010). Writing to read: Evidence for how writing can improve reading. New York, NY: Carnegie Corporation.

18.

Graham

Perin

(2007). A meta-analysis of writing instruction for adolescent students. Journal of Educational Psychology, 99, 445-476.

19.

Hampton

D. D.

Lembke

E. S.

(2016). Examining the technical adequacy of progress monitoring using early writing curriculum-based measures. Reading & Writing Quarterly, 32, 336-352.

20.

Hresko

W. P.

Herron

S. R.

Peak

P. K.

Hicks

D. L.

(2012). Test of Early Written Language (TEWL-3). Austin, TX: Pro-Ed.

21.

Lembke

Deno

S. L.

Hall

(2003). Identifying an indicator of growth in early writing proficiency for elementary school students. Assessment for Effective Intervention, 28(3-4), 23-35.

22.

McMaster

K. L.

Pétursdóttir

A. L.

(2009). Technical features of curriculum-based measures for beginning writers. Journal of Learning Disabilities, 42, 41-60.

23.

McMaster

K. L.

Yeo

Deno

S. L.

Parker

Ellis

(2011). Curriculum-based measures of beginning writing: Technical features of the slope. Exceptional Children, 77, 185-206.

24.

McMaster

K. L.

Kunkel

Shin

Jung

Lembke

(in press). Early writing intervention: A best-evidence synthesis. Journal of Learning Disabilities. 0022219417708169

25.

McMaster

K. L.

Lembke

Shin

(2016). Technical report for the pilot study: DBI-TLC. Minneapolis: University of Minnesota.

26.

McMaster

K. L.

Ritchey

K. D.

Lembke

(2011). Curriculum-based measurement for beginning writers: Recent developments and future directions. In Scruggs

T. E.

Mastropieri

M. A.

(Eds.), Assessment and intervention: Advances in learning and behavioral disabilities (Vol. 24, pp. 111-148). Bingley, UK: Emerald.

27.

National Center for Education Statistics. (2012). The nation’s report card: Writing 2011 (NCES 2012–470). Washington, DC: Institute of Education Sciences, U.S. Department of Education.

28.

Poch

A. L.

McMaster

Lembke

(2016). Feasibility technical report: DBI-TLC. Columbia: University of Missouri.

29.

Poch

A. L.

Smith

R. A.

Jung

Wagner

Lembke

McMaster

(in preparation). Using academic coaching to support teachers’ use of data based instruction: A review of the literature.

30.

Raudenbush

S. W.

Bryk

A. S.

(2002). Hierarchical linear models: Applications and data analysis methods (Vol. 1). Thousand Oaks, CA: SAGE.

31.

Ritchey

K. D.

Coker

D. L.

Jr. Jackson

A. F.

(2015). The relationship between early elementary teachers’ instructional practices and theoretical orientations and students’ growth in writing. Reading and Writing, 28, 1333-1354.

32.

Shanahan

(2004). Overcoming the dominance of communication. In Adolescent literacy research and practice (pp. 59-74). New York: Guilford Press.

33.

Spear-Swerling

(2009). A literacy tutoring experience for prospective special educators and struggling second graders. Journal of Learning Disabilities, 42, 431-443.

34.

Spear-Swerling

Zibulsky

(2014). Making time for literacy: Teacher knowledge and time allocation in instructional planning. Reading and Writing, 27, 1353-1378.

35.

Stecker

P. M.

Fuchs

L. S.

Fuchs

(2005). Using curriculum-based measurement to improve student achievement: Review of research. Psychology in the Schools, 42, 795-819.

36.

Troia

G. A.

Lin

S. J. C.

Cohen

Monroe

B. W.

(2011). A year in the writing workshop. The Elementary School Journal, 112, 155-182.